How to Choose Persistent Storage for Kubernetes: A Decision Framework

Platform engineers choosing persistent storage for Kubernetes face a decision that affects every stateful workload in the cluster. The wrong choice shows up as latency spikes in production databases, restore failures during incident response, and operational overhead that compounds as the cluster grows.

This guide gives a structured framework for making the decision: how to profile your workloads, what data protection requirements look like in practice, how to compare operational models, and what a rigorous proof-of-concept should test.

Step 1: Profile Your Workloads Before Evaluating Storage

Storage decisions made before understanding workload behavior produce mismatches between storage capabilities and application requirements. Start with workload profiling.

Workload Types That Drive Storage Requirements

Transactional databases (PostgreSQL, MySQL, MongoDB)

Characteristics: random I/O, small block sizes (4-16KB), write-heavy or mixed, latency-sensitive. A PostgreSQL instance under load generates a high rate of 8KB random writes to WAL and data files simultaneously. Storage latency at P99 directly affects query response time.

What to measure: P99 write latency at the block device level. IOPS capacity at the database’s working queue depth. How latency changes as the database grows.

Analytical databases and data warehouses

Characteristics: sequential reads, large block sizes (64KB-1MB), read-heavy with periodic large writes. These workloads tolerate higher latency but require high sequential throughput.

What to measure: Sequential read throughput at the block sizes the database uses (check the database’s I/O block size configuration). CPU overhead of storage reads (important if the storage system is CPU-intensive).

Message queues and event streaming (Kafka, Pulsar)

Characteristics: sequential writes to segment files, sequential reads for consumer catch-up. Very sensitive to write latency consistency: tail latency spikes cause consumer lag.

What to measure: P99.9 write latency. Behavior under sustained write load. How storage handles the combination of high-throughput sequential writes with concurrent sequential reads.

Machine learning and AI workloads

Characteristics: large sequential reads for training data, periodic large writes for checkpoints. Storage access patterns can be bursty, with periods of high throughput followed by low I/O during GPU compute.

What to measure: Sequential read throughput at model file sizes. Checkpoint write performance (often large blocks written quickly). Behavior when multiple training jobs access shared storage concurrently.

Backup and ETL workloads

Characteristics: large sequential reads and writes, non-latency-sensitive, but high throughput requirements. These workloads are a noisy-neighbor risk for co-located production workloads.

What to measure: How the storage system handles high-throughput bulk workloads running alongside latency-sensitive workloads. Whether per-workload I/O limits are enforceable.

Step 2: Define Your Data Protection Requirements

Data protection requirements determine whether a storage system is technically eligible before any performance evaluation.

Recovery Point Objective (RPO)

RPO is the maximum amount of data you can lose in a failure event. It determines what kind of data protection the storage system must provide.

• RPO = 0: No data loss acceptable. Requires synchronous replication: writes are committed to both primary and secondary before acknowledgement. Limits where the secondary can be located (round-trip latency must be low enough to be sustainable).
• RPO < 15 minutes: Asynchronous replication with a short lag. Primary acknowledges writes immediately; replication to secondary happens with a configurable delay. Secondary can be geographically distant.
• RPO = point-in-time snapshot: Backup-based recovery. You accept losing data since the last snapshot. RPO equals your snapshot interval.

Most production databases require at least snapshot-based recovery. Regulatory environments (healthcare, finance) often require continuous replication with documented RPO targets.

Recovery Time Objective (RTO)

RTO is how quickly the system must be back online after a failure. It determines what kind of failover the storage system must support.

• RTO < 30 seconds: Requires automatic failover with no human intervention. Multi-path storage failover (NVMe ANA, iSCSI multipath) handles node-level failures transparently. Applications continue running.
• RTO < 15 minutes: Requires a live replica that can be promoted to primary. Cross-cluster replication with a promoted secondary cluster. No restore job: the secondary is already running.
• RTO = hours: Backup-based restore is acceptable. Large volumes restored from object storage. Suitable for non-production environments or workloads with low availability requirements.

Durability Requirements

Durability is the probability of not losing data once it is written and acknowledged. Cloud storage providers publish durability SLAs (eleven nines for S3). For on-premises and self-managed storage, durability is determined by the protection scheme.

• N+1 erasure coding (e.g., 4+2): Tolerates 2 simultaneous node/drive failures before data loss. Lower storage overhead than replication.
• Triple replication: Tolerates 2 simultaneous failures. 3x storage overhead.
• Single replication (RAID-like): Tolerates 1 failure. Lower overhead but higher risk.

For production workloads, N+2 failure tolerance is standard. Anything less requires compensating controls (frequent backups, tested restore procedures).

Choosing storage for a Kubernetes cluster and need to validate your requirements against real workloads? simplyblock’s engineering team helps platform engineers match their specific I/O profiles and data protection requirements to the right storage architecture. Talk to a storage architect

Step 3: Evaluate the Operational Model

Storage that works correctly in a proof-of-concept can still fail operationally if the day-2 model does not fit your team’s capabilities.

Questions to Ask About Operations

How are storage nodes upgraded?

Does the storage system support rolling upgrades with no I/O interruption to running pods? What happens if a node fails mid-upgrade? Does the upgrade procedure require a maintenance window? Can you test the upgrade under load before committing?

See Zero-Downtime Storage Upgrades: What ‘Live Upgrade’ Actually Requires for a detailed breakdown of what storage vendors mean when they claim live upgrade support.

How does the storage system surface health and performance information?

Does the storage system expose per-volume metrics (IOPS, latency, throughput) via Prometheus? Can you see which workload is consuming storage I/O when a database is slow? Does it integrate with your existing alerting pipeline?

Without per-volume observability, storage incidents become long multi-step investigations.

What does capacity planning look like?

Can you add storage capacity without taking the cluster offline? How does the rebalancing operation affect I/O latency for running workloads? How long does a rebalance take for a cluster of your size?

What happens when something breaks?

What is the support model: community forum, ticketed support, or dedicated account? What is the escalation path for a P1 incident? Has the vendor published post-mortems or engineering blog posts that show how they handle failures?

Storage Provisioning Model

Hyperconverged: Storage and compute run on the same nodes. Lower infrastructure cost, easier network setup. Trade-off: storage and compute resources scale together, which limits flexibility.

Disaggregated (dedicated storage nodes): Storage nodes are separate from compute nodes. Storage and compute scale independently. Higher infrastructure overhead but better flexibility for workloads with asymmetric storage and compute requirements.

Cloud-managed storage (EBS, Azure Disk, GCP PD): Storage is a managed service. Simplest operations but limited performance headroom, higher per-GB cost at scale, and no control over the underlying system. Latency is typically 1-5ms, not achievable below 1ms.

Step 4: Structure a Rigorous Proof-of-Concept

A synthetic benchmark is not a proof-of-concept. A benchmark runs artificial workloads against a single volume. A proof-of-concept runs representative workloads against representative volumes and tests the failure modes that matter in production.

POC Checklist

Performance validation

Run FIO workloads that match your application’s I/O profile (block size, read/write ratio, queue depth). Measure P50, P99, P99.9 latency. Run the benchmark long enough for warm-up effects to pass (at least 10 minutes of steady state).

Do not compare FIO results across storage systems without ensuring the test configuration is identical. Small differences in queue depth, block size, or I/O pattern produce large differences in results.

Failure mode testing

Kill a storage node while a FIO workload is running. Measure the I/O interruption window from the application’s perspective. Verify it matches the vendor’s documented failover time.

Kill a storage node during a rolling upgrade. Verify that the upgrade procedure handles concurrent failures.

Concurrent workload isolation

Run a bulk workload (high-throughput FIO) alongside a latency-sensitive workload (low-queue-depth random write FIO). Measure whether the latency-sensitive workload’s P99 latency increases. If the storage system supports per-PVC QoS, configure limits and verify enforcement.

Snapshot and restore

Take a snapshot of a large volume while a write workload is running. Measure snapshot creation time. Restore the snapshot to a new volume. Measure restore time. Verify that the application behaves correctly after restore.

Monitoring integration

Verify that storage metrics are available in your Prometheus stack. Confirm you can alert on P99 latency thresholds. Check that volume health events surface in Kubernetes.

What a Good POC Output Looks Like

A POC should produce:

• Latency curves (P50, P99, P99.9) for each workload type
• Failover time measurements from multiple failure injections
• Storage overhead calculation (raw capacity vs. usable capacity)
• Operational runbooks for upgrade, capacity expansion, and failure recovery
• Evidence that monitoring integration works end-to-end

This evidence should be sufficient for an architecture review. Any vendor claim not validated in the POC is a risk.

Storage Option Quick Reference

Summary

A defensible storage decision follows this sequence:

1. Profile your workloads: what I/O pattern, what latency target, what throughput requirement.
2. Define your data protection requirements: RPO, RTO, durability (failure tolerance).
3. Evaluate the operational model: upgrades, observability, capacity planning, support.
4. Run a rigorous POC: performance under load, failure mode behavior, concurrent workload isolation.

Storage decisions made without workload profiling and failure mode testing produce clusters where the first incident reveals what the benchmark did not.

Related Reading

• NVMe-oF for Kubernetes Storage: A Platform Engineer’s Guide: how the protocol layer works
• Storage Observability for Kubernetes: how to instrument storage with Prometheus
• QoS and Performance Isolation: how to prevent noisy-neighbor storage contention
• How to Evaluate Storage Vendor Reliability Claims: a framework for assessing vendor durability and availability claims

Questions and Answers

What are the most important factors when choosing Kubernetes persistent storage?

The most important factors depend on your workloads. Start with I/O profile: transactional databases need low-latency random I/O, analytical workloads need sequential throughput, and message queues need consistent write latency. Layer on data protection requirements (RPO, RTO, durability), then operational model (upgrades, observability, capacity planning, support). Storage that performs well in a benchmark but cannot be upgraded without downtime or lacks per-volume metrics will create operational problems that dwarf any performance advantage.

How do I define RPO and RTO requirements for my Kubernetes storage?

RPO (recovery point objective) is the maximum data loss you can accept: zero RPO requires synchronous replication, under 15 minutes requires asynchronous replication with short lag, and longer RPO can use snapshot-based backup. RTO (recovery time objective) is how fast the system must be back online: under 30 seconds requires automatic multi-path failover, under 15 minutes requires a live replica that can be promoted, and longer RTO can use backup restore. Regulatory environments (healthcare, finance) often have explicit RPO and RTO targets documented in compliance requirements.

What is the difference between hyperconverged and disaggregated storage for Kubernetes?

Hyperconverged storage runs on the same nodes as compute workloads, lowering infrastructure cost and simplifying networking. The tradeoff is that storage and compute scale together, which is limiting for workloads with asymmetric requirements. Disaggregated storage runs on dedicated nodes, allowing storage and compute to scale independently. This is better for workloads that need more storage than compute or vice versa, but adds infrastructure overhead and network configuration complexity. Cloud-managed storage (EBS, Azure Disk) is a third category: fully managed, simplest to operate, but with a 1-5ms latency floor and higher per-GB cost at scale.

What should a Kubernetes storage proof-of-concept actually test?

A POC should test five things beyond basic performance: (1) P99 and P99.9 latency at your application’s queue depth and block size, not just throughput; (2) failover behavior when a storage node is killed during an active workload; (3) concurrent workload isolation between a latency-sensitive workload and a bulk workload running simultaneously; (4) snapshot and restore time on a volume of representative size with a write workload running; (5) end-to-end monitoring integration confirming Prometheus metrics and Kubernetes events surface correctly. Any vendor claim not validated in the POC is a risk.

How do I compare NVMe-oF storage against Ceph and other open-source options?

The comparison depends on your workload profile. Ceph (via Rook) provides mature, general-purpose distributed storage with broad feature coverage but high operational complexity and multi-millisecond latency for block workloads. Longhorn is simpler to operate but uses full-volume synchronous replication, which limits throughput for large volumes. NVMe-oF-based storage (such as simplyblock) delivers sub-millisecond latency for high-IOPS workloads but requires NVMe drives on storage nodes. For production databases and AI training workloads where storage latency directly affects application performance, NVMe-oF is the right starting point for evaluation. For large clusters with mixed workload types and a dedicated storage operations team, Ceph remains competitive.