Cassandra on Kubernetes: Persistent Storage Architecture and High Availability

Apache Cassandra is a wide-column, peer-to-peer distributed database built for write-heavy workloads that cannot tolerate a single point of failure. Running it on Kubernetes alongside other stateful workloads is now common practice, but Cassandra’s storage engine has requirements that most CSI drivers were not designed around. Every node writes to a commit log for durability and flushes in-memory tables to immutable SSTables on disk, and both paths compete for the same volume unless the platform separates them deliberately.

The bigger operational cost shows up when a node fails. On local storage, a failed Cassandra pod cannot simply reschedule with its data intact: the standard recovery path is nodetool replace, which bootstraps a replacement node by streaming its full share of the data from other replicas across the network. For a multi-terabyte node, that stream can take hours, during which the cluster runs with reduced replication and elevated read/write latency on the nodes serving the stream.

This guide covers the persistent volume architecture for Cassandra on Kubernetes, configuration for the two most common operators, and how disaggregated storage changes the node-failure recovery model from a multi-hour data stream to a volume reattachment.

Why Cassandra Storage Is Different on Kubernetes

Apache Cassandra uses a log-structured merge-tree (LSM-tree) storage engine, and that design drives two distinct, concurrent I/O patterns on every node:

1. Commit log writes. Every write is appended to the commit log before it is acknowledged, giving Cassandra durability independent of when the corresponding memtable flushes to disk. Commit log writes are small, sequential, and latency-sensitive: a slow commit log volume directly increases write-path latency for every client request.

2. Memtable flush and compaction. When a memtable fills, Cassandra flushes it to an immutable SSTable file on disk. A background compaction process continuously merges SSTables to remove overwritten and deleted data (tombstones) and keep read amplification under control. Compaction is throughput-heavy and bursty, and it reads and writes the same data directory that client reads query.

Cassandra’s own documentation recommends placing the commit log and the data directories on separate devices specifically to prevent compaction I/O from adding latency to commit log fsyncs. On a single shared Kubernetes persistent volume, a heavy compaction cycle can stall commit log writes and produce write-latency spikes that look like a database problem but are actually a storage contention problem.

The second Kubernetes-specific issue is node replacement. Cassandra’s own replication (via NetworkTopologyStrategy and a configurable replication factor) is what makes the cluster durable, not the underlying block storage. When a node’s local disk is lost, or when a pod on hostPath or local-PV storage cannot reschedule because its data lives on one specific machine, the standard repair path is to bootstrap a brand-new node and stream its full data range from the remaining replicas. That stream is proportional to the amount of data owned by the failed node, not the length of the outage.

Persistent Volume Architecture for Cassandra StatefulSets

Cassandra runs on Kubernetes as a StatefulSet (directly, or through an operator that manages one), with each pod requesting a PersistentVolumeClaim through volumeClaimTemplates. Each node owns its own data and commit log files; nothing is shared between pods at the storage layer, only at the Cassandra replication layer.

Table 1: Persistent volume configuration recommendations for Cassandra on Kubernetes.

A StorageClass for Cassandra using the simplyblock CSI driver:

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: cassandra-nvme
5
provisioner: csi.simplyblock.io
6
parameters:
7
  replication: "2"
8
  compression: "false"
9
  qos_iops_per_gb: "40"
10
reclaimPolicy: Retain
11
volumeBindingMode: WaitForFirstConsumer
12
allowVolumeExpansion: true

compression is left off at the storage-class level because Cassandra already compresses SSTables per-table (typically with LZ4). The qos_iops_per_gb limit exists to keep one node’s compaction burst from consuming bandwidth that a neighboring node’s commit log needs on the same shared NVMe pool.

For clusters that need I/O isolation between the commit log and data paths, provision two volumes per pod, one per volumeClaimTemplates entry, and mount them at /var/lib/cassandra/commitlog and /var/lib/cassandra/data respectively. This mirrors Cassandra’s own bare-metal recommendation and, on a disaggregated NVMe pool, each volume is still an independent namespace with its own QoS ceiling rather than a physically separate disk.

Running Cassandra on Kubernetes and dreading the multi-hour rebuild after a node failure? Simplyblock’s disaggregated NVMe/TCP storage lets a replacement Cassandra pod reattach its predecessor’s volume in seconds, skipping the full-stream bootstrap entirely. Talk to a storage architect

Configuring Cassandra Operators for Persistent Storage

Two operator projects dominate production Cassandra-on-Kubernetes deployments: DataStax’s cass-operator, and K8ssandra, which packages cass-operator together with Reaper (repair scheduling), Medusa (backup/restore), and Stargate (data API access).

cass-operator (CassandraDatacenter CRD)

Storage is configured under spec.storageConfig.cassandraDataVolumeClaimSpec on the CassandraDatacenter custom resource. Racks map to Kubernetes topology, and each rack should span a distinct failure domain:

1
apiVersion: cassandra.datastax.com/v1beta1
2
kind: CassandraDatacenter
3
metadata:
4
  name: dc1
5
spec:
6
  clusterName: cassandra-cluster
7
  serverType: cassandra
8
  serverVersion: "4.1.5"
9
  size: 3
10
  racks:
11
    - name: rack1
12
    - name: rack2
13
    - name: rack3
14
  storageConfig:
15
    cassandraDataVolumeClaimSpec:
16
      accessModes:
17
        - ReadWriteOnce
18
      storageClassName: cassandra-nvme
19
      resources:
20
        requests:
21
          storage: 500Gi

cass-operator generates the underlying StatefulSet per rack and handles rolling restarts one rack at a time so that at most one replica of any token range is briefly unavailable at once.

K8ssandra Operator

K8ssandra Operator wraps the same CassandraDatacenter storage configuration inside a higher-level K8ssandraCluster resource, and adds Reaper for continuous repair scheduling and Medusa for CSI-snapshot-backed backup and restore:

1
apiVersion: k8ssandra.io/v1alpha1
2
kind: K8ssandraCluster
3
metadata:
4
  name: cassandra-cluster
5
spec:
6
  cassandra:
7
    serverVersion: "4.1.5"
8
    datacenters:
9
      - metadata:
10
          name: dc1
11
        size: 3
12
        storageConfig:
13
          cassandraDataVolumeClaimSpec:
14
            storageClassName: cassandra-nvme
15
            accessModes: ["ReadWriteOnce"]
16
            resources:
17
              requests:
18
                storage: 500Gi
19
    racks:
20
      - name: rack1
21
      - name: rack2
22
      - name: rack3

K8ssandra’s built-in Reaper integration matters for the storage discussion because scheduled repair reads generate the same kind of throughput-heavy, bursty I/O as compaction. Per-volume QoS limits on the underlying StorageClass keep repair from starving client-facing read latency on the same node.

Table 2: Cassandra operator storage and operations comparison on Kubernetes.

High Availability: Replication Factor, Consistency, and Node Recovery

Cassandra’s availability model is independent of the underlying storage layer: it comes from NetworkTopologyStrategy, a per-datacenter replication factor (RF), and the consistency level chosen per query. A common production configuration is RF=3 with LOCAL_QUORUM reads and writes, which tolerates one node being unavailable per datacenter while every request still reaches a quorum of replicas.

What the storage layer does control is how quickly an unavailable node returns to full standing, and that difference is largest at node failure, not during normal operation:

• Local NVMe (hostPath or local PV): When the node’s disk is lost, or the pod cannot reschedule because it is bound to that specific machine, an operator runs nodetool replace on a new pod. The replacement streams its full token range from the remaining replicas across the network. For a multi-terabyte node this can take hours, during which the cluster serves reads and writes at RF-1 for the affected range and the replicas doing the streaming carry extra I/O load.

• Cloud block storage (gp3, Premium SSD): The pod can reschedule and reattach its existing volume, avoiding a full stream, but only within the same availability zone. In a multi-AZ node pool, a pod that reschedules to a different zone than its volume fails to attach, forcing the full-stream path anyway. Reattachment that does succeed still adds 20 to 60 seconds of pod-startup latency.

• Disaggregated NVMe/TCP (simplyblock): The replacement pod reattaches the failed node’s existing data and commit log volumes on any node in the cluster within seconds, with the SSTables and commit log segments intact. Cassandra replays any hints held by other nodes (hinted handoff) and the node participates in the next scheduled repair to reconcile any writes it missed during the outage, but it skips the full-stream bootstrap entirely.

Table 3: Cassandra node-failure recovery comparison by storage model.

Rack awareness compounds this. Cassandra distributes replicas across racks so that a single rack failure never takes an entire token range’s replica set offline. On Kubernetes, racks are typically mapped to node pools or availability zones via the operator’s racks field. Pairing that rack topology with storage that is not bound to any single node means a rack-wide maintenance event drains and reschedules pods without triggering a wave of full-stream rebuilds across the cluster.

Snapshot-based backup benefits from the same underlying capability. Simplyblock’s CSI external snapshotter integration, which Medusa can call into for K8ssandra clusters, takes consistent point-in-time snapshots of a node’s data volume without pausing writes, and restoring from a snapshot brings a node back to a known-good state far faster than a full repair-based rebuild for large datasets.

For teams that have followed the CSI driver discussion earlier in this series, Cassandra is the clearest case where a storage-layer capability, volume reattachment independent of node identity, eliminates an entire class of database-level recovery work. See also the earlier entries covering MySQL, MongoDB, Redis, and Kafka on Kubernetes.

Questions and Answers

What persistent storage settings work best for Cassandra on Kubernetes?

Use volumeBindingMode: WaitForFirstConsumer so the volume provisions in the same zone and rack as the scheduled pod, reclaimPolicy: Retain to prevent data loss on accidental PVC deletion, and allowVolumeExpansion: true for online capacity growth. Plan for 15,000 to 40,000 IOPS per node depending on write throughput, replication factor, and compaction strategy. If the commit log and data directories share a volume, size IOPS headroom for both paths; if you split them into two PVCs, configure QoS separately so compaction bursts on the data volume cannot starve commit log fsyncs.

How does Cassandra’s replication factor interact with Kubernetes persistent volumes?

Cassandra replication happens at the cluster layer through NetworkTopologyStrategy and a configured replication factor, entirely independent of how each node’s own data is stored. Each node’s PVC holds only that node’s local share of the data. When a node fails and its pod reschedules, the CSI driver reattaches the existing PVC (with SSTables and commit log intact) to the new pod, and Cassandra’s own hinted handoff and repair mechanisms reconcile any writes made during the outage. The PVC provides durability for one node’s local copy; Cassandra’s replication provides durability across the cluster.

Does NVMe/TCP storage avoid the full-stream node replacement in Cassandra?

Yes, in the common case. nodetool replace full-stream rebuilds are needed when a node’s data is genuinely lost or unreachable. With disaggregated NVMe/TCP storage, a failed pod’s existing data and commit log volumes reattach to a replacement pod on any node in the cluster, so the data is not lost or unreachable, it is just temporarily unavailable while the pod restarts. The node rejoins after replaying hints and participating in scheduled repair, which is materially faster than streaming its full token range from other replicas.

How does simplyblock compare to local NVMe for Cassandra on Kubernetes?

Local NVMe gives the lowest absolute commit log latency, at the cost of binding every pod to a specific machine. Simplyblock’s disaggregated NVMe/TCP storage delivers throughput comparable to local NVMe for the commit log and compaction I/O patterns, typically in the 100 to 400 microsecond latency range on 25 GbE fabric, while allowing pods to reschedule freely, applying per-volume QoS to isolate compaction and repair from commit log writes, and supporting CSI snapshots for fast backup and restore. For teams running cass-operator or K8ssandra in production, the reduced node-replacement time and rack-maintenance flexibility offset the small latency delta for most workloads.

Does this architecture work with K8ssandra’s Reaper and Medusa components?

Yes. Reaper’s scheduled repair reads generate throughput-heavy I/O similar to compaction, and per-volume QoS on the StorageClass keeps that traffic from degrading client-facing read latency on the same node. Medusa’s backup and restore workflow can call into simplyblock’s CSI external snapshotter for consistent, non-blocking point-in-time snapshots of each node’s data volume, which restores faster than a repair-based rebuild for large keyspaces.