CockroachDB on Kubernetes: Persistent Storage Architecture and High Availability

CockroachDB is a distributed SQL database that speaks the PostgreSQL wire protocol while spreading data across a shared-nothing cluster of nodes. It splits every table into fixed-size ranges, replicates each range through Raft consensus, and persists the result to disk with Pebble, an LSM-tree storage engine written in Go. Running CockroachDB on Kubernetes is well supported through an official operator and Helm chart, but the storage layer underneath still decides two things that operators care about most: how much latency a write commit picks up from disk, and how long a node failure takes to recover from.

Every Raft write requires a majority of a range’s replicas to fsync their log entry before the write returns as committed, so the slowest replica in the quorum sets the commit latency floor for that range. Every node loss triggers CockroachDB’s automatic rebalancer to bring the cluster back to full replication, and how much data that rebalancer has to move over the network depends entirely on whether the node’s on-disk state survived the failure.

This guide covers the persistent volume architecture for CockroachDB on Kubernetes, storage configuration for the CockroachDB Kubernetes Operator and Helm chart, and how disaggregated storage changes the node-failure recovery model from a full range up-replication to a volume reattachment.

Why CockroachDB Storage Is Different on Kubernetes

CockroachDB organizes every table and index into ranges, contiguous slices of the keyspace that default to roughly 512 MiB before CockroachDB splits them in two. Each range is an independent Raft group with its own leader, log, and, by default, three replicas spread across nodes or availability zones according to locality and replication-zone constraints. A range’s Raft log and the SSTables that hold its committed data both live in the same Pebble store on each replica’s node, one Pebble instance per CockroachDB node covering every range replica that node holds.

This has two consequences that make CockroachDB’s storage requirements stricter than a typical stateless service:

1. Commit latency is a quorum property, not a single-node property. A transaction cannot commit until a majority of its range’s replicas have fsynced the Raft log entry to disk. With the default replication factor of three, that means two out of three nodes, so one slow disk directly adds to the commit latency every client observes for writes touching that range, even if the other two replicas are fast.

2. Recovery is a per-range, not per-node, operation. CockroachDB’s rebalancer does not think in terms of “replace this node’s disk.” It thinks in terms of “this range is down to two of three replicas, add a new one.” When a node’s storage is genuinely lost, every range that had a replica on that node needs a brand-new replica added elsewhere, each one streamed as a full range snapshot from the surviving leader, an operation CockroachDB deliberately rate-limits (kv.snapshot_rebalance.max_rate) so a large node loss does not saturate cluster network and CPU all at once.

On Kubernetes specifically, CockroachDB runs as a StatefulSet with one PersistentVolumeClaim per pod via volumeClaimTemplates, and replicas are pinned to that pod’s stable identity for locality and constraint purposes. If the underlying volume is tied to a specific node, as with hostPath or local-PV storage, a node failure does not just lose a disk. It forces the rebalancer into the full snapshot-streaming path for every range that node was holding, proportional to the amount of data on that node rather than to the length of the outage.

Persistent Volume Architecture for CockroachDB Clusters

CockroachDB’s own deployment documentation recommends fast, locally attached NVMe SSDs specifically because Pebble’s write-ahead entries and compaction I/O compete for the same disk that Raft log fsyncs depend on. Provisioning generous headroom matters too: Pebble’s log-structured compaction temporarily needs extra space to write new SSTables before reclaiming space from old ones, so store capacity should run well ahead of logical data size rather than at the edge of it.

Table 1: Persistent volume configuration recommendations for CockroachDB on Kubernetes.

Planning a CockroachDB cluster on Kubernetes? Talk to us about sizing NVMe/TCP volumes for Raft-heavy write paths before you commit to a StorageClass. Talk to a storage architect

Deploying CockroachDB on Kubernetes: Operator and Helm Chart Storage Configuration

Cockroach Labs ships two supported paths to run CockroachDB on Kubernetes, and both configure storage the same way underneath: a volumeClaimTemplates entry on the generated StatefulSet.

The CockroachDB Kubernetes Operator manages the cluster through a CrdbCluster custom resource, with storage set under spec.dataStore.pvc:

1
apiVersion: crdb.cockroachlabs.com/v1alpha1
2
kind: CrdbCluster
3
metadata:
4
  name: cockroachdb
5
spec:
6
  dataStore:
7
    pvc:
8
      spec:
9
        storageClassName: cockroachdb-nvme
10
        accessModes:
11
          - ReadWriteOnce
12
        resources:
13
          requests:
14
            storage: 200Gi
15
  nodes: 3

The Helm chart (cockroachdb/cockroachdb) exposes the equivalent configuration under statefulset.persistentVolume:

1
statefulset:
2
  persistentVolume:
3
    storageClass: cockroachdb-nvme
4
    size: 200Gi

Both paths generate the same StatefulSet shape: one PVC per pod, a stable pod identity used for locality-aware replica placement, and a cockroach start command pointing --store at the mounted volume. Deploying without the operator or Helm chart, by hand-writing the StatefulSet, works the same way but leaves upgrade orchestration, TLS certificate rotation, and node decommissioning to be scripted manually.

Table 2: CockroachDB Kubernetes deployment paths and storage configuration comparison.

High Availability: Range Replication and Node Recovery

CockroachDB’s default replication factor of three tolerates the loss of one replica per range without losing quorum. What determines how fast a cluster returns to full redundancy, and how much load that recovery places on the rest of the cluster, is what happens to the failed node’s storage:

• Local NVMe (hostPath or local PV): When the node’s disk is genuinely lost, or the pod cannot reschedule because it is bound to that specific machine, every range that had a replica there drops to two of three and the rebalancer adds a new replica for each one. Each new replica is a full range snapshot streamed from the current leader, rate-limited to protect the cluster, so a node holding thousands of ranges can take a long time to fully re-replicate, during which those ranges run with reduced fault tolerance.

• Cloud block storage (gp3, Premium SSD): The pod can reschedule and reattach its existing volume, skipping the snapshot-streaming path entirely, but only within the same availability zone. CockroachDB’s locality-aware replica placement often spreads a range’s three replicas across three zones specifically for resilience, and a pod that reschedules to a different zone than its own volume either fails to attach or ends up violating the locality constraint the rebalancer was maintaining, pushing the cluster back into rebalancing anyway.

• Disaggregated NVMe/TCP (simplyblock): The replacement pod reattaches the failed node’s existing Pebble store and Raft logs on any node in the cluster within seconds, with every range’s committed data and log entries intact. Each range replica simply catches up on the handful of Raft log entries committed during the brief restart window instead of receiving a fresh full-range snapshot.

Table 3: CockroachDB node-failure recovery comparison by storage model.

Running cockroach node decommission follows the same pattern deliberately, draining a node ahead of a planned removal by up-replicating every range it holds to other nodes before the node leaves the cluster. That drain is a full data copy for every range on the node, which is why Cockroach Labs’ own guidance recommends decommissioning nodes one at a time and monitoring replica counts during the process. Disaggregated storage does not remove the need to decommission a node you are permanently retiring, but it does mean an unplanned node failure, the far more common event, no longer forces that same full-copy path.

For teams that have followed the CSI driver discussion earlier in this series, CockroachDB is another case where volume reattachment independent of node identity removes an entire class of database-level rebalancing work. See also the earlier entries covering MySQL, MongoDB, Redis, Kafka, Cassandra, and RabbitMQ on Kubernetes.

Questions and Answers

What persistent storage settings work best for CockroachDB on Kubernetes?

Use volumeBindingMode: WaitForFirstConsumer so the volume provisions in the same zone as the scheduled pod, reclaimPolicy: Retain to prevent range and Raft log loss on accidental PVC deletion, and allowVolumeExpansion: true for online capacity growth. Plan for 8,000 to 30,000 IOPS per node depending on range count and write volume, since Pebble’s compaction I/O and Raft log fsyncs compete for the same volume unless you size headroom generously above logical data size.

How does range replication interact with Kubernetes persistent volumes?

Each range’s three replicas live on independent PVCs across different nodes, and CockroachDB’s Raft layer, not the storage layer, provides the cross-replica durability guarantee. When a node’s pod reschedules and its existing PVC reattaches intact, every range replica on that node rejoins its Raft group from its own on-disk state and only needs to catch up on log entries committed during the outage. If the PVC’s data is genuinely lost, the rebalancer instead adds a brand-new replica for each affected range, streamed as a full snapshot from that range’s current leader.

Does NVMe/TCP storage avoid the full range up-replication in CockroachDB?

Yes, in the common case. A full range snapshot is needed when a replica’s data is genuinely lost or permanently unreachable. With disaggregated NVMe/TCP storage, a failed pod’s existing Pebble store and Raft log volume reattach to a replacement pod on any node in the cluster, so the range’s data is not lost, it is just briefly unavailable while the pod restarts. The replica rejoins after replaying the small number of log entries committed since its last snapshot, which is materially faster than streaming the range’s entire current state.

How does simplyblock compare to local NVMe for CockroachDB on Kubernetes?

Local NVMe gives the lowest absolute fsync latency for Pebble’s write-ahead entries and Raft log, at the cost of binding every pod to a specific machine. Simplyblock’s disaggregated NVMe/TCP storage delivers throughput comparable to local NVMe for Raft-commit write patterns, typically in the 100 to 400 microsecond latency range on a 25 GbE fabric, while allowing pods to reschedule freely and applying per-volume QoS so one node’s rebalance traffic cannot degrade Raft commit latency for ranges on another volume. For clusters running the CockroachDB Kubernetes Operator in production, the reduced re-replication time after a node failure offsets the small latency delta for most OLTP workloads.

Does this architecture work with the CockroachDB Kubernetes Operator and Helm chart?

Yes. Both the Operator’s spec.dataStore.pvc field and the Helm chart’s statefulset.persistentVolume values ultimately configure the same volumeClaimTemplates entry on the generated StatefulSet, so pointing either one at a disaggregated NVMe/TCP StorageClass changes nothing about how CockroachDB itself is deployed or upgraded. The difference shows up only at node-failure and decommission time, where volume reattachment replaces the full range-snapshot path the rebalancer would otherwise take.