Kafka on Kubernetes: Persistent Storage Architecture and High Availability

Apache Kafka is the most widely deployed distributed message broker in the world, and running it on Kubernetes is standard practice for teams that want a single platform for both stateless microservices and stateful workloads. Unlike stateless services, each Kafka broker maintains a local log directory where it physically stores topic partition data as sequential log segments. That log directory is the source of both Kafka’s durability and its most common Kubernetes operational problem.

When Kafka brokers run on local storage, each pod becomes pinned to the specific node where its disk lives. Drain a node for maintenance, upgrade a kernel, or replace failed hardware and the broker pod cannot reschedule. It stays offline until the node returns. During that window, partitions that broker led must fail over to replica leaders, but the failed pod’s data is inaccessible for recovery until the node is back. This is not a Kafka design flaw: it is the expected consequence of using node-local storage in an environment built for pod mobility.

This guide covers the persistent volume architecture for Kafka on Kubernetes, StorageClass configuration for the two most widely used Kafka operators, and how disaggregated storage changes the operational model for broker recovery and rolling upgrades.

Why Kafka Storage Is Different on Kubernetes

Kafka’s durability model is built on sequential log writes. Each broker maintains a log directory containing log segments for every partition it leads. A log segment is a pair of files: a .log file containing raw message data written sequentially, and an .index file for offset lookups. Producers append to the active segment’s .log file, and Kafka rolls to a new segment when the active one reaches the configured log.segment.bytes or log.roll.hours limit.

By default, Kafka delegates fsync to the OS (log.flush.interval.messages and log.flush.interval.ms are both unset). This delivers maximum throughput: the OS page cache absorbs write bursts, and sequential log writes reach disk through normal kernel writeback. Durability comes from partition replication, not from per-message fsync. When a broker fails before the OS flushes its page cache, any unflushed writes are lost from that broker’s perspective, but the replica leader on another broker already acknowledged those writes to producers using acks=all.

The fsync behavior matters for storage sizing and StorageClass design for two reasons:

1. Log compaction read pressure. Kafka’s log cleaner runs as a background thread on each broker. It reads old log segments to rewrite compacted versions, creating a concurrent sequential read-and-write load on the same volume used for active producer writes. On local NVMe, the cleaner and the active log writer share a single device’s bandwidth. With a shared NVMe/TCP pool, that I/O is isolated at the volume level.

2. Broker startup replay. When a Kafka broker restarts after a failure or pod eviction, it reads through the log segments for each partition it owns to determine the latest committed offset and rebuild its in-memory metadata. The larger the log retention and the more partitions per broker, the longer this replay takes. Fast sequential read throughput from the storage layer directly controls broker recovery time.

On Kubernetes, these I/O characteristics combine with a scheduling constraint. Kafka runs as a Kubernetes StatefulSet with one persistent volume claim per broker pod. If that storage is local NVMe (hostPath or local PersistentVolumes), the pod is permanently bound to the node where the disk lives. When that node drains, the pod cannot reschedule and the broker goes offline.

Persistent Volume Architecture for Kafka StatefulSets

Kafka brokers require dedicated persistent volumes provisioned through the StatefulSet’s volumeClaimTemplates field. Each broker pod gets its own PVC automatically. Kafka does not share partition data between brokers: each broker owns the log files for the partitions it leads, and replicas maintain independent copies synchronized at the message level.

Table 1: Persistent volume configuration recommendations for Kafka StatefulSets on Kubernetes.

A StorageClass for Kafka using the simplyblock CSI driver:

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: kafka-nvme
5
provisioner: csi.simplyblock.io
6
parameters:
7
  replication: "2"
8
  compression: "false"
9
  qos_iops_per_gb: "50"
10
reclaimPolicy: Retain
11
volumeBindingMode: WaitForFirstConsumer
12
allowVolumeExpansion: true

The qos_iops_per_gb parameter limits the maximum IOPS per gigabyte of provisioned capacity. This prevents the Kafka log cleaner from saturating the shared NVMe pool during heavy compaction cycles and starving producer I/O on other volumes or other brokers in the cluster.

The volumeClaimTemplates block in the StatefulSet spec references this StorageClass directly. For a three-broker cluster with 200 GiB of log retention per broker:

1
volumeClaimTemplates:
2
  - metadata:
3
      name: data
4
    spec:
5
      accessModes: ["ReadWriteOnce"]
6
      storageClassName: kafka-nvme
7
      resources:
8
        requests:
9
          storage: 200Gi

Each broker pod receives a PVC named data-<statefulset-name>-<ordinal>. The WaitForFirstConsumer binding mode ensures the volume is provisioned in the same availability zone as the pod, which prevents the zone mismatch failure where a PVC binds in one zone but the pod schedules in another.

Running Kafka on Kubernetes and losing brokers to node drains or upgrades? simplyblock’s disaggregated NVMe/TCP storage lets Kafka broker pods reschedule to any node, with the same PVC reattaching in seconds rather than waiting for the original node to return. Talk to a storage architect

Configuring Kafka Operators for Persistent Storage

Most production Kafka deployments on Kubernetes use an operator to manage cluster topology, rolling upgrades, and partition reassignment. The two most widely deployed operators are Strimzi and Confluent for Kubernetes, and both wrap the StatefulSet volumeClaimTemplates mechanism with operator-specific configuration fields.

Strimzi Operator

Strimzi is the leading open-source Kafka operator for Kubernetes. Storage configuration lives in the spec.kafka.storage field of the Kafka custom resource. Use type: persistent-claim with deleteClaim: false to prevent PVC deletion when the cluster CR is deleted:

1
apiVersion: kafka.strimzi.io/v1beta2
2
kind: Kafka
3
metadata:
4
  name: kafka-cluster
5
spec:
6
  kafka:
7
    replicas: 3
8
    storage:
9
      type: persistent-claim
10
      size: 200Gi
11
      class: kafka-nvme
12
      deleteClaim: false
13
  zookeeper:
14
    replicas: 3
15
    storage:
16
      type: persistent-claim
17
      size: 10Gi
18
      class: kafka-nvme
19
      deleteClaim: false

Strimzi also supports JBOD (Just a Bunch of Disks) storage via type: jbod, which provisions multiple volumes per broker. JBOD is the recommended configuration for high-throughput clusters: isolating the active write log from log compaction reads eliminates the compaction I/O contention entirely. With disaggregated storage, each JBOD volume is an independent NVMe namespace provisioned from the same shared pool.

For KRaft clusters (ZooKeeper-free, available in Strimzi 0.40+ with Kafka 3.7+), the controller nodes use a spec.kafka.roles field and the same storage configuration applies to both broker and controller roles.

Confluent for Kubernetes (CFK)

Confluent for Kubernetes uses a simplified storage API. Specify the StorageClass name and volume capacity directly on the Kafka component resource:

1
apiVersion: platform.confluent.io/v1beta1
2
kind: Kafka
3
metadata:
4
  name: kafka
5
spec:
6
  replicas: 3
7
  storageClass:
8
    name: kafka-nvme
9
  dataVolumeCapacity: 200Gi

CFK generates the StatefulSet and volumeClaimTemplates internally. The operator handles rolling upgrades, partition reassignment, and rack awareness automatically.

Table 2: Kafka operator storage configuration comparison on Kubernetes.

High Availability: Partition Replication and Broker Recovery

Kafka’s built-in high availability relies on partition replication. Each partition has one leader broker and one or more replica brokers. The replication factor (RF) and min.insync.replicas (ISR) configuration determine how many broker failures a cluster can sustain while still accepting producer writes with acks=all.

The standard production configuration is RF=3 with min.insync.replicas=2. A single broker failure triggers leader election for the partitions that broker led, but the remaining two in-sync replicas keep every topic available for producers and consumers. The cluster continues operating in a degraded state: two ISR replicas instead of three, but writes still commit.

The storage layer determines how quickly the failed broker can return to the ISR:

• Local NVMe (hostPath or local PV): The failed broker pod cannot reschedule until the original node returns. During a planned maintenance drain, the broker is offline for the entire maintenance window. During an unplanned node failure, the broker is offline until the node is repaired or replaced. A second broker failure before the first recovers pushes the cluster below min.insync.replicas and blocks all acks=all producers.

• Cloud block storage (gp3, Premium SSD): The pod can reschedule, but the PVC reattaches only within the same availability zone. In a multi-AZ node pool, the pod may schedule in a different zone from the volume, causing a failed attach. Volume reattachment, when it succeeds, takes 20 to 60 seconds. Cross-zone volume access adds network latency to every broker I/O operation.

• Disaggregated NVMe/TCP (simplyblock): The failed broker pod reschedules to any node in the cluster. The CSI driver reattaches the same PVC to the new node in seconds. The broker reads through its log segments (startup replay), rebuilds metadata, and begins catching up on any missed messages before rejoining the ISR. The time to ISR recovery is controlled by the volume of missed writes, not by hardware repair timelines.

Table 3: Kafka storage model comparison for high availability on Kubernetes.

Rolling upgrades illustrate the operational difference most clearly. Strimzi and CFK perform rolling upgrades by draining one broker pod at a time. With local NVMe, each broker drain requires waiting for partitions to rebalance off that broker before the node can be drained, because the pod cannot restart on a different node. With disaggregated NVMe/TCP, the pod can be terminated and rescheduled on any other node immediately. The log reattachment replaces the rebalance wait. For a three-broker cluster, this changes rolling upgrade time from tens of minutes to the sum of three broker restart-and-catch-up cycles.

Snapshot-based point-in-time recovery is a further operational advantage. simplyblock’s CSI external snapshotter integration allows consistent storage snapshots of Kafka log volumes without pausing producer writes. Restoring from a snapshot brings a broker back to a known-good state significantly faster than waiting for full partition resync from other replicas, particularly for large partitions with long retention periods.

For teams using the CSI driver model discussed in this series on stateful Kubernetes workloads, Kafka illustrates the clearest case where the storage layer directly controls application availability, not just data durability. See also the earlier entries in this series covering MySQL, MongoDB, and Redis on Kubernetes.

Questions and Answers

What persistent storage settings work best for Kafka on Kubernetes?

Use volumeBindingMode: WaitForFirstConsumer to prevent zone mismatches, reclaimPolicy: Retain to prevent log data loss on PVC deletion, and allowVolumeExpansion: true for online capacity growth. For IOPS, plan for 10,000 to 50,000 IOPS per broker depending on topic throughput and the number of partitions. Log compaction adds concurrent read pressure on top of the normal write load, so provision headroom beyond peak producer IOPS alone. If using a shared NVMe pool, configure per-volume QoS limits via StorageClass parameters to prevent compaction from starving producer I/O on other volumes.

How does Kafka partition replication interact with Kubernetes persistent volumes?

Kafka replication happens at the message level, broker to broker, independently of the storage layer. Each broker maintains its own copy of the partition data in its log directory, synchronized by the Kafka replication protocol. The Kubernetes PVC holds that broker’s copy of the log. When a broker fails and its pod reschedules, the CSI driver reattaches the existing PVC (containing the broker’s log data) to the new node, and the broker replays any messages it missed during the outage by fetching from the current leader. The storage layer provides durability for each broker’s local copy; Kafka replication provides cluster-level durability across brokers.

Can Kafka broker pods reschedule freely with NVMe/TCP persistent storage?

Yes. NVMe/TCP volumes are network-attached and not bound to any physical node. When a Kafka broker pod is evicted (node drain, node failure, or operator rolling upgrade), the CSI driver re-attaches the persistent volume to the new node. The broker performs log startup replay and then begins fetching missed messages from the current partition leader before rejoining the in-sync replica set. For clusters with RF=3 and min.insync.replicas=2, the cluster continues serving producers and consumers during the single-broker recovery window.

How does simplyblock compare to local NVMe for Kafka on Kubernetes?

Local NVMe delivers the lowest absolute latency for sequential log writes, but at the cost of pod scheduling flexibility. A broker on local NVMe cannot reschedule when its node drains. simplyblock’s disaggregated NVMe/TCP storage delivers sequential write throughput comparable to local NVMe for log segment workloads, with latency in the 100 to 400 microsecond range on 25 GbE fabric. The scheduling flexibility, storage-layer replication independent of Kafka replication, and per-volume QoS control reduce the operational burden significantly for teams running Kafka Strimzi or Confluent Operator in production.

Does this architecture work with Strimzi’s JBOD storage configuration?

Yes. Strimzi JBOD provisioning creates multiple PVCs per broker, one per configured volume. Each JBOD volume becomes an independent NVMe namespace provisioned from the simplyblock pool. Kafka distributes partitions across the JBOD volumes according to the broker’s log.dirs configuration. Isolating log compaction reads to a dedicated JBOD volume eliminates the read/write contention on the active write volume. When a broker pod reschedules, the CSI driver reattaches all JBOD volumes to the new node before the broker starts, in the same way a single-volume configuration operates.