RabbitMQ on Kubernetes: Persistent Storage Architecture and High Availability

RabbitMQ is a message broker built around durable queues, publisher confirms, and, since version 3.8, quorum queues that replicate via the Raft consensus protocol. Running it on Kubernetes alongside other stateful workloads is now standard practice, but RabbitMQ’s write path has requirements most CSI drivers were not designed around. Every durable publish waits on an fsync before the broker returns a confirm, and every quorum queue replica maintains its own Raft log and snapshot files that must stay consistent with the leader’s.

The bigger operational cost shows up when a node fails. A quorum queue tolerates the loss of a minority of its replicas without losing messages, but the replacement replica does not start from zero: it joins as a new Raft member and must receive a full snapshot of the queue’s current state plus every log entry since, streamed from the leader. For a queue with a large backlog, that catch-up can take minutes to hours and adds read load to the leader precisely when the cluster is already down a member.

This guide covers the persistent volume architecture for RabbitMQ on Kubernetes, configuration for the RabbitMQ Cluster Operator and Messaging Topology Operator, and how disaggregated storage changes the node-failure recovery model from a full Raft resync to a volume reattachment.

Why RabbitMQ Storage Is Different on Kubernetes

RabbitMQ’s storage layout depends on the queue type, and the two dominant types drive very different I/O patterns:

1. Classic durable queues. Messages persist to a per-node message store (msg_store_persistent) plus a queue index that tracks message location and delivery state. A durable publish with publisher confirms enabled does not return until the message is fsynced to the message store, making this path directly latency-sensitive to storage write performance.

2. Quorum queues. Introduced in RabbitMQ 3.8 and the recommended replacement for classic mirrored queues (removed entirely in RabbitMQ 4.0), quorum queues replicate through the Raft consensus algorithm. Each replica writes its own Raft log entries and periodic snapshots to disk, and a replica’s on-disk state must be internally consistent for it to safely rejoin the queue’s replica set.

RabbitMQ documentation recommends fast, low-latency block storage for both paths specifically because fsync latency on the message store or Raft log directly gates publisher-confirm latency. On a Kubernetes persistent volume with high write latency, a burst of durable publishes or quorum-queue writes stalls confirms across every queue sharing that volume.

The Kubernetes-specific issue is node identity. RabbitMQ nodes are addressed by a stable name (rabbitmq-server-0, rabbitmq-server-1, and so on, from the StatefulSet), and quorum queue replicas are tied to that node identity. When a node’s local disk is lost, or a pod on hostPath or local-PV storage cannot reschedule because its data lives on one specific machine, the cluster must remove the old replica and add a brand-new one. The new replica receives the queue’s full current snapshot and replays every subsequent log entry from the leader, an amount of data proportional to queue depth and message size, not to the length of the outage.

Persistent Volume Architecture for RabbitMQ Clusters

RabbitMQ runs on Kubernetes as a StatefulSet managed by the RabbitMQ Cluster Operator, with each pod requesting its own PersistentVolumeClaim. Each node’s message store, queue index, and Raft logs live entirely on its own volume; nothing is shared between pods at the storage layer, only at the RabbitMQ replication layer.

Table 1: Persistent volume configuration recommendations for RabbitMQ on Kubernetes.

A StorageClass for RabbitMQ using the simplyblock CSI driver:

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: rabbitmq-nvme
5
provisioner: csi.simplyblock.io
6
parameters:
7
  replication: "2"
8
  compression: "false"
9
  qos_iops_per_gb: "30"
10
reclaimPolicy: Retain
11
volumeBindingMode: WaitForFirstConsumer
12
allowVolumeExpansion: true

compression is left off because message payloads (JSON, protobuf, binary blobs) are frequently already compressed by the publishing application, and RabbitMQ’s own message store does not benefit from a second compression pass. The qos_iops_per_gb ceiling exists so that one node’s Raft snapshot transfer, which is throughput-heavy and bursty, cannot starve the fsync path a neighboring node’s durable publishers depend on across the same shared NVMe pool.

Watching a quorum queue replica resync its entire backlog after a node failure? Simplyblock’s disaggregated NVMe/TCP storage lets a replacement RabbitMQ pod reattach its predecessor’s volume, message store and Raft log intact, in seconds, skipping the full snapshot transfer entirely. Talk to a storage architect

Configuring the RabbitMQ Cluster Operator for Persistent Storage

Two operator-managed CRDs cover production RabbitMQ deployments on Kubernetes: the RabbitMQ Cluster Operator, which owns the cluster’s StatefulSet and PVCs, and the Messaging Topology Operator, which manages queues, policies, and users declaratively.

RabbitMQ Cluster Operator (RabbitmqCluster CRD)

Storage is configured under spec.persistence on the RabbitmqCluster custom resource:

1
apiVersion: rabbitmq.com/v1beta1
2
kind: RabbitmqCluster
3
metadata:
4
  name: rabbitmq-cluster
5
spec:
6
  replicas: 3
7
  persistence:
8
    storageClassName: rabbitmq-nvme
9
    storage: 100Gi
10
  resources:
11
    requests:
12
      cpu: 2
13
      memory: 4Gi

The operator generates the underlying StatefulSet, one PVC per replica via volumeClaimTemplates, and handles rolling restarts one pod at a time so a quorum queue never loses more than one replica during a planned upgrade.

Messaging Topology Operator (Queue and Policy CRDs)

The Messaging Topology Operator manages queues, exchanges, bindings, and policies as Kubernetes custom resources instead of imperative rabbitmqctl commands or a definitions.json import:

1
apiVersion: rabbitmq.com/v1beta1
2
kind: Queue
3
metadata:
4
  name: orders-queue
5
spec:
6
  name: orders
7
  vhost: "/"
8
  type: quorum
9
  durable: true
10
  rabbitmqClusterReference:
11
    name: rabbitmq-cluster
12
  arguments:
13
    x-quorum-initial-group-size: 3

x-quorum-initial-group-size: 3 fixes the replica count for this queue at creation. Declaring queue type and replica count in a CR, rather than in application connection code, keeps the durability guarantee visible in the cluster’s GitOps state and reviewable alongside the storage configuration.

Table 2: RabbitMQ operator scope and configuration comparison on Kubernetes.

High Availability: Quorum Queues and Node Recovery

RabbitMQ’s availability model for quorum queues comes from the Raft replica set, typically 3 or 5 members per queue, not from the underlying block storage. A queue with x-quorum-initial-group-size: 3 tolerates one replica being unavailable while the remaining two still form a Raft majority for writes and reads.

What the storage layer controls is how quickly a lost replica returns to full standing, and that gap is largest at node failure, not during normal operation:

• Local NVMe (hostPath or local PV): When the node’s disk is lost, or the pod cannot reschedule because it is bound to that specific machine, the Raft layer removes the old member and adds a new one. The new replica receives a full snapshot of current queue state and replays every log entry since, an amount of data proportional to queue depth. For a queue with a large, slow-draining backlog this can take minutes to hours, during which the replica set runs at reduced redundancy and the leader absorbs extra read load serving the snapshot transfer.

• Cloud block storage (gp3, Premium SSD): The pod can reschedule and reattach its existing volume, avoiding a full resync, but only within the same availability zone. In a multi-AZ node pool, a pod that reschedules to a different zone than its volume fails to attach, forcing the full-snapshot path anyway. A successful same-zone reattach still adds pod-startup latency before the replica rejoins.

• Disaggregated NVMe/TCP (simplyblock): The replacement pod reattaches the failed node’s existing message store and Raft log volumes on any node in the cluster within seconds, with committed log entries and snapshots intact. The replica rejoins the Raft group directly from its own on-disk state, catching up on only the handful of log entries committed during the brief restart window, no full snapshot transfer required.

Table 3: RabbitMQ node-failure recovery comparison by storage model.

Classic durable queues, which RabbitMQ 4.0 no longer mirrors across nodes, depend on this same recovery model even more directly: a classic queue’s messages exist on exactly one node’s message store, so if that node’s disk is genuinely lost, the messages are gone regardless of Raft. Disaggregated storage does not add mirroring to classic queues, but it does mean the node’s disk is never actually lost when the pod reschedules, which removes the most common cause of that data-loss scenario in practice.

For teams that have followed the CSI driver discussion earlier in this series, RabbitMQ is another case where a storage-layer capability, volume reattachment independent of node identity, eliminates an entire class of broker-level recovery work. See also the earlier entries covering MySQL, MongoDB, Redis, Kafka, and Cassandra on Kubernetes.

Questions and Answers

What persistent storage settings work best for RabbitMQ on Kubernetes?

Use volumeBindingMode: WaitForFirstConsumer so the volume provisions in the same zone as the scheduled pod, reclaimPolicy: Retain to prevent message and Raft log loss on accidental PVC deletion, and allowVolumeExpansion: true for online capacity growth. Plan for 5,000 to 20,000 IOPS per node depending on publish throughput and the number of quorum queues per broker, since both the message store fsync path and Raft log writes compete for the same volume unless you separate high-traffic queues across StorageClasses with independent QoS.

How do quorum queues interact with Kubernetes persistent volumes?

Each quorum queue replica maintains its own Raft log and periodic snapshots on the node’s PVC, independent of RabbitMQ’s cluster-wide replication guarantee. When a node’s pod reschedules and its existing PVC reattaches intact, the replica rejoins the Raft group using its own on-disk state and only needs to catch up on log entries committed during the outage. If the PVC’s data is genuinely lost, the Raft layer instead adds a brand-new replica that must receive the queue’s full current snapshot from the leader, a data transfer proportional to queue depth.

Does NVMe/TCP storage avoid the full Raft snapshot resync in RabbitMQ?

Yes, in the common case. A full snapshot transfer is needed when a replica’s data is genuinely lost or unreachable. With disaggregated NVMe/TCP storage, a failed pod’s existing message store and Raft log volumes reattach to a replacement pod on any node in the cluster, so the replica’s data is not lost, it is just temporarily unavailable while the pod restarts. The replica rejoins after replaying the small number of log entries committed since its last checkpoint, which is materially faster than transferring the queue’s entire current state.

How does simplyblock compare to local NVMe for RabbitMQ on Kubernetes?

Local NVMe gives the lowest absolute fsync latency for the message store and Raft log, at the cost of binding every pod to a specific machine. Simplyblock’s disaggregated NVMe/TCP storage delivers throughput comparable to local NVMe for durable-publish and Raft log write patterns, typically in the 100 to 400 microsecond latency range on 25 GbE fabric, while allowing pods to reschedule freely and applying per-volume QoS to keep one queue’s snapshot transfer from degrading another queue’s publisher-confirm latency. For teams running the RabbitMQ Cluster Operator in production, the reduced replica-resync time after a node failure offsets the small latency delta for most workloads.

Does this architecture work with the Messaging Topology Operator?

Yes. The Messaging Topology Operator manages queue and policy definitions as CRDs layered on top of the RabbitMQ Cluster Operator’s StatefulSet and PVCs; it does not change how storage is provisioned. Setting x-quorum-initial-group-size in a Queue CR determines how many replicas a given queue’s Raft group has, and pairing that with disaggregated NVMe/TCP storage means each of those replicas recovers from a node failure by volume reattachment rather than a full snapshot transfer, regardless of how the queue topology itself is declared.