Rob Pankow 
The Kafka storage guide, the RabbitMQ storage guide, and the Pulsar storage guide each cover a different point on the messaging-system spectrum: Kafka and RabbitMQ keep broker and log on the same process, Pulsar splits them into a stateless broker and a separate BookKeeper storage tier. NATS JetStream sits in between. It embeds its own optimized Raft implementation directly in the NATS server process, deliberately merging the consensus data plane with message replication so a replicated stream’s write path is a single round trip instead of two. That efficiency is also exactly why JetStream storage on Kubernetes needs closer attention than it usually gets: one NATS server pod still persists two distinct things to disk for every replicated stream, and a generic PersistentVolumeClaim sized only for capacity treats them as one undifferentiated write.
Why NATS’s Optimized Raft Puts Two Write Paths on One JetStream Node
JetStream organizes replicated work into Raft groups: a meta group for cluster-wide stream and consumer placement, and one NATS Raft Group (NRG) per replicated stream and per durable consumer. NATS documents this design as an “optimized RAFT” precisely because it folds the data plane that would normally replicate messages separately into the same messages Raft uses to reach consensus, so an R3 stream commits a message to its quorum in one write instead of a Raft proposal plus a second application write.
That optimization still leaves two persistent artifacts on every node that hosts a stream replica. The first is the Raft group’s own log: the append-only record of proposed entries a node needs to participate in consensus and recover its term and index after a restart without asking every other peer. The second is the stream’s file store, the message blocks JetStream keeps on disk so consumers can replay, redeliver, or catch up on a durable subscription. Both live under the same store_dir on the same node by default, and both are written on the same critical path as every publish: a slow, contended disk shows up as consensus and publish latency at the same time, not just as one or the other.
The Raft Log vs the Message Store: Two I/O Profiles, One Volume
The two write paths JetStream puts on a single node have close to opposite storage requirements, the same shape of split already documented for etcd’s Raft commit path and for Pulsar’s BookKeeper journal:
The Raft group log is small, sequential, and fsync-gated. Every proposed entry for a stream’s NRG must be durable before the node can vote to commit it, so this write directly gates how fast a replicated (R3) stream can accept publishes. NATS’s own guidance to run JetStream’s data directory on fast, low-latency storage, and to avoid network file systems and other high-latency backends, exists because of this write, not because of stream volume.
The message store (the stream’s on-disk blocks plus their index) is throughput- and capacity-oriented. Normal operation is sequential appends, but a consumer replaying a backlog, a new durable consumer starting from the beginning of a stream, or a redelivery after a nak all turn into random reads against the same file store. This tier tolerates far more latency variance than the Raft log without breaking a producer’s publish-ack SLA.
| JetStream storage tier | I/O pattern | Latency sensitivity | Best-fit backend |
|---|---|---|---|
| Raft group log (per replicated stream/consumer) | Small, sequential, fsync-per-entry | Very high, gates publish-ack latency directly | Disaggregated NVMe/TCP or NVMe/RoCE, provisioned for consistent low-latency fsync |
| Message store (stream blocks + index) | Sequential writes, random reads on replay/redelivery | Medium, tolerates more variance than the Raft log | Same NVMe/TCP tier, sized for throughput and backlog read capacity |
| In-memory streams (no file storage) | RAM-resident, no disk I/O | N/A, lost on restart by design | Not applicable, use file storage for anything that must survive a pod restart |
Table 1: NATS JetStream storage tier characteristics and the backend each tier fits best.
A generic StorageClass that provisions one persistent volume per JetStream pod, sized only for total message retention, forces the Raft log and the message store to compete for the same disk’s I/O budget. On a busy stream that shows up as publish latency and even quorum instability that has nothing to do with subscriber load, and everything to do with a replay job or a large redelivery batch hitting the same spindle or network volume the Raft log needs for its next fsync.
A contended Raft log turns into publish-ack latency and, at the extreme, quorum instability for every producer on that stream. Talk to a storage architect about giving JetStream’s Raft group log and message store tiers correctly sized, low-latency NVMe/TCP volumes instead of one generic PVC per pod. Talk to a storage architect
How Disaggregated NVMe/TCP Fixes JetStream’s Two Write Paths on Kubernetes
Running JetStream pods with persistent volume claims backed by disaggregated NVMe/TCP (or NVMe/RoCE on RDMA-capable fabrics) closes both gaps a single generic volume leaves open:
Each write path gets its own profile from one pool. Because JetStream’s data directory sits on a shared NVMe fabric rather than one node’s local disk, the deployment can provision the Raft log path for consistent low-latency fsync while the message-store path is sized for backlog throughput, without a replay job on one stream stealing I/O headroom from another stream’s consensus writes on the same node. This is the same underlying argument already covered for Pulsar’s journal/ledger split and for ClickHouse’s background merges: give the latency-critical write its own headroom instead of sharing one undifferentiated volume.
Node loss becomes a volume reattachment, not a full stream catch-up. When a JetStream replica falls too far behind or is lost outright today, the cluster’s recovery path is a Raft snapshot install: the leader streams the current stream state to the lagging peer so it can rejoin the replica set, an operation whose cost scales with how much data that stream is holding. When a stream’s file store lives on the fabric instead of node-local disk, a rescheduled pod reattaches the same PVCs and resumes participating in its Raft groups from where it left off, without triggering a full snapshot transfer for data that never actually left the fabric.
JetStream deployments also gain the same snapshot pattern already covered for CSI snapshot architecture: a VolumeSnapshot of a stream’s message-store volume is a point-in-time, storage-side capture of its current blocks and index, restorable in seconds as a faster complement to JetStream’s own stream backup and restore mechanism, which has to walk and re-stream every message in the backup set. A VolumeSnapshotClass tuned to the message-store volume, with the Raft log volume excluded since it is only meaningful in the context of a live quorum, mirrors the tiered snapshot pattern already used for CockroachDB and etcd elsewhere in this series.
The same architecture supports NATS clusters running hyperconverged, with JetStream pods and their volumes co-located on shared nodes, or fully disaggregated, with the fabric behind a dedicated storage layer, on vSphere, bare-metal Kubernetes, or OpenShift. Simplyblock licenses per usable storage TB per year with no per-core fee, so scaling a JetStream cluster’s replica count or adding streams to absorb more publish throughput is a capacity decision, not an automatic per-node cost multiplier.
Questions and Answers
Why does NATS JetStream need two separate storage tiers instead of one volume per pod? Every replicated stream in JetStream writes to a Raft group log (fsync-gated, gates publish-ack latency) and to a message store (throughput-oriented, serves replay and redelivery reads). Provisioning both from one generic persistent volume sized for retention capacity lets message-store I/O compete with the Raft log’s fsync path, which surfaces as publish-ack latency spikes and, under enough contention, quorum instability unrelated to producer load.
What happens when a JetStream replica is lost on generic Kubernetes storage? The cluster recovers a lagging or lost replica through a Raft snapshot install, streaming the current stream state from the leader so the peer can rejoin, an operation whose cost scales with how much data that stream holds. A persistent volume claim backed by simplyblock’s NVMe/TCP storage reattaches to a rescheduled pod in seconds instead, avoiding a full snapshot transfer in most node-failure scenarios.
Does JetStream’s “optimized RAFT” remove the need to think about storage architecture? No. Merging the consensus data plane with message replication makes the write path more efficient, but it does not remove the two distinct persistent artifacts, the Raft group log and the message store, that still land on the same node’s disk. A deployment still needs to provision for the Raft log’s fsync-latency requirement separately from the message store’s throughput and capacity requirement.
How does simplyblock fit into a NATS JetStream deployment on Kubernetes? Simplyblock provides NVMe/TCP (or NVMe/RoCE) block storage as persistent volumes sized for JetStream’s Raft log and message-store I/O, instead of one undifferentiated volume per pod. Simplyblock’s CSI driver adds fast, storage-side snapshots of message-store volumes for point-in-time recovery, and the same architecture supports both hyperconverged and disaggregated deployment models.
How should I size storage for JetStream on Kubernetes? Size the volume serving the Raft group log for consistent, low-latency fsync throughput at expected publish rate, and size the message-store volume separately for retention capacity plus replay and redelivery read throughput. Simplyblock’s pricing is per usable TB per year with no per-core fee, so scaling stream replica counts or adding streams for higher throughput is a capacity decision, not an automatic per-node cost increase.
