Rob Pankow 
The Kafka storage guide and the RabbitMQ storage guide both cover messaging systems where the broker and its log or queue storage live on the same process. Apache Pulsar is different by design: the broker that serves producers and consumers is stateless, and all durable data lives in a separate storage layer called Apache BookKeeper, run by processes called bookies. That split is already the closest any mainstream messaging system gets to a genuinely disaggregated compute/storage architecture, which makes Pulsar on Kubernetes an unusually direct match for simplyblock’s own storage model, and an unusually easy architecture to get wrong if the bookie’s storage is provisioned as one generic volume.
Why Pulsar’s BookKeeper Layer Already Splits Compute from Storage
In Pulsar, brokers handle topic ownership, routing, and the pub/sub protocol, but they hold no durable state of their own. Every message a broker accepts is immediately written to BookKeeper as an entry in a ledger, and every ledger is striped across an ensemble of bookies according to three settings: ensemble size (how many bookies hold fragments of the ledger), write quorum (how many bookies each entry is written to), and ack quorum (how many acknowledgments the broker waits for before confirming the write to the producer). Losing a broker is inexpensive: Pulsar reassigns topic ownership to another broker, which reads the ledger metadata from BookKeeper and keeps serving. Losing a bookie is a storage-layer event, and that is where most Kubernetes deployments run into trouble, because a single bookie is not one I/O workload, it is two very different ones sharing whatever volume it was given.
Journal vs Ledger: Two I/O Profiles, One Bookie Pod
Every BookKeeper bookie writes each entry to two places before it is durable, and the two writes have almost opposite storage requirements:
The journal is a write-ahead log. Every entry is appended and fsynced before the bookie acknowledges the write, so journal latency directly gates producer-visible publish latency. This is small, continuous, fsync-heavy sequential I/O, the same shape of requirement documented for etcd’s Raft commit path and for any write-ahead-log-backed system: consistent low tail latency matters far more than raw capacity.
Ledger storage (BookKeeper’s entry log plus its index files) holds the same entries in a form optimized for later reads: sequential writes during normal operation, but random reads whenever a consumer replays a backlog or a broker needs to serve data that has aged out of the bookie’s read cache. This tier is throughput- and capacity-oriented, and it tolerates more latency variance than the journal without breaking producer-facing SLAs.
BookKeeper’s own deployment guidance already recommends separate disks for journalDirectories and ledgerDirectories for exactly this reason. A generic StorageClass provisioning both from the same pool of persistent volumes, sized for capacity rather than for the journal’s fsync latency requirement, lets ledger-storage throughput contend with journal writes and shows up as publish-latency spikes that have nothing to do with producer load.
| BookKeeper storage tier | I/O pattern | Latency sensitivity | Best-fit backend |
|---|---|---|---|
| Journal (write-ahead log) | Small, continuous, fsync-per-entry | Very high, gates publish latency directly | Disaggregated NVMe/TCP or NVMe/RoCE, provisioned for consistent low-latency fsync |
| Ledger storage (entry log + index) | Sequential writes, random reads on replay | Medium, tolerates more variance than the journal | Same NVMe/TCP tier, sized for throughput and backlog read capacity |
| Cold / offloaded ledgers | Large sequential reads, infrequent | Low | S3-compatible object storage, via Pulsar’s tiered storage offload |
Table 1: BookKeeper storage tier characteristics and the backend each tier fits best.
Pulsar’s own tiered storage offload already handles the cold tier correctly, moving ledgers that have aged past a topic’s retention policy out to S3-compatible storage. What it does not solve is the hot path: the journal and the active ledger tier that every producer and consumer touches still need to be provisioned for their own I/O shape, not just for total capacity.
A slow journal turns into slow publish acks for every producer on that bookie. Talk to a storage architect about giving BookKeeper’s journal and ledger tiers separate, correctly-sized NVMe/TCP volumes instead of one generic PVC per bookie. Talk to a storage architect
How Disaggregated NVMe/TCP Maps Onto BookKeeper’s Journal and Ledger Tiers
Running bookie pods with persistent volume claims backed by disaggregated NVMe/TCP (or NVMe/RoCE on RDMA-capable fabrics) closes the two gaps a single generic volume leaves open:
Each tier gets its own latency and throughput profile from one pool. Because the journal and ledger volumes both sit on a shared NVMe fabric rather than one node’s local disk, the journal volume can be provisioned for consistent low-latency fsync while the ledger volume is provisioned for throughput, without either tier stealing I/O from a neighboring bookie’s local disk contention. This is the same underlying argument covered for ClickHouse’s MergeTree background merges and OpenSearch’s segment merges: give the latency-critical tier its own headroom instead of sharing one undifferentiated volume.
Bookie replacement becomes a volume reattachment, not a full ledger re-replication. When a bookie is lost today, BookKeeper’s auto-recovery process re-replicates every under-replicated ledger fragment that bookie held, streaming data from the surviving members of each ledger’s ensemble, an operation whose cost scales with how much data that bookie was holding. When the bookie’s storage lives on the fabric instead of node-local disk, a rescheduled bookie pod reattaches the same journal and ledger PVCs and resumes serving from where it left off, without triggering ensemble-wide re-replication for data that never actually left the fabric.
Pulsar deployments on Kubernetes also gain the same snapshot pattern already covered for CSI snapshot architecture: a VolumeSnapshot of a bookie’s ledger volume gives a point-in-time, storage-side capture of its current entry log and index state, restorable in seconds, as a faster complement to rebuilding a bookie’s local state purely from ensemble re-replication or from offloaded ledgers in object storage. A VolumeSnapshotClass per tier, frequent snapshots on the ledger volume and none needed on the journal (which is inherently ephemeral once entries are durable in the ledger), mirrors the tiered snapshot pattern already used for CockroachDB and etcd elsewhere in this series.
The same architecture supports Pulsar clusters running hyperconverged, with brokers and bookies co-located on shared nodes, or fully disaggregated, with bookie storage on a dedicated fabric behind the cluster, on vSphere, bare-metal Kubernetes, or OpenShift. Simplyblock licenses per usable storage TB per year with no per-core fee, so scaling the bookie fleet horizontally to absorb higher publish throughput is a compute decision, not an automatic storage cost increase.
Questions and Answers
Why does Apache Pulsar need two separate storage tiers instead of one volume per bookie? Every BookKeeper bookie writes each entry to a fsync-heavy journal (which gates producer-visible publish latency) and to ledger storage (which is throughput-oriented and serves replay reads). Provisioning both from one generic persistent volume sized for capacity lets ledger-storage I/O compete with the journal’s fsync path, which shows up as publish-latency spikes unrelated to producer load.
What happens when a bookie is lost on generic Kubernetes storage? BookKeeper’s auto-recovery process re-replicates every under-replicated ledger fragment the lost bookie held, streaming data from the surviving ensemble members, an operation whose cost scales with the amount of data that bookie was holding. A persistent volume claim backed by simplyblock’s NVMe/TCP storage reattaches to a rescheduled bookie pod in seconds instead, avoiding ensemble-wide re-replication in most node-failure scenarios.
Can Pulsar’s own tiered storage offload replace a dedicated NVMe/TCP tier for BookKeeper? Tiered offload correctly moves ledgers that have aged past a topic’s retention policy to S3-compatible object storage, and that part of the architecture does not need to change. The gap is the hot path: the journal and the active ledger tier that every producer and consumer touches still need low-latency block storage, which tiered offload assumes exists upstream but does not provision.
How does simplyblock fit into an Apache Pulsar deployment on Kubernetes? Simplyblock provides NVMe/TCP (or NVMe/RoCE) block storage as separate persistent volumes for a bookie’s journal and ledger directories, each sized for its own I/O shape instead of one undifferentiated volume. Simplyblock’s CSI driver adds fast, storage-side snapshots of ledger volumes for point-in-time recovery, and the same architecture supports both hyperconverged and disaggregated deployment models.
How should I size storage for Pulsar’s BookKeeper layer on Kubernetes? Size the journal volume for consistent, low-latency fsync throughput at expected publish rate, and size the ledger volume separately for backlog capacity plus replay-read throughput, letting Pulsar’s own tiered storage offload age data out to S3-compatible object storage. Simplyblock’s pricing is per usable TB per year with no per-core fee, so scaling the bookie fleet horizontally for higher throughput is a compute decision, not an automatic storage cost increase.
