MongoDB on Kubernetes: Persistent Storage Architecture and High Availability

MongoDB is the most widely deployed document database in the world, and running it on Kubernetes is increasingly the standard for teams that want a unified platform for stateless services and stateful workloads. The challenge is that MongoDB’s storage access pattern is nothing like a stateless web service. WiredTiger, the default MongoDB storage engine, writes to a journal file before every committed write to guarantee durability, and replica set members apply oplog entries through the same write path. Any storage that cannot honor those journal fsyncs will become a bottleneck for write latency and secondary replication lag.

Getting storage right for MongoDB on Kubernetes means understanding both the WiredTiger write path and the Kubernetes scheduling model. The two interact in ways that create operational problems if you start with local disk and discover the constraints only when a node drains or fails.

Why MongoDB Storage Is Demanding on Kubernetes

WiredTiger writes to two on-disk structures for every committed operation: the data files (which contain collections and indexes in a B-tree format) and the journal (a write-ahead log that records every change before it is applied to data files). Before acknowledging a write to the client at the default j: true journaling concern, WiredTiger calls fsync on the journal file. This is the core durability guarantee, and it makes MongoDB extremely sensitive to storage write latency.

The practical I/O profile on a busy MongoDB node is dominated by three patterns:

• Random small writes (WiredTiger B-tree page writes to collection and index files): 4 KB to 32 KB pages scattered across the data directory
• Sequential fsync writes (journal): high-frequency, low-volume, latency-critical
• Sequential reads (cache warmup, range queries, aggregation pipeline scans): workload-dependent but less latency-sensitive

The journal fsync path is the critical one for tail latency. Cloud block storage volumes, particularly those with burst IOPS credit models, can introduce P99 latency spikes during journal writes when burst budgets are exhausted. These spikes translate directly into write latency visible to the application and into secondary replication lag, since secondaries apply oplog entries through the same WiredTiger write path.

On Kubernetes, these storage requirements combine with a scheduling problem. MongoDB runs as a Kubernetes StatefulSet with persistent storage attached to each pod. If that storage is local NVMe (hostPath or local PersistentVolumes), the pod becomes pinned to the node where the disk lives. When the node drains for maintenance or fails unexpectedly, the pod cannot reschedule until the node returns or the data is manually migrated.

Persistent Volume Architecture for MongoDB Replica Sets

MongoDB replica sets require one persistent volume claim per replica member, minimum, and ideally two: one for the data directory and one for the journal. Separating journal and data onto distinct volumes improves durability isolation and allows you to tune each volume’s IOPS independently. The journal volume is small (10 to 20 GB is typical) but must sustain low-latency sequential writes. The data volume is larger and handles both random reads and random writes.

The standard StatefulSet pattern uses volumeClaimTemplates to provision PVCs per replica automatically. Each MongoDB pod gets its own volume, independent of other replica set members. This is the correct model for MongoDB because shared volumes between replica members introduce distributed locking complexity that WiredTiger is not designed to handle.

Table 1: Persistent volume configuration recommendations for MongoDB StatefulSets on Kubernetes.

A minimal StorageClass for MongoDB on Kubernetes using a CSI driver looks like this:

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: mongodb-nvme
5
provisioner: csi.simplyblock.io
6
parameters:
7
  replication: "2"
8
  compression: "false"
9
reclaimPolicy: Retain
10
volumeBindingMode: WaitForFirstConsumer
11
allowVolumeExpansion: true

The volumeClaimTemplates section of the StatefulSet references this StorageClass for both volumes:

1
volumeClaimTemplates:
2
  - metadata:
3
      name: mongodb-data
4
    spec:
5
      accessModes: ["ReadWriteOnce"]
6
      storageClassName: mongodb-nvme
7
      resources:
8
        requests:
9
          storage: 100Gi
10
  - metadata:
11
      name: mongodb-journal
12
    spec:
13
      accessModes: ["ReadWriteOnce"]
14
      storageClassName: mongodb-nvme
15
      resources:
16
        requests:
17
          storage: 20Gi

The volumeBindingMode: WaitForFirstConsumer is critical: without it, the PVC binds in a random zone before the pod is scheduled, often ending in a zone mismatch that prevents the pod from starting.

Running MongoDB on Kubernetes and hitting write latency or secondary lag? simplyblock’s disaggregated NVMe/TCP storage delivers consistent sub-millisecond journal fsync latency without credit-burst limits, and lets MongoDB pods reschedule freely across nodes. Talk to a storage architect

NVMe/TCP Disaggregated Storage for MongoDB

NVMe over TCP (NVMe/TCP) carries native NVMe commands over standard Ethernet using the upstream Linux kernel driver (upstream since kernel 5.0). It delivers latency in the 100 to 400 microsecond range on a well-tuned 25 GbE link, which is within the acceptable range for WiredTiger’s journal fsync path.

The key advantage for MongoDB is that NVMe/TCP storage is network-attached: the volume is not tied to a physical disk on any particular node. MongoDB pods can reschedule to any node in the cluster, and the CSI driver re-attaches the volume to the new node transparently. There is no data migration on failover, and no need to wait for a specific node to come back online.

simplyblock provides disaggregated storage for Kubernetes via a standard CSI driver. Each MongoDB pod gets a simplyblock volume provisioned from a shared, scale-out NVMe pool. The pool applies replication (configurable, typically 2 or 3 copies) at the storage layer, so you do not need to run additional MongoDB secondaries solely for data durability. Secondaries can be sized for read scaling and geographic distribution instead.

The disaggregated model also means storage capacity and IOPS scale independently of compute. You can add NVMe capacity to the pool without adding Kubernetes nodes, and you can add Kubernetes nodes without adding disks.

High Availability Without Node-Affinity Pinning

The combination of MongoDB replica sets and Kubernetes StatefulSets creates a high availability topology, but local storage forces a trade-off: either you over-provision nodes so every replica has a local failover host, or you accept a recovery window when a node fails and the pod waits for it to return.

Disaggregated storage removes this constraint. When a node drains or fails, the MongoDB pod reschedules on any available node. The CSI driver attaches the same volume to the new node. WiredTiger performs journal recovery (typically under 30 seconds for a clean shutdown; longer for an unclean crash depending on dirty cache pages), and the secondary rejoins the replica set oplog stream.

Table 2: High availability model comparison for MongoDB replica sets on Kubernetes.

For teams using the MongoDB Community Kubernetes Operator or the Percona Operator for MongoDB, the persistent volume configuration is identical: each replica set member receives its own PVC via volumeClaimTemplates, and the operator manages replica set election and oplog coordination at the database layer. The storage layer handles replication and durability independently.

Snapshot-based point-in-time recovery is another advantage of the storage layer approach. simplyblock’s CSI external snapshotter integration allows you to take a consistent storage snapshot of a MongoDB data volume in seconds, without pausing writes or relying on mongodump. Restoring from snapshot brings the database back to a known-good state significantly faster than replaying the oplog from a backup.

Questions and Answers

What persistent storage settings work best for MongoDB on Kubernetes?

Use volumeBindingMode: WaitForFirstConsumer to prevent zone mismatches, reclaimPolicy: Retain to prevent accidental data loss on PVC deletion, and allowVolumeExpansion: true for online capacity growth. For IOPS, provision at least 5,000 IOPS per MongoDB replica for development or staging workloads; production clusters with sustained write load typically need 10,000 to 15,000 IOPS per node. Separate the journal and data directory onto different PVCs to isolate their I/O profiles and allow independent tuning.

How does WiredTiger’s journal affect storage latency on Kubernetes?

WiredTiger writes to its journal file and calls fsync before acknowledging a write at the default j: true journaling concern. If the storage volume has burst IOPS limits (common with cloud block storage), the fsync queue backs up under sustained write load, and write latency grows. The solution is storage that delivers consistent IOPS without burst credit limits: NVMe/TCP disaggregated storage removes the burst model entirely and delivers journal fsync latency in the sub-millisecond range on a well-tuned network.

Can MongoDB replica set pods reschedule freely with NVMe/TCP persistent storage?

Yes. NVMe/TCP volumes are network-attached and not bound to any physical node. When a MongoDB pod reschedules (due to node drain, node failure, or normal pod eviction), the CSI driver re-attaches the persistent volume to the new node. WiredTiger performs journal recovery and the pod resumes as a replica set secondary, rejoining the oplog replication stream. The recovery time depends on the WiredTiger cache flush state at the time of the shutdown, not on data migration across the network.

How does simplyblock compare to local NVMe for MongoDB on Kubernetes?

Local NVMe delivers the lowest absolute latency but pins the MongoDB pod to a specific node, complicates replica set recovery after node failure, and requires node-level RAID for durability. simplyblock’s disaggregated NVMe/TCP storage delivers latency in the 100 to 400 microsecond range on 25 GbE, which is within the acceptable range for WiredTiger’s journal path, while decoupling storage from compute. The scheduling flexibility and storage-layer replication reduce the operational burden significantly for teams running multiple replica set members.

Does this architecture work for MongoDB sharded clusters?

Yes. A MongoDB sharded cluster is composed of multiple replica sets (one per shard) plus config server replica sets. Each shard member uses the same StatefulSet and volumeClaimTemplates pattern described above. The storage architecture is identical per shard: each pod gets its own data and journal PVCs, provisioned from the same NVMe pool. The config servers follow the same pattern with smaller volume sizes. Mongos routers are stateless and do not need persistent storage.