MySQL on Kubernetes: Persistent Storage Architecture and High Availability

MySQL is the most widely deployed relational database in the world, and running it on Kubernetes is increasingly the standard for teams that want a unified platform for both stateless services and stateful workloads. The challenge is that MySQL’s storage access pattern is nothing like a stateless web service. InnoDB issues dozens to hundreds of fsync calls per second under write load, and any storage that cannot honor those durability flushes quickly becomes a bottleneck for query latency and replica lag.

Getting storage right for MySQL on Kubernetes means understanding both the InnoDB write path and the Kubernetes scheduling model. The two interact in ways that create operational problems if you start with local disk and discover the constraints only when a node drains or fails.

Why MySQL Storage Is Demanding on Kubernetes

InnoDB, the default MySQL storage engine, writes to two separate on-disk structures for every committed transaction: the redo log (a circular sequential log that records every change before it touches the data pages) and the data files themselves. Before a transaction is acknowledged to the client, InnoDB calls fsync on the redo log to ensure the changes are durable on disk. This is the core durability guarantee, and it makes InnoDB extremely sensitive to storage latency.

The practical I/O profile on a busy MySQL node is dominated by three patterns:

• Small-block random writes (InnoDB dirty page writes to data files): 4 KB to 16 KB pages scattered across the tablespace
• Sequential fsync writes (redo log): high-frequency, low-volume, latency-critical
• Sequential reads (buffer pool warmup, range scans, SELECT queries): workload-dependent but less latency-sensitive

The fsync path is the critical one for tail latency. Cloud block storage volumes, particularly those with burst IOPS credit models, can introduce P99 latency spikes during fsync when the burst budget is exhausted. These spikes translate directly into query stalls and replica lag on MySQL replicas, since replicas apply binlog events through the same InnoDB write path.

On Kubernetes, these storage requirements combine with a scheduling problem. MySQL runs as a Kubernetes StatefulSet with persistent storage attached to each pod. If that storage is local NVMe (hostPath or local PersistentVolumes), the pod becomes pinned to the node where the disk lives. When the node drains for maintenance or fails unexpectedly, the pod cannot reschedule until the node returns or the data is manually migrated.

Persistent Volume Architecture for MySQL StatefulSets

MySQL on Kubernetes requires at least two persistent volume claims: one for the data directory and one for the redo log. Separating redo log and data onto distinct volumes improves durability isolation: a failing data volume does not corrupt in-flight redo entries, and vice versa.

The standard StatefulSet pattern uses volumeClaimTemplates to provision a PVC per replica automatically. Each MySQL pod gets its own volume, independent of other replicas. This is the correct model for MySQL because shared storage between MySQL replicas introduces distributed locking complexity that InnoDB is not designed to handle.

Table 1: Persistent volume configuration recommendations for MySQL StatefulSets on Kubernetes.

A minimal StorageClass for MySQL on Kubernetes using a CSI driver looks like this:

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: mysql-nvme
5
provisioner: csi.simplyblock.io
6
parameters:
7
  replication: "2"
8
  compression: "false"
9
reclaimPolicy: Retain
10
volumeBindingMode: WaitForFirstConsumer
11
allowVolumeExpansion: true

The volumeBindingMode: WaitForFirstConsumer is critical: without it, the PVC binds in a random zone before the pod is scheduled, often ending in a zone mismatch that prevents the pod from starting.

For the redo log volume, you can use the same StorageClass or a dedicated one with higher IOPS allocation. Separating them at the StorageClass level lets you tune each independently as the workload grows.

Running MySQL on Kubernetes and hitting write latency or replica lag? simplyblock’s disaggregated NVMe/TCP storage delivers consistent sub-millisecond fsync latency without credit-burst limits, and lets MySQL pods reschedule freely across nodes. Talk to a storage architect

NVMe/TCP Disaggregated Storage for MySQL

NVMe over TCP (NVMe/TCP) carries native NVMe commands over standard Ethernet using the upstream Linux kernel driver (upstream since kernel 5.0). It delivers latency in the 100 to 400 microsecond range on a well-tuned 25 GbE link, which is within the acceptable range for InnoDB’s redo log fsync path.

The key advantage for MySQL is that NVMe/TCP storage is network-attached: the volume is not tied to a physical disk on any particular node. MySQL pods can reschedule to any node in the cluster, and the CSI driver re-attaches the volume to the new node transparently. There is no data migration on failover, and no need to wait for a specific node to come back online.

simplyblock provides disaggregated NVMe/TCP block storage for Kubernetes via a standard CSI driver. Each MySQL pod gets a simplyblock volume provisioned from a shared, scale-out NVMe pool. The pool applies replication (configurable, typically 2 or 3 copies) at the storage layer, so you do not need to run additional MySQL replicas solely for data durability. Replicas can be sized for read scaling and geographic distribution instead.

The disaggregated model also means storage capacity and IOPS scale independently of compute. You can add NVMe capacity to the pool without adding Kubernetes nodes, and you can add Kubernetes nodes without adding disks.

High Availability Without Node-Affinity Pinning

The combination of MySQL replication and Kubernetes StatefulSets creates a high availability topology, but local storage forces a trade-off: either you over-provision nodes so every replica has a local failover host, or you accept a recovery window when a node fails and the pod waits for it to return.

Disaggregated NVMe/TCP storage removes this constraint. When a node drains or fails, the MySQL pod is rescheduled on any available node. The CSI driver attaches the same volume to the new node. InnoDB performs crash recovery using the redo log (typically under 10 seconds for a clean shutdown, longer for a crash depending on dirty page count), and the replica rejoins the replication stream.

Table 2: High availability model comparison for MySQL on Kubernetes.

For MariaDB operators running Galera Cluster, the same architecture applies: each Galera node gets an independent persistent volume, and the storage layer handles replication independently of Galera’s own synchronous replication. The two replication layers serve different purposes: Galera replication provides multi-primary write availability, while storage-layer replication provides durability against disk failure.

Snapshot-based point-in-time recovery is another advantage of the storage layer approach. simplyblock’s CSI external snapshotter integration allows you to take a consistent storage snapshot of a MySQL volume in seconds, without pausing writes or relying on mysqldump. Restoring from snapshot brings the database back to a known-good state significantly faster than replaying binlogs from a backup.

Questions and Answers

What persistent storage settings work best for MySQL on Kubernetes?

Use volumeBindingMode: WaitForFirstConsumer to prevent zone mismatches, reclaimPolicy: Retain to prevent accidental data loss on PVC deletion, and allowVolumeExpansion: true for online capacity growth. For IOPS, provision at least 5,000 IOPS per MySQL node for a development or staging workload; production clusters with sustained write load typically need 10,000 to 20,000 IOPS per node. Separate the redo log and data directory onto different PVCs to isolate their I/O profiles.

Why does InnoDB replica lag increase under high write load on Kubernetes?

InnoDB replicas apply binlog events through the same write path as the primary: each applied event triggers an fsync on the replica’s redo log. If the replica’s storage has burst IOPS limits (common with cloud block storage), the fsync queue backs up under sustained write load, and replica lag grows. The solution is storage that delivers consistent IOPS without burst credit limits: NVMe/TCP disaggregated storage removes the burst model entirely.

Can MySQL pods reschedule freely with NVMe/TCP persistent storage?

Yes. NVMe/TCP volumes are network-attached and not bound to any physical node. When a MySQL pod reschedules (node drain, node failure, or normal pod eviction), the CSI driver re-attaches the persistent volume to the new node. InnoDB performs crash recovery from the redo log and the pod resumes normal operation. The recovery time depends on the dirty page count at the time of the last checkpoint, not on data migration.

How does simplyblock compare to local NVMe for MySQL on Kubernetes?

Local NVMe delivers the lowest absolute latency, but it pins the MySQL pod to a specific node, complicates StatefulSet recovery, and requires node-level RAID for durability. simplyblock’s disaggregated NVMe/TCP storage delivers latency in the 100 to 400 microsecond range on 25 GbE, which is within the acceptable range for InnoDB’s redo log path, while decoupling storage from compute. The scheduling flexibility and storage-layer replication reduce the operational burden significantly for teams running multiple MySQL replicas.

Does this architecture work for MariaDB and Galera Cluster?

Yes. MariaDB uses the same InnoDB write path and the same StatefulSet + PVC pattern on Kubernetes. Galera Cluster adds synchronous multi-primary replication at the database layer; the persistent volume configuration is identical. Each Galera node gets its own PVC. Storage-layer replication (from simplyblock) and Galera replication serve different failure domains: storage replication protects against disk loss, Galera replication provides write availability across nodes.