Rob Pankow 
Every object a Kubernetes cluster knows about, every pod spec, ConfigMap, Secret, and custom resource, lives in etcd, the distributed key-value store behind the API server. Application storage gets most of the attention in Kubernetes storage conversations, but etcd’s own storage layer is arguably more sensitive to latency than any workload running on top of it, because a slow disk under etcd does not just slow one application. It can stall the entire control plane.
etcd replicates its data through Raft consensus, and Raft’s safety guarantees depend on every write being durable before it is acknowledged. That durability requirement means a synchronous disk fsync sits directly in the write path of every API server request that changes cluster state. When that fsync is slow, or worse, inconsistent, the effects show up as apply latency warnings, leader election churn, and in the worst case, a control plane that stops accepting writes at all.
This guide covers etcd’s Raft commit path and where disk latency enters it, what etcd’s own operational guidance says about storage requirements, and how disaggregated NVMe/TCP (or NVMe/RoCE) storage gives control-plane nodes a latency floor that does not depend on which physical disk happens to be installed in a given node.
Why etcd Storage Decides Kubernetes Control Plane Stability
etcd is a single Raft group, not a sharded system like the application databases most storage teams size for. Every key, across every namespace and every custom resource definition in the cluster, is replicated through one Raft log to (typically) three or five members. That has two consequences that make etcd’s storage requirements different from, and often stricter than, an application-layer database:
- There is no partitioning to spread hot keys across disks. A busy cluster with many controllers reconciling CRDs, many ConfigMap updates, or heavy leader-election traffic among operators all funnels through the same Raft log and the same underlying store. One slow disk affects every object in the cluster, not just one tenant’s data.
- The API server has no fallback path. If etcd cannot commit a write, the API server cannot complete the corresponding
kubectl apply, scheduler decision, or controller reconcile. Application-layer storage failures are usually contained to the workload that owns the volume. An etcd storage failure degrades the platform itself.
etcd’s own documentation is explicit that disk performance is the most common root cause of cluster instability report it receives: slow disks show up as apply took too long warnings in etcd logs, elevated backend_commit_duration_seconds metrics, and, if sustained, as false leader elections triggered when a leader’s own disk-bound heartbeat processing falls behind the follower’s election timeout.
The Raft Commit Path: How fsync Latency Becomes Cluster Latency
When the API server writes a key, here is what has to happen before that write is considered durable:
- The etcd leader appends the entry to its write-ahead log (WAL) and issues an
fsyncto guarantee the entry survives a crash before proposing it to followers. - Each follower does the same: append to its own WAL, fsync, then acknowledge.
- Once a majority of the cluster (2 of 3, or 3 of 5) has fsynced and acknowledged, the leader considers the entry committed and applies it to etcd’s underlying store, a single-writer BoltDB (bbolt) file that holds the actual key-value data and the MVCC revision history.
The commit latency for that write is bound by the slowest disk in the quorum, exactly the same “quorum property, not a single-node property” pattern that shows up in Raft-based application databases. But etcd adds a second constraint on top: Raft’s heartbeat and election-timeout mechanism assumes disk-bound operations (fsync, WAL append, periodic backend commits) stay well inside a predictable window. If a leader’s disk stalls long enough that it cannot process heartbeats in time, followers time out and trigger a new election, even though the leader was never actually unreachable on the network. A single slow or congested disk can trigger a leadership change across the whole cluster.
This is why etcd’s hardware guidance is unusually specific compared to most stateful workloads: it recommends a dedicated SSD, ideally NVMe, for the data directory, explicitly discourages network file systems and shared SAN volumes with unpredictable tail latency, and treats fsync duration as the metric to watch first when diagnosing instability. The guidance is not about throughput. A modestly sized etcd cluster does not need high bandwidth. It needs a disk whose 99th-percentile write latency stays low and consistent under contention, because a rare slow fsync is exactly the kind of event that triggers a false leader election.
Storage Backends for etcd: Meeting the Latency Budget
Kubernetes control planes run on a range of storage backends depending on how the cluster was deployed, and each one has a different relationship to the latency and consistency etcd needs.
| Storage backend | Typical p99 write latency | Tail-latency consistency | Node-failure recovery |
|---|---|---|---|
| Local hostPath / direct-attached SSD | Low, hardware-dependent | Good on dedicated hardware, but tied to that specific disk’s health and wear | Full etcd member re-add and snapshot restore; volume does not survive the node |
| Shared NAS / network file system | Often exceeds etcd’s fsync budget | Poor; shared clients and network contention introduce spikes | Depends on NAS availability, adds another failure domain |
| Generic cloud block storage (network-attached) | Variable, workload-dependent | Inconsistent under noisy-neighbor contention | Volume can reattach, but latency floor is not guaranteed |
| Disaggregated NVMe/TCP | Consistently low, decoupled from any one node’s local disk | High; NVMe/TCP delivers predictable per-request latency at the volume level | Volume reattaches to a replacement node in seconds, no full member resync from snapshot required |
The pattern that matters for etcd specifically is the last column combined with the second. A storage layer only helps a control plane if it keeps p99 latency low and lets a replacement node pick up the same volume without a full data resync, because etcd’s own member-replacement path (add a new member, let it stream a full snapshot from the leader) is itself an operation that briefly increases load on the remaining quorum members. Avoiding that path on every node failure removes one more source of transient latency spikes.
Disaggregated NVMe/TCP as etcd’s Storage Layer
Running etcd on local, direct-attached NVMe satisfies the latency requirement, but it ties every control-plane node’s stability to that one physical disk and forces a full snapshot restore whenever that node is replaced. Disaggregated storage over NVMe/TCP keeps the same low, consistent per-request latency etcd’s Raft commit path needs, while decoupling the volume from the node’s local hardware: a control-plane node can be replaced, and the etcd data volume reattaches to its successor instead of requiring a fresh member add and snapshot stream.
This does not replace etcd’s own backup and defragmentation practices, periodic snapshotting, and BoltDB compaction remain the operator’s job regardless of the storage layer underneath. What it changes is the failure mode: instead of every node loss triggering etcd’s snapshot-stream recovery path, most node replacements become a volume reattachment, and the disk-latency variance that causes false leader elections is bounded at the storage layer rather than left to whatever local disk happens to be in a given server.
Sizing the control plane storage layer for a VMware exit or fresh Kubernetes build? Talk to us about the latency and failover profile your etcd cluster needs before you pick a storage backend for control-plane nodes. Talk to a storage architect
Questions and Answers
Why is etcd more sensitive to disk latency than application databases running on Kubernetes?
etcd is a single Raft group for the entire cluster’s state, with no sharding to isolate a slow write to one tenant or namespace. Every API server write, from any namespace or controller, funnels through the same WAL fsync path, so a slow disk affects the whole platform rather than one workload.
Can a slow disk actually cause a Kubernetes leader election, not just slow requests?
Yes. etcd’s leader must process heartbeats within the follower’s election timeout. If disk-bound operations like fsync or backend commits stall long enough, heartbeats fall behind, followers assume the leader is unreachable, and a new election starts, even though the network connection was fine the whole time.
Does simplyblock replace etcd’s own snapshot and backup process?
No. etcd’s periodic snapshotting, defragmentation, and backup practices are still required regardless of the storage layer. What changes with disaggregated NVMe/TCP is the failure mode for node loss: a volume reattachment rather than a full member re-add and snapshot stream from the remaining quorum.
Should every Kubernetes control-plane node use disaggregated storage for etcd, or is local NVMe still an option?
Local, direct-attached NVMe can meet etcd’s latency requirements on its own, and remains valid for smaller or single-vendor clusters. Disaggregated NVMe/TCP becomes the stronger choice once you need to standardize latency across heterogeneous control-plane hardware, or want node replacement to skip the full snapshot-restore path, which matters more as cluster count and node churn increase.
How does this relate to CSI volume performance for application storage?
etcd’s own data directory typically is not provisioned through a CSI volume in the same way application PVCs are, but the underlying requirement, low and consistent write latency, is the same one that determines whether CSI-provisioned volumes are fast enough for latency-sensitive stateful applications. See why CSI alone isn’t enough for stateful Kubernetes at scale for the application-layer version of this problem, and CSI snapshot architecture for how volume-level recovery mechanics compare to etcd’s own snapshot model.
