NVMe Multi-Path in Kubernetes: How Failover Actually Works

Storage failover in NVMe-based distributed storage is often described in general terms: “multi-path ensures high availability,” “failover is automatic,” “the application doesn’t see the failure.” These descriptions are accurate but they obscure the mechanism, and the mechanism matters when you’re making architecture decisions or debugging an incident.

This post is a technical explainer of how NVMe multi-path works in Kubernetes-deployed distributed storage: how paths are established, how the NVMe driver detects failure, what the reconnect sequence looks like, and what the application sees during the transition.

The context is simplyblock’s implementation over NVMe/TCP, but the NVMe-oF protocol behavior described here is standard across compliant implementations.

What Multi-Path Means for Distributed NVMe Storage

NVMe multi-path is a mechanism that presents a single logical volume through multiple independent paths to the host. A “path” is a TCP connection (for NVMe/TCP) or an RDMA connection (for NVMe/RoCE) between the host-side NVMe driver and a specific storage node’s SPDK target.

In a distributed storage system like simplyblock, each logical volume (LVOL) has a primary storage node and one or more replica nodes. Multi-path means the host maintains an active connection to the primary and standby connections to the replicas. If the primary becomes unreachable, the host-side driver switches I/O to a secondary path.

The critical property is that this switch happens at the block device layer, below the filesystem and the application. A PostgreSQL database running on an NVMe-backed PVC does not call a failover function. The Kubernetes kubelet does not restart the pod. The storage layer detects the path failure, switches the active path, and the database’s next read or write goes to the secondary without application involvement.

How Paths Are Established

When a simplyblock volume is attached to a Kubernetes node, the path setup sequence works as follows:

1. The CSI driver’s node plugin receives a NodeStageVolume call from the kubelet.
2. The node plugin establishes NVMe/TCP connections to each storage node that holds a replica of the volume, calling the NVMe connect sequence for each target.
3. The kernel NVMe driver creates an nvme device for each connection. The multipath layer assembles these into a single block device that the filesystem layer sees.
4. The block device is formatted (if new) and mounted at the staging path. The kubelet then bind-mounts it into the pod.

The multipath configuration assigns ANA (Asymmetric Namespace Access) state to each path: Optimized for the primary, Non-Optimized for replicas. I/O is directed to the Optimized path by default. The Non-Optimized paths carry standby connections that accept no I/O unless the Optimized path becomes unavailable.

This is different from traditional MPIO (Multipath I/O) where all paths are typically active. In NVMe ANA, path selection is explicit: the target advertises which path should carry I/O, and the host honors that preference until the preferred path fails.

Path Failure Detection

The NVMe/TCP keep-alive mechanism is the primary failure detector. The host sends a KeepAlive NVMe command to each connected target on a configurable interval (the Linux default keep_alive_tmo is 5 seconds for NVMe/TCP). If the target does not respond within the timeout, the driver marks the controller as failed.

Three events trigger a controller failure:

TCP connection closed. The target drops the connection actively. The driver receives a TCP RST or FIN and marks the controller failed immediately, without waiting for keep-alive timeout. This is what happens when the SPDK process inside a storage node is killed: the kernel on that node closes its TCP connections and the host driver sees the disconnect within milliseconds.

Keep-alive timeout. The target is unreachable due to a network partition, host crash, or hardware failure, but no TCP RST arrives. The driver waits for keep_alive_tmo before marking failure. For most production NVMe/TCP configurations this is tuned to 1-2 seconds rather than the 5-second default, because the shorter window reduces the latency spike applications see during failover.

ANA state change. The target explicitly transitions the path’s ANA state to Inaccessible or Persistent Loss. This is the mechanism used during graceful shutdowns: the storage node announces that it’s going offline, transitions its paths to Non-Optimized and then Inaccessible in a controlled sequence, and the host switches I/O before the connection closes.

The difference between TCP disconnect detection and keep-alive timeout detection is significant for the application-visible latency spike during failover. A container_stop failure (SPDK process killed) triggers TCP disconnect detection: the path switch completes in milliseconds. A network partition that blocks all traffic without dropping connections triggers keep-alive timeout detection: the path switch completes within the keep-alive timeout window.

The Reconnect Sequence

After primary path failure is detected, the NVMe multipath layer selects the next available Optimized path. If no Optimized path is available, it falls back to Non-Optimized. The I/O queue that was pending on the failed path is requeued to the selected path.

The application-visible impact of this transition depends on two things: how quickly the path switch completes and whether any in-flight writes were pending acknowledgement at the moment of failure.

Path switch latency for TCP disconnect. In the container_stop scenario, the TCP connection close is immediate and the path switch happens within milliseconds. The NVMe driver does not wait for keep-alive timeout when it receives a TCP disconnect. For workloads that can tolerate brief I/O queueing during path selection, this is transparent.

In-flight write handling. Writes that were submitted to the NVMe driver but not yet acknowledged at the moment of path failure are requeued to the secondary path. The storage system must determine whether these writes were committed before the failure. simplyblock uses a write journal to track in-flight operations: any write that was journaled but not confirmed committed is replayed on the secondary node. The host does not see duplicate writes at the NVMe layer because the protocol-level acknowledgement hasn’t been issued yet.

Partial network partitions. The interface_partial_NW failure mode is more complex. The host still has a TCP connection to the storage node, but the connection may be degraded: keep-alive responses may arrive intermittently. The driver must decide whether to mark the path failed based on a degraded connection that’s still technically alive. This is the scenario where split-brain risk exists in distributed systems, and it’s why testing partial partition behavior specifically, rather than only complete failures, is a meaningful distinction between storage systems.

What the Application Sees

For a standard container_stop failure with a healthy secondary path available, the application sees I/O latency spikes during the path switch period. The size of this spike depends on the keep-alive configuration and the workload’s queue depth.

For PostgreSQL running on an NVMe-backed PVC, a path switch during a write-heavy workload typically looks like this in pg_stat_activity:

• Queries in progress at the moment of failure are suspended at the I/O wait stage.
• The NVMe driver completes the path switch and requeues in-flight operations.
• Queries resume when the requeued operations complete on the secondary path.
• No query errors occur unless the path switch takes longer than the application’s statement timeout, which in well-configured environments it doesn’t.

The key condition for transparent failover is that the secondary path’s storage node has a current copy of all committed data at the moment of primary failure. In simplyblock’s replication model, writes are committed to a quorum of nodes before being acknowledged to the host. This means the secondary path always has the data that the primary path acknowledged.

For read-heavy workloads, the behavior is simpler: reads can be served from any path that has the data. The path switch is faster and the latency impact is smaller because no write requeue is needed.

How the container_stop Test Validates This

simplyblock’s container_stop chaos test exercises this exact sequence under load: 20 volumes running FIO workloads continuously, SPDK process killed abruptly on a storage node, FIO continues without any I/O errors. The test doesn’t check that the cluster recovers. It checks that FIO reports zero errors throughout the transition, including during the path switch.

The test validates these properties:

1. Path failure detection completes before FIO’s I/O timeout triggers an error
2. The secondary path accepts requeued operations without integrity errors
3. Every write acknowledged before the kill is present and correct in a checksum scan after recovery
4. The primary path reconnects after SPDK restarts and resumes Optimized status without requiring application or operator intervention

The interface_partial_NW and interface_full_NW tests cover the more adversarial scenarios. These take longer to detect because keep-alive timeout, not TCP disconnect, drives failure detection. The FIO timeout parameters in these tests reflect the longer detection window: the test expects that brief I/O queueing will occur during the detection window, but requires that no I/O errors result.

The test source is publicly available: github.com/simplyblock/sbcli/tree/main/e2e. The container_stop and network partition failure types are implemented in the E2E failover test classes.

Need to understand how NVMe multi-path failover behaves on your specific Kubernetes platform and workload? The simplyblock engineering team can walk through the configuration and expected behavior for your environment. Talk to a storage architect

Questions and Answers

What is the default keep-alive timeout for NVMe/TCP connections, and should I change it?

The Linux NVMe/TCP driver default keep_alive_tmo is 5 seconds. For production storage, this is typically too long. Reducing it to 1-2 seconds means the host detects failure faster in network partition scenarios where TCP disconnect doesn’t occur. The tradeoff is that shorter timeouts increase false positive risk on high-latency links. simplyblock recommends 1-2 seconds for NVMe/TCP production configurations and adjusts this as part of the standard deployment setup.

What is the difference between NVMe/TCP and NVMe/RoCE for failover behavior?

Both use the same NVMe-oF protocol and ANA path selection mechanism. The difference is the transport layer: NVMe/TCP runs over standard TCP/IP and is available on any Ethernet network; NVMe/RoCE uses RDMA over Converged Ethernet and requires hardware support. Failover behavior is similar, but NVMe/RoCE can achieve lower latency during the path switch because RDMA disconnect notification is faster than TCP keep-alive expiry. For most Kubernetes clusters, NVMe/TCP is the practical choice because it requires no special hardware. simplyblock supports both transports.

Does the Kubernetes CSI driver have to do anything special to support multi-path?

The CSI driver must ensure that NVMe connect is called for all replica paths, not just the primary, when staging a volume. The multipath device is then assembled by the kernel’s NVMe multipath layer. The CSI driver doesn’t manage path switching directly; that happens in the kernel’s NVMe driver using ANA state information from the targets. simplyblock’s CSI driver handles multi-path setup during NodeStageVolume and ensures paths are correctly reconnected after node reboots or kubelet restarts.

Does multi-path work across different network interfaces on the same host?

Yes. In simplyblock’s configuration, each NVMe/TCP path can use a different network interface on the host, providing not just storage node redundancy but also network path redundancy. A failure of one host NIC doesn’t take down all paths to storage if each path is on a separate interface. This configuration requires that the storage nodes are reachable from each of the host’s storage network interfaces, which is standard in rack-scale NVMe deployments.