NVMe-oF for Kubernetes Storage: A Platform Engineer's Guide

NVMe over Fabrics (NVMe-oF) is the protocol family that extends NVMe command semantics across a network fabric. In the context of Kubernetes storage, it means that a pod running on one node can read from and write to an NVMe device located on a different node, with latency measured in hundreds of microseconds rather than milliseconds.

This guide covers what NVMe-oF is, why it matters for Kubernetes persistent storage, how the two main transport options differ, and what platform engineers need to validate before deploying NVMe-oF-based storage in production.

What NVMe-oF Is and Why It Matters

Traditional Kubernetes storage over a network uses iSCSI, NFS, or similar block/file protocols. These protocols were designed for spinning disk, where millisecond-level latency was acceptable and CPU overhead per I/O operation was not a concern.

NVMe-oF is different in three ways:

1. It carries NVMe commands directly. The host sends the same NVMe commands it would send to a locally attached NVMe SSD, but the target device is remote. There is no translation layer, no SCSI conversion, and no protocol overhead that iSCSI adds.

2. It has explicit queue depth management. NVMe supports up to 65,535 queues per device, each with up to 65,535 outstanding commands. iSCSI supports one queue per session. For high-throughput Kubernetes workloads like databases, message queues, and AI training, this queue depth difference translates directly to throughput capacity.

3. It was designed for flash. The latency model assumes sub-millisecond device response times. Protocol overhead is low enough that network round-trip time, not protocol processing, dominates storage access latency.

For Kubernetes workloads that benefit from high IOPS and low latency, like PostgreSQL, MySQL, Kafka, and vector databases, NVMe-oF changes what is achievable over a shared storage pool.

NVMe/TCP vs. NVMe/RoCE

NVMe-oF defines a transport-agnostic command model and then specifies bindings for different network fabrics. The two most relevant for Kubernetes environments are NVMe/TCP and NVMe/RoCE.

NVMe/TCP

NVMe/TCP runs NVMe-oF over standard TCP/IP, which means it works on any Ethernet network including existing 10GbE, 25GbE, and 100GbE infrastructure. No specialized network hardware or RDMA-capable NICs are required.

Trade-offs:

• Slightly higher CPU overhead than RoCE because TCP processing is done in software
• Latency is typically 100-400 microseconds for round-trip, depending on network conditions
• Works with standard Linux kernel NVMe/TCP driver (in-tree since kernel 5.0)
• Compatible with all Ethernet switches, including commodity hardware

NVMe/TCP is the right choice for most Kubernetes deployments. The CPU overhead is manageable on modern multi-core servers, and the latency is an order of magnitude lower than iSCSI.

NVMe/RoCE

NVMe/RoCE (RDMA over Converged Ethernet) uses RDMA instead of TCP. RDMA bypasses the kernel network stack: data transfers happen directly between the NIC and application memory without CPU intervention on the data path.

Trade-offs:

• Requires RDMA-capable NICs (RoCE v2, InfiniBand with RoCE emulation)
• Requires a lossless network fabric (PFC-enabled switches)
• Latency is typically 20-100 microseconds, meaningfully lower than NVMe/TCP
• More infrastructure complexity: DCQCN congestion control, PFC configuration, RoCE NIC tuning
• Higher hardware cost

NVMe/RoCE makes sense for workloads where sub-100-microsecond storage latency is a hard requirement, such as financial trading systems, high-frequency write applications, or AI training where storage access time directly affects GPU utilization.

For most Kubernetes platform teams, NVMe/TCP provides the right balance of performance and operational simplicity.

Evaluating NVMe-oF storage for your Kubernetes cluster? simplyblock delivers sub-200-microsecond storage latency over NVMe/TCP with no specialized hardware, built for the stateful workloads your cluster actually runs. Talk to a storage architect

How the Kubernetes CSI Layer Sits on Top

The Container Storage Interface (CSI) is the standard API by which Kubernetes communicates with storage systems. A CSI driver handles:

• Volume provisioning: responding to PersistentVolumeClaim creation by creating a volume on the storage system
• Volume attachment: connecting the volume to the node where the pod will run
• Volume mounting: making the volume available as a filesystem or block device inside the container

For NVMe-oF-based storage, the CSI driver’s volume attachment step is where NVMe-oF comes in. The driver establishes an NVMe-oF connection from the node to the storage target and exposes the resulting NVMe block device to Kubernetes. The pod sees a standard block device. It does not need to know that the device is remote.

This means NVMe-oF storage is transparent to applications. A PostgreSQL pod reads and writes to a PersistentVolume exactly as it would read and write to a local disk. The storage system handles the network transport, data protection, and replication behind the CSI interface.

CSI Features Relevant to NVMe-oF Storage

When evaluating an NVMe-oF Kubernetes storage system, check that the CSI driver implements:

• VolumeSnapshots: copy-on-write snapshots exposed as standard Kubernetes VolumeSnapshot objects, compatible with Velero and other backup tools
• Volume expansion: online resize of PersistentVolumeClaims without pod restart
• Topology awareness: placement hints that ensure volumes are accessible from the nodes where pods will run
• Volume health monitoring: surfacing storage device conditions as Kubernetes events on PVC objects

NVMe Multipath in Kubernetes

NVMe-oF supports multiple paths to the same volume simultaneously via NVMe ANA (Asymmetric Namespace Access). Multipath serves two purposes:

1. Failover: if one path fails (NIC failure, network switch failure, storage node restart), I/O automatically continues on surviving paths. Applications see no error.
2. Load balancing: I/O can be distributed across multiple paths for higher aggregate throughput.

In a Kubernetes context, the CSI driver manages path establishment during volume attachment. When a path fails, the NVMe driver handles reconnection automatically. The Kubernetes pod does not restart, and the application does not see an I/O error unless all paths fail simultaneously.

The reconnect timeout is configurable. For storage where node-level failures are expected (rolling upgrades, node replacements), setting a short reconnect timeout (10-30 seconds) ensures fast failover. For storage where paths are expected to be stable, a longer timeout reduces unnecessary reconnect chatter.

See NVMe Multi-Path in Kubernetes: How Failover Actually Works for a detailed walkthrough of the reconnect sequence.

What to Validate Before Deploying in Production

Before running NVMe-oF-based Kubernetes storage in production, platform engineers should validate the following:

1. Latency Under Load

Run FIO workloads that match your application’s read/write ratio, block size, and queue depth. Measure P50, P99, and P99.9 latency. NVMe-oF latency degrades under saturation differently than iSCSI, and the tail latency profile matters for database workloads.

2. Failover Behavior

Simulate a storage node failure while a FIO workload is running. Measure how long I/O is interrupted from the application’s perspective. Verify that no I/O errors surface to the application during the failover window. Verify that path reconnection completes within your acceptable RTO for node-level failures.

3. Concurrent Workload Isolation

Run two workloads simultaneously: one high-throughput (backup, ETL) and one latency-sensitive (database). Verify that the latency-sensitive workload’s P99 latency does not degrade during the high-throughput operation. If the storage system supports per-PVC QoS, verify that QoS limits prevent the noisy-neighbor effect.

4. Network Saturation Behavior

Confirm what happens when the network connecting storage nodes is saturated. Does the storage system throttle gracefully or does it drop connections? Understand the behavior before it happens in production.

5. Upgrade Procedure

Verify that storage node upgrades can be performed without interrupting I/O to running pods. Test rolling upgrades under load. The storage system’s documentation should describe the expected I/O behavior during an upgrade.

Key Concepts Summary

Next Steps

For platform engineers evaluating NVMe-oF storage for Kubernetes:

• See the storage observability guide for how to instrument NVMe-oF storage with Prometheus and validate performance in production
• See the QoS and performance isolation guide for how to prevent batch and backup workloads from degrading database latency
• See the storage vendor reliability evaluation guide for how to assess vendor claims about durability and availability

Questions and Answers

What is NVMe-oF and how does it differ from iSCSI for Kubernetes storage?

NVMe over Fabrics carries native NVMe commands across a network fabric. iSCSI wraps SCSI commands over TCP, which adds a translation layer and limits queue depth to one per session. NVMe-oF supports up to 65,535 queues per device, each with up to 65,535 outstanding commands. For Kubernetes database and AI workloads where concurrent I/O depth determines throughput, this queue model is the primary reason NVMe-oF delivers meaningfully higher performance than iSCSI at the same network bandwidth.

What is the difference between NVMe/TCP and NVMe/RoCE?

NVMe/TCP runs NVMe-oF over standard TCP/IP, which works with any Ethernet switch and requires no specialized NICs. Latency is typically 100-400 microseconds round-trip. NVMe/RoCE uses RDMA, bypassing the kernel network stack and delivering 20-100 microseconds latency, but requires RoCE-capable NICs and a lossless network fabric with PFC-enabled switches. For most Kubernetes storage deployments, NVMe/TCP provides sufficient performance with significantly lower infrastructure complexity and cost.

Does my Kubernetes cluster need special hardware to use NVMe-oF storage?

For NVMe/TCP, no specialized hardware is required. Any standard Ethernet infrastructure (10GbE, 25GbE, or 100GbE) works, and the Linux kernel NVMe/TCP driver has been in-tree since kernel 5.0. Storage nodes require NVMe drives, but compute nodes accessing the storage over the network use standard NICs. NVMe/RoCE requires RDMA-capable NICs and lossless switch configuration, which is only justified when sub-100-microsecond latency is a hard workload requirement.

How does multipath failover work with NVMe-oF storage in Kubernetes?

NVMe Asymmetric Namespace Access (ANA) allows a volume to have multiple active paths simultaneously. The CSI driver establishes these paths during volume attachment. If a path fails due to a NIC failure, switch failure, or storage node restart, the NVMe driver switches to a surviving path automatically. The Kubernetes pod does not restart and the application does not see an I/O error unless all paths fail simultaneously. Path reconnection completes within the configurable reconnect timeout, typically 10-30 seconds for node-level failures.

What should I validate before deploying NVMe-oF storage in production?

Five validation points matter most: (1) P99 and P99.9 latency under load at your application’s actual queue depth and block size, not just P50 averages; (2) failover behavior when a storage node is killed during an active workload, measured from the application’s perspective; (3) concurrent workload isolation between latency-sensitive and bulk workloads; (4) network saturation behavior to confirm the system throttles gracefully rather than dropping connections; (5) rolling upgrade procedure under load to verify I/O is not interrupted during storage node upgrades.