Rob Pankow 
NVMe over Fabrics (NVMe-oF) is the protocol that lets Kubernetes clusters consume remote NVMe block storage with latency and queue depth that standard iSCSI or NFS cannot match. The protocol itself is stable. The decision that matters operationally is the transport underneath it: NVMe/TCP or NVMe/RoCE.
Both carry native NVMe commands over the network. Both support the same queue model, up to 65,535 queues per device with up to 65,535 outstanding commands each, which is the primary reason NVMe-oF outperforms iSCSI for database and AI workloads that need deep concurrent I/O. The difference is how the bits move from host to storage node, and that difference has real consequences for hardware budget, network configuration, failover behavior, and the latency floor your applications can actually reach.
How Each Transport Works
NVMe/TCP: Standard Ethernet, No Specialized Hardware
NVMe/TCP encapsulates NVMe commands inside TCP segments and sends them over ordinary Ethernet. The Linux kernel NVMe/TCP driver has been in-tree since kernel 5.0, which covers every major distribution released in the last several years. Initiator and target implementations are stable, multipath failover via NVMe Asymmetric Namespace Access (ANA) works with standard switchports, and you need no hardware beyond the NICs already in your servers.
The cost of using TCP is that the kernel network stack is involved on both the initiator and the target side. Every I/O operation traverses the OS stack twice. That overhead adds roughly 100–400 microseconds of round-trip latency, depending on server CPU speed, NIC driver quality, and whether you tune IRQ affinity and CPU governor settings. For most database workloads, this range is entirely acceptable. PostgreSQL, MySQL, and Kafka all perform well at these latencies because their dominant bottleneck is CPU or lock contention, not raw block latency.
NVMe/RoCE: RDMA on a Lossless Fabric
NVMe/RoCE (RDMA over Converged Ethernet) takes a fundamentally different path. Instead of going through the kernel network stack, it uses Remote Direct Memory Access to transfer data directly between the memory of the storage node and the memory of the compute node, bypassing the operating system on both sides. The kernel does not copy buffers; the NIC DMA-writes data into the target address space directly.
The result is latency in the 20–100 microsecond range, with CPU overhead dramatically lower per I/O operation. The trade-off is infrastructure complexity: RoCE requires NICs that support RDMA (typically Mellanox/NVIDIA ConnectX-series or equivalent), Ethernet switches configured with Priority Flow Control (PFC) to create a lossless fabric, and careful DCQCN (Data Center Quantized Congestion Notification) tuning to prevent congestion collapse. A single misconfigured switch port that allows packet drops under load can cause RDMA queue pair errors that take down storage paths, which is not a failure mode you encounter with TCP.
Performance Compared: Latency, Throughput, and Queue Depth
The headline numbers are useful, but they can mislead if you look at them in isolation. Here is the comparison that actually matters for capacity planning:
| Dimension | NVMe/TCP | NVMe/RoCE |
|---|---|---|
| Round-trip latency | 100–400 µs | 20–100 µs |
| Throughput (25 GbE) | ~2.5 GB/s per link | ~2.5 GB/s per link |
| CPU cost per I/O | Moderate (kernel stack) | Low (kernel bypass) |
| Queue depth | Up to 65,535 per NS | Up to 65,535 per NS |
| Packet-drop sensitivity | Tolerant (TCP retransmit) | High (RDMA path aborts) |
| NIC requirement | Any Ethernet NIC | RoCE-capable NIC (RDMA) |
| Switch requirement | Standard L2/L3 | Lossless, PFC-enabled |
Table 1: Transport comparison for NVMe-oF on Kubernetes storage nodes.
Throughput is roughly equivalent at the same link speed because both transports saturate the same physical wire. The meaningful difference is latency and CPU efficiency. For workloads that issue many small random writes at high queue depth, NVMe/RoCE delivers lower tail latency and frees CPU cycles the application can use instead. For workloads that issue large sequential reads, the bottleneck is almost always network bandwidth, where both transports are equal.
The queue depth advantage is transport-independent: both NVMe/TCP and NVMe/RoCE expose the same NVMe queue model to the application, and that model is what makes NVMe-oF materially better than iSCSI (which caps at one queue per session) for multi-threaded database and AI inference workloads.
Infrastructure and Operational Trade-offs
The transport choice has consequences that show up in procurement, network operations, and failure handling, not just benchmark numbers.
Deployment complexity. An NVMe/TCP cluster runs on equipment you already own. Drop simplyblock storage nodes onto your existing 25 GbE or 100 GbE switches, configure the NVMe/TCP initiator on compute nodes (a single kernel module and a target discovery endpoint), and you are running. NVMe/RoCE requires verifying RoCE support on every NIC in every storage and compute node, enabling PFC on every port and trunk in the storage fabric, and validating DCQCN behavior under congestion. That configuration work takes days on an existing network and needs re-validation after firmware upgrades.
Failure behavior. NVMe/TCP retransmits dropped segments transparently, the same way any TCP connection recovers from transient congestion. An application sees higher latency during a burst, not an error. NVMe/RoCE is unforgiving: a dropped packet in a lossless-fabric sense means a misconfiguration, and the result is a queue pair error that terminates the path. ANA multipath will failover to a surviving path, but the path teardown and reconnect is more abrupt than TCP’s graceful backoff. For clusters where the network operations team has deep RDMA experience, this is a known quantity. For teams whose expertise is in Kubernetes and application delivery, TCP failure behavior is easier to reason about and debug.
Cost. RoCE-capable NICs (ConnectX-6 and above) cost three to five times more than commodity 25 GbE NICs at the same link speed. Lossless-capable switches are similarly priced at a premium over standard L3 switches. For a 10-node storage cluster the delta can reach tens of thousands of euros in NIC and switch costs alone, before counting the operational overhead of maintaining a lossless fabric.
Running stateful workloads on Kubernetes and not sure which fabric fits your latency requirements? simplyblock runs on both NVMe/TCP and NVMe/RoCE, so you can start with TCP and migrate the storage fabric without changing your Kubernetes storage interface. Talk to a storage architect
Choosing the Right Transport for Your Kubernetes Cluster
The decision reduces to a single question: does your workload have a hard latency requirement below 100 microseconds at P99.9?
Most production Kubernetes workloads do not. PostgreSQL with connection pooling, Kafka with replication, Redis Cluster, and most OLAP databases are IO-bound at 200–400 microseconds without degradation. These workloads are excellent candidates for NVMe/TCP because the simpler infrastructure means faster deployment, easier debugging, and lower total cost of ownership.
NVMe/RoCE is justified when latency is a hard architectural constraint: financial trading systems with sub-millisecond order-processing SLAs, high-performance computing (HPC) workloads where MPI jobs are latency-coupled, and AI training jobs where GPU-to-storage stalls are measured in microseconds. In these cases the infrastructure investment and operational discipline of a lossless fabric is the right trade.
A practical path is to start with NVMe/TCP and instrument your workloads. If P99.9 latency from the application layer is consistently above your SLA and storage I/O is the identified bottleneck, then NVMe/RoCE is the upgrade. If the bottleneck is CPU, lock contention, or network bandwidth, switching transports will not move the needle.
How simplyblock Supports Both NVMe/TCP and NVMe/RoCE
Simplyblock is a disaggregated NVMe storage platform that presents persistent volumes to Kubernetes through a standard CSI driver. The storage layer runs off the worker nodes, on dedicated storage nodes, so it never competes with your application workloads for CPU or memory.
Both transports connect to the same storage pool. You configure the transport at cluster deployment time based on your network infrastructure. If your environment has RoCE-capable hardware and a properly configured lossless fabric, you deploy with NVMe/RoCE and reach sub-100-microsecond latency. If you are running standard Ethernet, you deploy with NVMe/TCP and get NVMe-class queue depth and performance without any specialized hardware.
The Kubernetes interface is identical either way: the same StorageClass, the same PersistentVolumeClaim, the same snapshot and thin-provisioning capabilities. A future migration from NVMe/TCP to NVMe/RoCE as your infrastructure evolves does not require changes to application manifests or storage policies.
Questions and Answers
What is the latency difference between NVMe/TCP and NVMe/RoCE in production?
NVMe/TCP delivers 100–400 microseconds of round-trip latency over standard Ethernet. NVMe/RoCE delivers 20–100 microseconds when deployed on a properly configured lossless fabric with RDMA-capable NICs. The gap is meaningful for latency-critical workloads like financial trading systems or tightly coupled HPC jobs, but most database and Kubernetes workloads see no degradation at NVMe/TCP latencies.
Does NVMe/TCP require special hardware?
No. NVMe/TCP works with any standard Ethernet NIC and any L2/L3 Ethernet switch. The initiator driver has been in the Linux kernel since version 5.0. All major Kubernetes-supported Linux distributions (RHEL 8+, Ubuntu 20.04+, Debian 11+) ship with it. No firmware modifications or RDMA support are required on either the compute or the storage side.
Can a Kubernetes cluster use both NVMe/TCP and NVMe/RoCE at the same time?
Simplyblock supports both transports, and you can configure the transport per storage cluster. In practice, most deployments standardize on one transport across all storage nodes to keep the network fabric consistent. Running mixed transports is technically possible but adds operational complexity to path failover and network troubleshooting that most teams prefer to avoid.
Which transport does simplyblock recommend for most Kubernetes deployments?
For teams starting out or running mixed workloads, NVMe/TCP is the recommended default. It runs on existing infrastructure, requires no hardware changes, and delivers NVMe-class queue depth and performance that outperforms iSCSI or NFS for every workload type. NVMe/RoCE is the upgrade path for teams with proven sub-100-microsecond latency requirements and the network operations expertise to run a lossless RDMA fabric reliably.
How does multipath failover work with NVMe-oF on Kubernetes?
Both NVMe/TCP and NVMe/RoCE support NVMe Asymmetric Namespace Access (ANA), which maintains multiple active paths to each volume. If a path fails because a storage node restarts, a NIC fails, or a switch port goes down, the NVMe driver switches to a surviving path automatically. The Kubernetes pod continues running and the application does not receive an I/O error unless all paths fail simultaneously. TCP failover is transparent to a transient congestion event; RoCE failover is triggered by a queue pair error, which is more abrupt but equally handled by the ANA path selection logic.
