Our software engineer, Israel Geoffrey, presented “Building a CSI Driver on SPDK: What We Learned Moving NVMe/TCP Into Kubernetes” at ContainerDays & AI Context Singapore 2026 on October 28 in Singapore.
Kubernetes is declarative, retries everything, and can move a pod to another node at any time. NVMe expects microsecond latency and a stable path to a device that stays put. The CSI driver is where the two meet, and the talk covers what broke when we put them together.
NVMe/TCP and CSI in short
NVMe/TCP carries the native NVMe command set over plain Ethernet. Where iSCSI has one command queue per session, NVMe/TCP has many, each on its own TCP connection. The Linux initiator has been in the kernel since 5.0, so workers need no special hardware.
A volume is an NVMe namespace inside a subsystem. Each worker reaches it through one controller per path. With two or more paths, the kernel’s native multipath merges them into one block device, and ANA (Asymmetric Namespace Access) tells the host which path to prefer.

The CSI driver is two components. A controller plugin handles provisioning, snapshots, clones and expansion. A node plugin runs on every worker and does the real work: connecting NVMe/TCP, formatting, mounting and watching paths. There is no ControllerPublish step. With NVMe/TCP, attaching a volume means the worker opens TCP connections to the target, so it happens on the node. Neither plugin sits in the I/O path. They set it up, watch it and repair it.

Inside NodeStage the driver connects every path, finds the device, then formats and mounts it.
- Connect every path. One
nvme connectper path. “Already connected” counts as success, because the call may be a retry. - Find the device. udev is asynchronous, so the driver waits for it and looks the device up by a stable ID. The
/dev/nvmeXnYnumbering can change across reconnects. - Format and mount. Format only if the device is proven empty.
The connect flags set the policy, and ctrl-loss-tmo matters most: too short and a routine failover kills I/O, infinite and a dead path hangs the pod forever.
Six lessons from moving NVMe/TCP into Kubernetes
1. The kernel outlives your pod
An NVMe/TCP connection is kernel state on the host, not state in the container. When the node plugin restarts or upgrades, the controllers, devices and mounts stay and I/O keeps flowing. The plugin forgets everything, so it never trusts its own memory. It reads sysfs as the source of truth and keeps its records in a directory on the host. That takes real host access: privileged mode, host networking, the host’s /dev, /sys and kernel modules, and bidirectional mount propagation.
2. Host identity is node state
A host is identified by a host NQN and a hostid. The kernel refuses to use one hostid with two different host NQNs, and the connect fails with “found same hostid but different hostnqn”. We hit this when we added access control, which needs a host NQN per Kubernetes node, optionally with DH-HMAC-CHAP authentication, while ordinary volumes on the same node kept the default NQN. The fix was to derive the hostid from the NQN’s own UUID. Kubernetes gives you nothing to manage this, so you have to own it.
3. Connections are a budget
Every queue is its own TCP connection, the admin queue included. Connections equal subsystems × paths × (I/O queues + 1).

Sharing a subsystem cuts connections tenfold, but its namespaces connect and disconnect together, and a path problem hits all of them at once. Set the queue count too. Without nr-io-queues, the Linux host asks for one I/O queue per CPU, so a sixteen-core worker opens 17 connections per path instead of 4. Treat connections as a per-node budget, like CPU and memory.
4. “Connected” is not “working”
The kernel can report a controller as connected while it has no working path. So the node plugin runs its own path monitor, with four rules:
- Poll on a short interval with jitter, so a hundred nodes do not hit the control plane in lockstep.
- Debounce. A normal ANA failover looks degraded for a moment, so recheck before acting.
- Verify from two sources before reconnecting: the control plane says the target is online, and it answers on TCP.
- Back off with a circuit breaker, so the monitor never becomes the outage.
5. Some failures look healthy
Four cost the most debugging time:
- Controller up, namespace gone. I/O hangs for hours with nothing in the logs. Tear down the subsystem and restage.
- “Already connected”, no path. Connect reports success while the controller contributes no path. Detect the empty controller and reset it.
- Mounted, but every read returns EIO. “Is it mounted” is the wrong question. Ask “is it alive”, and restage in repair mode, which never formats.
- Encrypted volume reads as blank. A naive driver formats it. Record the filesystem outside the device, and refuse to format anything you cannot prove is new.
6. Pods leave, volumes stay
NodeUnstage fires every time a pod leaves a node, not only when a volume is deleted, so it must only release. Early on we treated it as cleanup and tore down far too much. Teardown has to finish even if the kubelet gives up on the call, or half-disconnected devices stay behind, and the connection is dropped only when the last namespace leaves the subsystem.
When a device vanishes past a grace period, a watchdog restarts the pods that opted in, with a backoff between restarts. Pods sharing a subsystem restart together, because a stale mount survives a shared reconnect. It is opt-in on purpose: restarting someone’s database is a decision, not a default.
Where CSI stops
CSI is a good contract for the volume lifecycle: create, attach, mount, snapshot, expand, place. It stops at everything that happens between calls. There is no event for “your path is degraded”, and a driver cannot restart a pod through CSI. So the path monitor and the watchdog run beside the driver on the node. Replication and failover come from csi-addons, and consistency groups from a separate controller.
Israel’s advice: keep the CSI surface boring and standard, and put the storage-aware logic beside it, not inside it. All six lessons come down to one rule: assume every call already ran.
You can download the presentation as a PDF.
Related reading: NVMe/TCP storage for Kubernetes, what is a CSI driver and Streamlining Storage Lifecycle Automation in Kubernetes.