AI Inference Storage on Kubernetes: Why Model Weight Loading Needs NVMe/TCP

Organizations leaving VMware for Kubernetes are asking a question that did not appear on the original migration checklist: will the new platform handle AI workloads? The infrastructure that replaced vSphere handles stateless services and databases well. Inference workloads expose a gap that most teams do not anticipate until a GPU node goes down and the model takes twelve minutes to come back online.

The gap is a storage problem. Inference pods load large model weight files before they can serve a single request. They accumulate KV-cache state across requests. They restart on nodes that may not have the weights available locally. Each of these behaviors makes inference pods some of the most storage-sensitive workloads in a Kubernetes cluster, and standard hyperconverged or object-backed storage architectures are not designed for them.

This post covers the specific storage requirements that AI inference imposes on Kubernetes, where common architectures fall short, and how disaggregated NVMe/TCP storage removes the bottlenecks.

Why AI Inference Pods Are Stateful Workloads

The Kubernetes mental model treats pods as ephemeral and stateless by default. Inference pods break this model in at least three ways.

Model weights are large and must be loaded before serving. A 7-billion-parameter model in bfloat16 precision occupies roughly 14 GB on disk. A 70-billion-parameter model occupies 140 GB. Before the inference process can answer a single request, it must read the entire weight file into GPU memory. From object storage, this can take five to ten minutes per pod, depending on network throughput and object-storage latency characteristics. During that window, the pod is alive in Kubernetes terms but unable to serve traffic, and the HPA cannot distinguish it from a healthy replica.

KV-cache state accumulates per session. Large language model inference caches the key-value attention tensors for tokens already processed in a conversation. This cache grows with each turn and allows subsequent requests in the same session to avoid recomputing attention from scratch. When the pod that holds the cache is evicted or rescheduled, the cache is lost. The next request restarts computation from token zero, increasing latency and GPU utilization for long sessions.

Pod rescheduling onto a new node restarts the weight-load cycle. If model weights are stored on node-local disk, a pod rescheduled to a different node must re-download or re-copy the full weight file before serving. This creates a locality dependency: the scheduler must place the inference pod on the node that has the weight cache, which conflicts with the bin-packing logic that Kubernetes uses to optimize GPU utilization across the cluster.

These three behaviors mean that inference pods have a persistent, latency-sensitive dependency on storage even though they do not use databases or maintain traditional application state. The persistent volume is not optional; it is the mechanism that decouples weight storage from node locality and makes pod rescheduling fast.

The Model Weight Cold Start Bottleneck

Cold start time is the elapsed duration between a pod being scheduled and the first inference token leaving the process. On a Kubernetes cluster where model weights are not on a fast persistent volume, cold start time is dominated by the weight-load step.

The table below shows how different storage approaches affect cold start time and autoscaling behavior for a representative 14 GB model (7B parameters, bf16):

Table 1: Storage approach impact on AI inference cold start time and scheduling constraints for a 14 GB model weight file.

The locality column is the key constraint. When node-local disk is the weight-storage mechanism, every rescheduling event requires either placing the pod back on the same node or reloading the weights from scratch. This behavior conflicts with horizontal pod autoscaling: a load spike that triggers a new replica requires a new GPU node to have the weights ready, which it usually does not. The new replica sits in a degraded state for several minutes while weights load, defeating the purpose of autoscaling.

Disaggregated storage breaks the locality coupling. The model weight file lives on the storage cluster as a persistent volume claim. When a new inference pod starts on any GPU node in the cluster, it attaches the PVC over NVMe/TCP and starts reading weights at fabric speed. The weights are not transferred between nodes; the GPU node reads directly from the disaggregated storage cluster over the NVMe/TCP fabric. For a storage cluster with multiple NVMe devices in a RAID-equivalent layout, sequential read throughput for a 14 GB weight file takes seconds, not minutes.

AI inference cold starts are a storage architecture problem, not a GPU problem. Talk to a storage architect about sizing a disaggregated NVMe/TCP cluster for your inference fleet. Talk to a storage architect

KV-Cache Persistence: The Hidden Stateful Dependency in LLM Inference

KV-cache is one of the less-discussed storage dependencies in inference deployments. Most architecture discussions focus on model weights as the primary storage concern. In production, KV-cache becomes equally important as session length grows.

A KV-cache stores the computed attention key and value tensors for each token already processed in a conversation. For a 7B model with a 4096-token context window, the KV-cache for a fully populated context occupies several gigabytes per session. Inference servers that support long-context workloads or multi-turn conversations accumulate KV-cache in GPU VRAM and, when VRAM fills, spill to CPU memory or local disk.

The storage implications have two dimensions.

Cache loss on pod eviction. If the inference pod is evicted by the scheduler (due to memory pressure, node drain, or a resource limit violation), the in-process KV-cache is gone. The next request in the session restores from scratch, recomputing attention for the full context. For long sessions this is both slow (increased time to first token) and computationally expensive (GPU cycles already spent are wasted).

Cache portability for load balancing. When a session is routed to a different inference replica, the receiving replica does not have the KV-cache for the conversation. Without a persistent, shared cache store, each replica must maintain its own cache, which prevents effective load balancing across replicas for stateful sessions.

A persistent volume backed by NVMe/TCP addresses both dimensions. The KV-cache can be written to a PVC that survives pod eviction and can be re-attached by a replacement pod on any node. The sub-millisecond latency of NVMe/TCP makes spilling the KV-cache to a PVC a viable alternative to CPU memory, especially for models with large context windows. IOPS on a multi-NVMe disaggregated cluster are sufficient to sustain cache read and write throughput without adding measurable latency to the inference pipeline.

How Disaggregated NVMe/TCP Storage Solves the Inference Storage Problem

The architectural difference between hyperconverged and disaggregated storage becomes most visible at the inference layer, because inference combines three demanding properties: high sequential throughput for weight loading, low latency for KV-cache access, and no tolerance for scheduling constraints that limit GPU utilization.

Hyperconverged storage runs storage software on the same nodes as compute workloads, including GPU nodes. This design allocates a portion of each node’s CPU cycles, memory, and network bandwidth to the storage layer. For CPU-bound workloads this overhead is often acceptable. For GPU inference workloads it is not: GPU nodes are already resource-constrained (VRAM, PCIe bandwidth, host memory for CUDA context), and a storage agent consuming host memory or CPU cycles directly reduces the inference headroom on that node.

Disaggregated NVMe/TCP separates the storage cluster from the compute cluster entirely. GPU nodes run no storage software; they connect to the storage cluster over a standard TCP network using the NVMe/TCP protocol. From the GPU node’s perspective, the model weight PVC is a block device that delivers IOPS at near-local-NVMe speeds with sub-millisecond latency. The storage cluster handles replication, tiering, and failure recovery independently, without consuming resources on GPU nodes.

This separation has three concrete effects on inference workloads:

No locality pinning. Any GPU node can attach any model weight PVC. Pod rescheduling events, node failures, and autoscaling events all resolve to a PVC re-attach, which takes seconds. The Kubernetes scheduler can place inference pods anywhere in the GPU pool without considering which nodes have which weights cached locally.

Full VRAM for inference. Because no storage agent runs on GPU nodes, the full host memory and CPU resources are available to the CUDA runtime and inference server. There is no memory balloon from a storage driver consuming headroom on a node where every gigabyte of host memory matters for model loading.

Autoscaling becomes effective. A new replica on a node that has never seen this model attaches the weight PVC and starts loading in seconds over NVMe/TCP. This means HPA can actually deliver latency reduction under load spikes, rather than adding replicas that spend several minutes loading before they can serve traffic. For teams building AI workloads on OpenShift or running inference in multi-tenant Kubernetes environments, effective autoscaling is a prerequisite for predictable SLAs.

For teams evaluating storage for GPU clusters, NVMe/TCP vs NVMe/RoCE covers the fabric selection decision in detail. NVMe/TCP runs over standard Ethernet and requires no specialized hardware beyond the existing cluster network, making it the lower-friction choice for most Kubernetes deployments. NVMe/RoCE reduces protocol overhead and is appropriate when every microsecond of inference latency counts.

The scale-out AI storage guide covers sizing a storage cluster for AI workloads, including how to calculate IOPS and throughput requirements from model size and replica count. For organizations migrating from VMware to Kubernetes for AI workloads, the overlay storage replacement guide covers the storage architecture decision at the destination side of the migration.

Simplyblock licenses per usable storage TB per year with no per-core fee. This matters for GPU inference deployments specifically: as the GPU cluster grows, storage cost scales with the amount of model data stored, not with the number of GPU cores added. A cluster running twenty GPU nodes with the same weight set stored once costs the same for storage as a cluster running five GPU nodes. The pricing model rewards density, which is the direction GPU inference deployments scale toward.

For teams building toward fully agentic Kubernetes deployments, making storage agentic-ready covers the additional requirements that multi-agent inference architectures introduce, including parallel PVC access patterns and rapid PVC cloning for model experiment isolation.

Questions and Answers

What is the cold start problem in Kubernetes AI inference? Cold start time is the delay between a pod being scheduled and its first successful inference response. The dominant component is model weight loading: the process of reading a large weight file (often 14 GB to 140 GB) from storage into GPU memory. On object storage or with weights on node-local disk that requires re-copying after a pod reschedule, this delay can reach five to thirty minutes per replica. Disaggregated NVMe/TCP storage reduces it to seconds by allowing any GPU node to attach the weight PVC directly and read at block-storage speeds over the NVMe/TCP fabric.

Why does GPU-node locality pinning hurt inference autoscaling? Locality pinning means the Kubernetes scheduler must place an inference pod on a node that already has model weights available locally. When a new replica is needed to handle a traffic spike, the scheduler must find a node with the weights, or wait for the weights to load from scratch on a new node. Both outcomes delay effective scale-out. Disaggregated NVMe/TCP removes the locality requirement entirely: any node can mount any weight PVC in seconds, so the scheduler is free to place pods anywhere in the GPU pool. Simplyblock implements this through standard Kubernetes CSI semantics, with no custom scheduler plugins required.

How does simplyblock handle KV-cache persistence in LLM inference? KV-cache can be written to a PVC backed by simplyblock NVMe/TCP storage. The low latency of the NVMe/TCP fabric makes spilling to a PVC viable: write throughput to a simplyblock volume is sufficient to keep up with KV-cache eviction from VRAM or CPU memory. If the inference pod is evicted and replaced, the new pod attaches the same PVC and the KV-cache is available without recomputation. This is especially valuable for long-context workloads where recomputing attention from token zero represents significant GPU-cycle waste.

Can I run inference on Kubernetes without GPU-node locality pinning using simplyblock? Yes. Simplyblock’s CSI driver implements standard ReadWriteOnce PVC semantics over NVMe/TCP. The PVC attaches to whichever node the pod is scheduled on; no affinity rules or node selectors tied to storage are required. For multi-replica inference deployments where multiple pods serve the same model simultaneously, a ReadWriteMany-capable configuration allows multiple pods to read the same weight PVC concurrently from different GPU nodes.

What storage sizing should I use for an inference fleet on Kubernetes? Start with the total uncompressed weight size of all model variants that will be live simultaneously, then multiply by your desired snapshot and warm-spare factor. A reasonable starting point for a fleet serving three model variants is: (sum of weight file sizes) times 1.5 for snapshots and clones, plus dedicated capacity for KV-cache PVCs per replica. The AI storage sizing guide covers this calculation in detail. Simplyblock’s pricing is per usable TB per year with no per-core fee, so cost scales with stored weight data rather than with GPU count.