Rob Pankow 
The AI inference storage guide covers what happens when a serving pod loads model weights, and the training checkpoint guide covers what happens when a training job writes state to disk. Milvus sits at a third point in the same pipeline: the retrieval step, where an application queries a vector index built from embeddings to find the nearest matches for a request in flight. That query path has its own storage requirements, and they do not match what most teams provision by default.
Milvus’s own architecture separates data into growing (in-memory and local-disk) segments actively being written, a write-ahead log for durability, and sealed segments pushed to object storage once a segment closes. The official deployment guidance treats S3-compatible storage as the tier for all of it. That works for the sealed, cold segments. It is the wrong tier for the growing segments and WAL, where every millisecond of query latency during a high-QPS similarity search comes directly out of application response time.
Why Milvus’s Segment-Based Storage Breaks Generic Kubernetes Volumes
Milvus does not store vectors as a single flat index. It splits a collection into segments, and each segment moves through a lifecycle that has very different I/O characteristics at each stage:
Growing segments hold newly inserted vectors that have not yet been indexed and compacted. These live in memory and on local disk on the query node, and they are read constantly during a similarity search because a query has to check both the sealed index and any recent, not-yet-sealed data to return accurate results.
The write-ahead log durably records every insert and delete before it is acknowledged. Milvus’s log broker (commonly Pulsar or Kafka, depending on deployment mode) depends on fast, low-latency disk for this log, the same write-ahead logging durability pattern used by database commit logs: a slow disk here adds latency to every write, not just large ones.
Sealed segments are immutable once a segment closes and gets its final index built. These are read far less frequently, at larger sequential block sizes, and are the segments Milvus’s own documentation points at S3-compatible object storage for.
Provisioning a single generic StorageClass for all three tiers means either over-provisioning expensive low-latency storage for cold sealed segments, or under-provisioning latency-sensitive storage for the growing segments and WAL that actually gate query response time. Neither is the right default.
Milvus Storage Tiers: Where Each Layer Should Actually Live
The table below maps each Milvus storage tier to the backend it needs, using a representative high-QPS similarity search workload as the reference point.
| Milvus storage tier | I/O pattern | Latency sensitivity | Best-fit backend |
|---|---|---|---|
| Growing segments (query node) | Small random reads, continuous | High, directly gates query latency | Disaggregated NVMe/TCP or NVMe/RoCE block storage |
| Write-ahead log (message broker) | Sequential append, fsync-bound | High, gates write acknowledgment | Same NVMe/TCP tier, low fixed p99 |
| Sealed segment index cache | Random reads on recently-queried segments | Medium, affects cold-query latency | Same tier, with a cache policy layered on top |
| Sealed segment cold storage | Large sequential reads, infrequent | Low | S3-compatible object storage |
Table 1: Milvus storage tier characteristics and the backend each tier fits best.
Milvus’s own guidance is correct for the bottom row and incomplete for the top three. A disaggregated storage architecture puts the growing segments, WAL, and a recently-queried-segment cache on a shared NVMe fabric separate from the query node’s local disk, so a query node restart does not lose in-flight segment state and a query node under memory pressure can spill to fast network storage instead of local disk that may not exist at the size needed.
A slow write-ahead log turns into slow inserts across the entire collection. Talk to a storage architect about sizing an NVMe/TCP tier for Milvus’s growing segments and WAL, separate from your S3-compatible cold tier. Talk to a storage architect
How Disaggregated NVMe/TCP Closes the Gap Between Hot Index I/O and Cold Tiering
Running Milvus’s query nodes and index nodes with persistent volumes backed by disaggregated NVMe/TCP (or NVMe/RoCE on RDMA-capable fabrics) addresses the two problems the object-storage-only guidance leaves open:
Query node restarts do not lose growing-segment state. Because the volume backing the growing segments and local WAL replay area lives on the storage fabric rather than node-local disk, a rescheduled query node pod reattaches the same PVC and resumes without a full segment reload from object storage, which would otherwise mean a cold-cache query node serving degraded latency until the working set rebuilds.
Sealed-segment index caching gets a real second tier. Instead of every query touching cold object storage for any segment not already resident in query-node memory, a storage layer sitting between memory and S3 caches the recently-queried sealed segments at network-fabric latency, well below what an S3-compatible endpoint returns on a cache miss.
Milvus deployments also inherit the same snapshot-based backup and rollback pattern already covered for CSI snapshot architecture: a VolumeSnapshot of the growing-segment and WAL volumes gives a point-in-time, storage-side capture of in-flight collection state, restorable in seconds, rather than relying solely on Milvus’s own backup tool to export and re-import segment data through object storage. For teams running multiple collections across environments, a VolumeSnapshotClass per collection tier (frequent, short-retention snapshots for actively-written collections; longer-retention snapshots for stable, production collections) mirrors the pattern already in production for database and training-checkpoint volumes.
Simplyblock licenses per usable storage TB per year, with no per-core fee, and no charge tied to the size of the query-node fleet. A team that scales query nodes horizontally to handle higher QPS does not pay more for the storage tier unless the actual volume of growing-segment, WAL, and cached-segment data grows. For the object-storage side of a Milvus deployment, the AI storage companies overview covers how block and object platforms typically split responsibilities in a mixed AI infrastructure stack, and the scale-out AI storage guide covers sizing a cluster across inference, training, and retrieval workloads together.
Questions and Answers
Why does Milvus need block storage in addition to S3-compatible object storage? Milvus’s own architecture separates data into growing segments, a write-ahead log, and sealed segments. Only the sealed segments are a good fit for S3-compatible object storage: the growing segments and WAL are latency-sensitive, fsync-bound, and read continuously during every similarity search, so they need low-latency block storage rather than an object-storage endpoint built for infrequent, larger reads.
What happens if a Milvus query node restarts on generic Kubernetes storage? If the growing segments and local WAL replay area live on node-local disk or a volume without fast reattachment, a rescheduled query node pod has to rebuild its in-memory working set from object storage before it can serve queries at full accuracy and speed. A persistent volume claim backed by simplyblock’s NVMe/TCP storage reattaches to the new pod in seconds without that cold-cache rebuild.
Can Milvus’s own backup tool replace CSI snapshots for disaster recovery? Milvus’s backup tool exports and re-imports collection data through object storage, which works for scheduled, infrequent backups but adds a full data-movement step to every backup and restore cycle. A CSI volume snapshot of the underlying growing-segment and WAL volumes is a storage-side, near-instant point-in-time capture, and restoring it is a volume-level operation rather than a re-import through the object-storage path.
How does simplyblock fit into a Milvus deployment on Kubernetes? Simplyblock provides NVMe/TCP (or NVMe/RoCE) block storage as persistent volumes for Milvus’s query nodes and index nodes, covering the growing-segment, WAL, and sealed-segment cache tiers that S3-compatible object storage alone cannot serve at the latency a high-QPS similarity search needs. Simplyblock’s CSI driver adds fast, storage-side snapshots for collection-level backup and rollback, alongside whatever object-storage backend handles Milvus’s cold sealed-segment tier.
How should I size storage for a Milvus deployment on Kubernetes? Size the NVMe/TCP tier for the working set of growing segments plus the WAL retention window across all query nodes, and add headroom for a sealed-segment cache sized to the collections queried most frequently. Cold, infrequently-queried sealed segments can stay on S3-compatible object storage. Simplyblock’s pricing is per usable TB per year with no per-core fee, so scaling the query-node fleet horizontally for higher QPS is a compute decision, not an automatic storage cost increase.
