After the VMware Exit: Replacing Overlay Storage on Kubernetes with Disaggregated NVMe/TCP

The VMware® exit conversation usually focuses on what you’re leaving: per-core vSAN licensing, Broadcom’s subscription terms, vCenter operational dependencies. What it rarely covers is the storage decision on the other side of the migration. Teams that move from vSphere to Kubernetes without a deliberate storage strategy often end up layering Longhorn, Ceph-Rook, or similar overlay solutions on top of their new compute cluster. That overlay storage approach solves the immediate problem of getting persistent volumes working. It also recreates, in container form, the same hyper-converged coupling that made vSAN expensive to operate.

This is the cost that procurement does not see on a migration proposal: the licensing line item is visible, but the operational and architectural cost of running overlay storage on top of Kubernetes is not. The result is a platform that looks different on paper but behaves the same way under pressure: storage tied to compute nodes, pods constrained in their scheduling, and two sets of snapshot and replication tooling to maintain instead of one.

The Three-Tax Stack VMware Shops Pay When Migrating to Kubernetes

Infrastructure teams migrating off VMware typically encounter three layers of cost that compound each other. The first is vSphere licensing, which under Broadcom’s current model is billed per physical CPU core on every node in the cluster. The second is the Kubernetes distribution fee, whether that is OpenShift, Tanzu, or Rancher Prime layered on top. The third, and the one most often underestimated in the migration plan, is the storage layer.

Overlay storage is what most teams reach for when they need persistent volumes in a new Kubernetes cluster fast. Longhorn installs via Helm. Ceph-Rook ships with a Kubernetes operator. Commercial overlay storage products integrate with the Kubernetes scheduler and expose a CSI driver. All of them work by running storage daemons on the same nodes that run application workloads, pooling local disks across nodes, and presenting them back as block volumes through the CSI interface.

That architecture solves the Kubernetes persistent storage problem, but it introduces a set of coupling constraints that are structurally identical to what vSAN imposes in the VMware world. The storage layer lives on the compute nodes. Scaling compute means scaling storage. Draining a compute node for maintenance involves moving data or constraining pod scheduling. The failure domain of the storage system and the failure domain of the application are the same physical boundary.

The framing that captures this most precisely: overlay storage “doubles the metadata, doubles the snapshot tooling” compared to a purpose-built storage layer. A team that migrated from vSAN to avoid hyper-converged coupling has, by deploying overlay storage on Kubernetes, rebuilt hyper-converged coupling in a different form.

Why Overlay Storage Replicates the vSAN Failure Model

The specific failure modes that drive enterprises to leave vSAN recur with overlay storage in Kubernetes. Understanding the pattern makes the architectural choice clearer.

Compute-storage coupling. In vSAN, every host that runs VMs also carries storage capacity. Adding compute means adding storage whether you need it or not. With overlay storage on Kubernetes, the same coupling applies. Longhorn and Ceph-Rook require storage daemons running on compute nodes to participate in the storage pool. A node without a daemon cannot contribute storage capacity, and a node with a daemon carries storage overhead whether or not any pod on that node needs high-throughput I/O.

Pod scheduling constraints. Kubernetes is built on the assumption that workloads are portable. A pod scheduled on node A should be reschedulable to node B without manual intervention. Overlay storage breaks this: when a pod’s volume lives on a set of replica nodes in a Longhorn or Ceph pool, the scheduler must either keep the pod near its data or tolerate the additional latency of remote reads through the overlay layer. In practice, this creates soft affinities that constrain cluster utilization and make maintenance windows more complex than they appear on the surface.

Snapshot and replication duplication. vSAN provides its own snapshot and replication tooling, integrated with vCenter. When teams move to Kubernetes and deploy overlay storage, they introduce a second snapshot system: Longhorn’s native snapshot mechanism, Ceph-RBD snapshots, or the CSI external snapshotter bundled with commercial overlay storage products. Each of these has its own consistency model, its own backup integration, and its own operational surface. For teams running both legacy VMs on vSphere and new workloads on Kubernetes during a multi-year migration, this duplication is unavoidable, but teams that have completed the migration should not carry two snapshot stacks into steady state.

Operational headcount. vSAN requires certified VMware administrators. Ceph requires Ceph-certified administrators. A team that replaces vSAN with Ceph-Rook has not eliminated the specialist dependency, it has replaced one specialty with another. The day-2 operational profile of Ceph includes placement group management, CRUSH map tuning, OSD rebalancing, and cluster recovery procedures that differ entirely from what a Kubernetes platform team normally operates.

Disaggregated NVMe/TCP as the Architectural Break

Disaggregated storage means the storage layer does not run on the same physical nodes as the application workloads. Storage nodes hold NVMe drives and expose block volumes over a network fabric. Compute nodes mount those volumes through a host driver. The two layers scale independently, fail independently, and are operated independently.

NVMe/TCP is the protocol that makes this practical at commodity hardware cost. It runs over standard Ethernet, does not require specialized networking hardware like InfiniBand, and delivers latency characteristics close to local NVMe because the NVMe protocol itself is designed for sub-millisecond queue depths. A compute node accessing a disaggregated NVMe/TCP volume over a 25GbE or 100GbE switch sees latency that is measurably higher than directly attached NVMe but substantially lower than Ceph-RBD or iSCSI over the same fabric.

The CSI integration for disaggregated NVMe/TCP storage presents the storage layer to Kubernetes exactly as overlay storage does: as a StorageClass, with PVC semantics, volume expansion, and VolumeSnapshot support. Pods do not see a difference in the storage API. What they see differently is scheduling freedom. Because the volume is not replicated across compute nodes, there is no node affinity imposed by the storage layer. A pod can reschedule to any node in the cluster after a node drain, and the volume reattaches over the network without data movement.

Planning your Kubernetes storage architecture after leaving VMware? Simplyblock’s architecture team works directly with infrastructure teams during VMware exits to design a storage layer that does not recreate the coupling problem. Talk to a storage architect

For KubeVirt and OpenShift Virtualization environments, the disaggregated model has a specific operational advantage: VM live migration. A VM running on KubeVirt stores its disk as a PVC. If that PVC is backed by overlay storage with node-local replicas, live migration requires transferring replica data before the VM can move. With disaggregated NVMe/TCP, the volume is already network-attached, so live migration completes without data movement: only the VM’s compute context migrates, not its storage.

Choosing the Right Storage Layer: A Decision Table

The overlay vs. disaggregated decision is not binary in every environment. The right choice depends on what Kubernetes workloads you run, what your scaling model looks like, and what operational capabilities your team has. The table below maps the key decision dimensions.

Table 1: Storage architecture decision dimensions for Kubernetes clusters in post-VMware environments.

Simplyblock’s Position in the VMware Migration Stack

Simplyblock is licensed per usable storage TB provisioned, not per CPU core and not per Kubernetes node. For teams coming from vSAN’s per-core billing model, this is the structural break. A compute-heavy scale-out (adding nodes to handle more pods or VMs) does not increase the storage license cost. A storage-heavy expansion (adding capacity to absorb data growth) does not require adding compute capacity. The two budgets stay separate.

The architecture is hypervisor-agnostic by design. The same simplyblock storage cluster can serve vSphere VMs over NVMe/TCP, KubeVirt pods on bare-metal Kubernetes, and OpenShift Virtualization VMs on the same fabric simultaneously. Teams running a multi-year VMware exit program that moves workloads progressively from vSphere to Kubernetes do not need to maintain two storage layers during the transition: one storage cluster covers both platforms while the migration runs.

The migration pattern most commonly used is additive: disaggregated NVMe/TCP is introduced as a new storage tier for Kubernetes PVCs and KubeVirt disks, while vSAN continues serving legacy VMs until those workloads migrate or are decommissioned. This avoids a cutover event and aligns the storage transition with the workload migration schedule rather than forcing them to synchronize. For the detailed cost math on what the vSAN side of this looks like, the vSAN exit cost economics post models the per-core numbers at 200 cores. For the KubeVirt-specific storage architecture, see the KubeVirt persistent storage guide.

Questions and Answers

What is overlay storage in Kubernetes, and why is it a problem for VMware migration teams?

Overlay storage refers to software-defined storage systems that run as daemons on the same nodes that host application workloads, pooling local disks into a shared volume layer. Longhorn and Ceph-Rook are the most widely used open-source examples; commercial overlay products follow the same pattern. They solve the persistent volume problem but introduce hyper-converged coupling: storage and compute share the same physical nodes, the same failure domain, and the same scaling constraints. For teams migrating from VMware specifically because vSAN’s hyper-converged model made the platform expensive to scale and operate, overlay storage recreates that same pattern in Kubernetes form. The compute-storage coupling persists; only the vendor and the API change.

How does disaggregated NVMe/TCP eliminate the overlay storage tax?

Disaggregated NVMe/TCP places storage on physically separate nodes from compute. Storage nodes expose block volumes over NVMe/TCP, a protocol that runs on standard Ethernet. Compute nodes mount those volumes via a host driver and the CSI interface. Because the storage layer does not run on compute nodes, pods are not constrained in their scheduling by data locality. Storage nodes can be scaled independently when capacity grows, and compute nodes can be scaled independently when workload density increases. There is no storage daemon overhead on compute nodes, no data-local replica management, and no node affinity imposed by the storage layer.

Is simplyblock compatible with both Kubernetes and OpenShift during a phased VMware migration?

Yes. Simplyblock uses a standard CSI driver and exposes storage as Kubernetes PVCs and StorageClasses, which are supported natively by both upstream Kubernetes distributions and Red Hat® OpenShift®. A single simplyblock storage cluster can serve both Kubernetes and OpenShift clusters simultaneously. For KubeVirt and OpenShift Virtualization, simplyblock supports RWX (ReadWriteMany) block volumes, which are required for VM live migration. This means a team can point new OpenShift Virtualization workloads at simplyblock while legacy VMware VMs continue running on vSAN, then migrate storage progressively as workloads move.

How does simplyblock compare to Longhorn for post-VMware Kubernetes storage?

Longhorn is a CNCF-graduated storage project that runs as a daemon on Kubernetes nodes, replicating data across a configurable number of node-local replicas. It is free and well-integrated with Kubernetes. The operational trade-offs are the overlay storage constraints described in this post: pod scheduling is soft-constrained by replica locality, storage and compute scale together, and Longhorn’s replication protocol adds CPU and network overhead on compute nodes. Simplyblock runs on separate storage nodes, does not impose scheduling constraints, and licenses per usable TB rather than per node. For teams that already have Longhorn running and it is meeting their performance and operational requirements, the switching cost may not be justified. For teams that are designing the Kubernetes storage layer as part of a VMware exit, starting with disaggregated NVMe/TCP avoids accumulating the coupling debt that overlay storage introduces.

What does the phased VMware migration look like when storage is disaggregated?

The most common pattern is additive: simplyblock is deployed as a new storage tier serving Kubernetes PVCs and KubeVirt or OpenShift Virtualization VM disks. Legacy VMware VMs continue on vSAN until they are migrated or decommissioned. No cutover of existing storage is required. Workloads migrate progressively: a VM is migrated to OpenShift Virtualization, its disk is provisioned from simplyblock, and the vSAN volume for that VM is freed. Over a migration program of six to eighteen months, the vSAN footprint shrinks and the simplyblock footprint grows in proportion to the workloads migrated. This model aligns the storage transition with the workload schedule rather than requiring a separate, parallel storage migration event.