OpenShift Virtualization Storage Architecture: Disaggregated NVMe for Live Migration and Multi-Tenancy

Red Hat® OpenShift® Virtualization builds VM workload support directly into OpenShift using KubeVirt as the execution engine. Each VM runs as a VirtualMachineInstance with its persistent disk backed by a DataVolume, which is a standard Kubernetes persistent volume provisioned by a CSI driver. The OpenShift control plane manages VM lifecycle exactly as it manages containers: same scheduler, same networking, same storage primitives.

This architectural choice creates two specific storage requirements that generic Kubernetes storage guides miss. VM live migration requires ReadWriteMany block access: two nodes must simultaneously hold an active attachment to the VM’s disk during the migration window. And the multi-tenant cluster pattern, where tenant Kubernetes clusters run as VMs inside OpenShift Virtualization, means each tenant cluster needs CSI-accessible persistent storage that works through the virtualization layer. Both requirements eliminate a range of storage approaches that work perfectly well for containerized workloads.

Why OpenShift Virtualization Storage Differs from Generic KubeVirt Storage

OpenShift Virtualization and KubeVirt share the same execution engine, but they serve different buyers with different storage expectations. KubeVirt on a generic Kubernetes cluster is a developer tool for running legacy VMs alongside containers on existing infrastructure. OpenShift Virtualization is a production VM platform positioned as the destination for VMware-exit migrations. Enterprises moving from vSphere expect live migration, per-VM storage policy management, and enterprise-grade data protection from the same platform that hosts their containerized workloads.

The storage consequence of that positioning is significant. A VM estate migrated from VMware carries implicit assumptions: VMs live-migrate without downtime during host maintenance, storage policies apply per-VM rather than cluster-wide, and individual VMs access independent block volumes that behave like directly attached SAN LUNs. Meeting those expectations on OpenShift Virtualization requires storage designed for block-level VM access, not storage originally built for container filesystem volumes.

KubeVirt storage in a generic cluster often uses local NVMe via hostPath or local PersistentVolumes. That approach pins each VM to the node where its disk lives and blocks live migration. For a developer environment, that trade-off is acceptable. For a production VMware-exit target running hundreds of VMs with maintenance windows and zero-downtime requirements, it is not.

Live Migration Storage Requirements: ReadWriteMany Block Is Non-Negotiable

VM live migration on OpenShift Virtualization follows the same protocol as vSphere vMotion: the guest operating system continues running while its memory state copies to the destination node. The migration controller iterates over dirty memory pages until the residual delta is small enough to complete the final cutover in milliseconds. Throughout this process, the VM’s disk must remain accessible from both the source node and the destination node simultaneously.

That simultaneous dual-node attachment is the storage requirement that creates problems for most storage architectures. Standard block storage volumes use ReadWriteOnce access mode: exactly one node holds the attachment at any given time. To live-migrate a VM backed by a ReadWriteOnce volume, the migration controller would have to detach the volume from the source, attach it to the destination, and then restart the VM. That detach-reattach sequence terminates the guest OS, turning live migration into a cold restart with data-movement cost on top.

ReadWriteMany block access solves this. The volume remains simultaneously attached to both the source and destination nodes throughout the migration window. The source node continues serving the VM’s disk I/O while the memory copy proceeds. When the final cutover completes, the source detaches and the destination takes full ownership. From the guest OS perspective, the disk never went offline.

RWX for block volumes is technically distinct from RWX for file volumes (NFS or CephFS). A file volume serves simultaneous reads and writes through filesystem-layer locking semantics. A block volume at the raw device level has no inherent locking mechanism: two nodes writing to the same raw block device concurrently would corrupt it. The reason RWX block works for live migration is that only one VM is ever issuing writes at a time. The OpenShift Virtualization migration controller enforces a single-writer protocol: the source node holds the active write attachment during migration; the destination holds a read-capable connection until final cutover, at which point roles switch atomically. The storage layer supports simultaneous connections from two nodes; the controller ensures write exclusivity.

Running a VMware exit onto OpenShift Virtualization and evaluating storage for live migration? Simplyblock’s disaggregated NVMe/TCP storage provides RWX block access using commodity NVMe hardware, without a proprietary storage array dependency. Talk to a storage architect

Multi-Tenant Kubernetes Clusters on OpenShift Virtualization

A more recent pattern extends OpenShift Virtualization beyond migrated VMware workloads. Enterprises are deploying tenant Kubernetes clusters as VMs inside OpenShift Virtualization: each tenant cluster is a set of VMs managed by the OpenShift Virtualization control plane, running a lightweight Kubernetes distribution that the tenant team operates independently.

This pattern solves a platform governance problem. The infrastructure team manages one OpenShift cluster with controlled VM templates, network policies, and resource quotas. Each development team gets a Kubernetes cluster with full root access to their VMs, without access to the underlying OpenShift infrastructure. VM isolation provides the security boundary that namespace isolation in a shared cluster does not.

The storage challenge for this pattern is that each tenant cluster’s workloads need persistent storage from within the VM. A database pod running inside the tenant cluster issues a CSI CreateVolume call. The CSI driver in the tenant cluster’s control plane must provision a volume, attach it to the correct tenant VM, and expose it as a block device to the pod. That entire operation happens inside the guest operating system, through the network stack of the VM, without any direct access to the host OpenShift Virtualization storage layer.

Storage that requires a proprietary backend faces a practical barrier here: the tenant cluster administrator may not have access to that vendor’s hardware, may not hold the licensing for additional nodes, or may be deploying in a location where that hardware is unavailable. Disaggregated NVMe/TCP removes that barrier. The simplyblock CSI driver runs inside the tenant VM, issues standard NVMe/TCP connection requests over the VM’s network interface, and receives block volumes from the simplyblock storage pool. The storage pool is the same one that backs the OpenShift Virtualization VMs themselves. No additional vendor hardware is required for tenant cluster persistent storage.

Storage Architecture Options for OpenShift Virtualization

The following table compares the three primary storage approaches for OpenShift Virtualization, evaluated against the requirements that matter for production VMware-exit deployments.

Table 1: Storage architecture comparison for OpenShift Virtualization production deployments.

ODF / Rook-Ceph: OpenShift Data Foundation is the default storage option bundled with OpenShift. It runs Ceph storage software on compute nodes and provides block, file, and object storage through a unified pool. For live migration, ODF supports RWX block via Ceph RBD. The constraint is topology: Ceph replicates each volume across a subset of nodes (default three-way replication). Live migration can only target nodes that are already replica holders without triggering a background re-sync. For clusters with high VM density and frequent maintenance windows, this limits eligible migration targets at any given time.

Proprietary array-backed storage: Vendor block storage solutions that integrate with OpenShift Virtualization typically provide RWX block volumes with low latency and mature enterprise data services. The operational constraint is vendor dependency: every deployment site, including each tenant cluster in the multi-tenant pattern, requires access to the vendor’s specific storage hardware and licensing. For multi-site or cloud-adjacent deployments, this creates procurement and logistics constraints.

Disaggregated NVMe/TCP (simplyblock): Simplyblock runs storage software on dedicated NVMe nodes and exposes volumes over the network using standard NVMe/TCP. Every compute node in the OpenShift cluster connects to the same storage pool over Ethernet. RWX block is available to every node simultaneously because NVMe/TCP supports multiple initiators connecting to the same namespace. Live migration is unrestricted: any compute node can attach any volume without a data movement step, because the storage pool is not co-located with any specific compute node. Tenant CSI drivers in guest VMs connect the same way over the VM’s network interface.

Configuring simplyblock for OpenShift Virtualization

Simplyblock integrates with OpenShift Virtualization through a standard CSI driver installed at the cluster level. Two StorageClass configurations cover the primary use cases.

StorageClass for VM live migration (RWX block):

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: simplyblock-vm-rwx
5
provisioner: csi.simplyblock.io
6
parameters:
7
  replication: "2"
8
  qos_rw_iops: "10000"
9
  qos_rw_mbytes: "500"
10
volumeBindingMode: Immediate
11
allowVolumeExpansion: true

This StorageClass provisions block volumes with two-way replication. The Immediate binding mode ensures volumes are provisioned as soon as the DataVolume is created, without waiting for a pod to schedule. OpenShift Virtualization’s VM import and clone operations require Immediate binding to complete the CDI (Containerized Data Importer) transfer before the VM starts.

StorageClass for tenant cluster workloads (ReadWriteOnce, provisioned from within a tenant VM):

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: simplyblock-tenant
5
provisioner: csi.simplyblock.io
6
parameters:
7
  replication: "2"
8
  qos_rw_iops: "5000"
9
  qos_rw_mbytes: "200"
10
volumeBindingMode: WaitForFirstConsumer
11
allowVolumeExpansion: true

Tenant cluster workloads typically use ReadWriteOnce volumes attached to one pod at a time. WaitForFirstConsumer ensures the volume is provisioned in the storage zone closest to the VM that will consume it, reducing cross-zone traffic when NVMe/TCP nodes are distributed across failure domains.

VirtualMachine DataVolume referencing the RWX StorageClass:

1
apiVersion: kubevirt.io/v1
2
kind: VirtualMachine
3
metadata:
4
  name: production-vm
5
spec:
6
  running: true
7
  template:
8
    spec:
9
      domain:
10
        devices:
11
          disks:
12
            - name: rootdisk
13
              disk:
14
                bus: virtio
15
      volumes:
16
        - name: rootdisk
17
          dataVolume:
18
            name: production-vm-root
19
  dataVolumeTemplates:
20
    - metadata:
21
        name: production-vm-root
22
      spec:
23
        storage:
24
          accessModes:
25
            - ReadWriteMany
26
          volumeMode: Block
27
          storageClassName: simplyblock-vm-rwx
28
          resources:
29
            requests:
30
              storage: 100Gi

The volumeMode: Block and accessModes: [ReadWriteMany] combination is the critical configuration. OpenShift Virtualization’s migration controller checks the DataVolume’s access mode before initiating the memory copy phase: a ReadWriteOnce volume causes the migration to fall back to cold restart rather than live migration.

Questions and Answers

What storage access mode does OpenShift Virtualization require for VM live migration?

OpenShift Virtualization live migration requires ReadWriteMany (RWX) with volumeMode: Block. The migration controller checks the DataVolume’s access mode before initiating the memory copy: if the volume uses ReadWriteOnce, live migration is blocked and the platform falls back to cold restart on the destination node. RWX block means both the source and destination node hold an active block device attachment simultaneously during the migration window. NVMe/TCP enables this through multi-initiator namespace access at the storage layer, with the migration controller enforcing write exclusivity between nodes.

Can simplyblock storage be provisioned from inside a tenant Kubernetes cluster running as a VM on OpenShift Virtualization?

Yes. The simplyblock CSI driver runs inside the guest VM’s Kubernetes control plane and issues NVMe/TCP connection requests over the VM’s standard network interface. From the storage pool’s perspective, the guest VM is another NVMe/TCP initiator. The CSI driver provisions volumes, attaches them to the correct tenant pod, and manages snapshots and resizes without requiring access to the host OpenShift Virtualization layer. Tenant teams get the same storage capabilities as the host cluster (dynamic provisioning, volume expansion, snapshots) without a proprietary storage array or special hardware in the tenant VM.

How does disaggregated NVMe/TCP compare to ODF for OpenShift Virtualization live migration?

ODF provides RWX block via Ceph RBD and supports live migration, but within topology constraints. Because Ceph replicates each volume to a subset of nodes, live migration can only target nodes that already hold a replica without triggering a re-sync. For clusters under high VM density or heterogeneous node configurations, this limits eligible migration targets at any given time. Disaggregated NVMe/TCP removes the topology constraint: the storage pool is physically separate from compute, every node connects over the network, and any node is eligible for any migration at any time. There is no per-volume replica placement to manage and no re-sync step before migration can proceed.

What is the difference between RWX block and RWX file storage for OpenShift Virtualization?

RWX file storage (NFS or CephFS) serves multiple nodes through filesystem semantics: a POSIX filesystem manages concurrent access, file locking, and consistency between nodes. RWX block storage exposes a raw block device simultaneously to multiple nodes, without filesystem-level coordination. For VM live migration, the guest OS runs its own filesystem (ext4, XFS, NTFS) on top of the block device. Two nodes cannot simultaneously mount that guest filesystem, but they can both hold a block device connection during the migration window provided only one issues writes at a time. OpenShift Virtualization’s migration controller enforces that single-writer protocol, making RWX block safe for VM disk access in ways RWX file is not suited for raw-block workloads.

How does simplyblock handle storage policy management for OpenShift Virtualization workloads?

Simplyblock uses StorageClass parameters to define per-volume QoS, replication factor, and placement constraints. Platform teams create separate StorageClasses for production VM disks (high IOPS limits, three-way replication), development VMs (lower limits, two-way replication), and tenant cluster workloads (tuned for the tenant’s application profile). This provides a Kubernetes-native equivalent to vSAN’s Storage Policy-Based Management without requiring a proprietary storage controller. Replication and QoS settings apply per-volume, not cluster-wide, so a latency-sensitive database VM and a bulk-storage archive VM can run on the same storage pool with independent performance guarantees.