Rob Pankow 
“The GPU hosts migrate like everything else, they just need a destination with the same cards in it.”
That sentence is half right, and the half that is wrong is the half that decides how the destination platform has to be built.
A virtual machine with a physical GPU assigned to it is not an ordinary VM carrying an extra device. The assignment is the thing that changes its availability properties, and it changes them in a direction that no amount of planning on the storage side reverses. This post answers that one question end to end, including the part of the answer that is not in our favour.
What the device assignment actually does to the VM
There are two ways a VM gets a GPU, and the difference between them is the whole story.
Full passthrough hands the PCI function to the guest. On a VMware® estate this is DirectPath I/O, or Dynamic DirectPath I/O where the host profile assigns a device from a pool rather than a named card. The guest loads the vendor driver and talks to the hardware. The hypervisor sets the mapping up and then steps out of the data path entirely, which is exactly why the performance is good and exactly why the VM stops being portable. Live migration requires the hypervisor to serialise the complete state of the machine and reproduce it somewhere else. For a passthrough device, a meaningful part of that state sits inside hardware the hypervisor has no way to read.
Mediated devices, the mechanism behind NVIDIA vGPU, keep the vendor driver on the host and present each guest a virtual slice of the physical card. Because the host driver still knows what the device is doing, there is something to capture, and that is the reason vGPU VMs could vMotion on vSphere at all. It was never a generic capability. It depended on matching driver versions, matching profiles and compatible hardware on both sides, and it was the vendor implementing a migration path, not the platform providing one.
So the estate arrives at the exit with a split it may not have written down. The full-passthrough VMs were already pinned and already handled by shutdown windows. The vGPU VMs had an escape hatch.
On KubeVirt and Red Hat® OpenShift® Virtualization, both kinds of device reach the guest through the same route: the node advertises them, the cluster permits them by vendor and product ID, and the VM requests one. Both kinds produce the same result. A VM holding a host device is reported as not live-migratable, and a migration request against it is declined rather than attempted. The escape hatch closes.
What follows is not one broken feature but a short cascade, and it is worth naming the pieces because teams usually discover them one at a time:
- Rebalancing stops applying to those VMs. Whatever the destination platform does to even out load across nodes, it cannot move this workload, so the GPU pool is scheduled once, at placement time.
- Host maintenance becomes a planned outage per VM. Draining a node is how every other workload gets out of the way during patching. A pinned VM has to be shut down, and someone has to own the window.
- Host failure becomes a restart, not a slide. The VM comes back on another node with a free, compatible device. It does not resume, it boots.
None of this is new to anyone who ran full passthrough on vSphere. What is new is that it now applies to the vGPU VMs too, and those are usually the larger and better-behaved half of the estate.
The honest answer, including the part that does not favour us
Here is the sentence worth putting in the migration document before anyone evaluates a storage layer: no storage product restores live migration for a device-pinned VM. It is a compute-side constraint created by where the device state lives. A vendor who implies otherwise is selling you something that does not exist, and a shortlist built around that promise will be rebuilt later under worse conditions.
Table 1: What the common claims about GPU VMs after the exit actually resolve to.
| Claim | What is true | What it depends on |
|---|---|---|
| ”vGPU migrates, so our GPU estate is portable” | It migrated on vSphere because the vendor implemented a path there. On KubeVirt and OpenShift Virtualization a VM with a host device is reported as non-migratable | Whether the destination platform implements that path at all, not whether the GPU vendor supports the concept |
| ”Full passthrough is simpler, we will standardise on it” | Correct, and most estates do. It also removes every remaining movement option, including suspend and resume | Whether any VM in that pool will ever need its host patched without a scheduled outage |
| ”The storage layer can give live migration back” | No. The pin is created by device state the hypervisor cannot capture, and storage sits nowhere near it | Nothing. This one does not depend on anything |
| ”Then the restart time is just what it is” | This is the claim that is actually wrong. Boot time is fixed, but the wait before boot is not | How quickly the VM’s volumes are usable on the node it restarts on |
The last row is where the decision moves. Once you accept that every move is a stop and a start, the only remaining variable is the length of the start, and that is a storage question rather than a GPU question.
It is worth being precise about the size of the prize. This does not make a pinned VM highly available in the sense a live-migratable VM is. It changes a recovery measured in the time it takes to make a dataset present on a new node into a recovery measured in seconds plus boot. On a node holding several GPU VMs with substantial working sets, that difference is the difference between a maintenance window the platform team schedules at night and one they schedule with the business.
Running GPU workloads on the destination platform and sizing the node pool now? Talk to us before the GPU hosts are specified, because the storage decision changes how many of them you need and how long each one takes to patch. Talk to a storage architect
What has to carry the availability instead
Three things decide how a GPU node pool behaves, and all three are storage decisions made before the hardware is ordered.
Where the data is when the VM restarts. If a VM’s volumes live on the local disks of the node that just failed, the restart somewhere else is gated by rebuilding or re-streaming that data onto the new node. On a GPU node pool this is the worst possible version of that problem, because the datasets are large and the nodes are the most expensive in the estate. When the volume lives in a shared pool and reaches the VM over the network, the restart is a detach and an attach: the new node opens the same volume and the guest boots. This is the same mechanism that makes live migration work for ordinary VMs on OpenShift Virtualization, applied to the one case where the VM itself cannot move. Simplyblock’s NVMe/TCP storage is built around that reattach path rather than around per-node replica placement.
What the GPU nodes are spending their capacity on. A storage design that places replicas on the nodes that run the workloads will place replicas on the GPU hosts, which means paying GPU-node prices for storage capacity and protection overhead. Keeping the storage pool’s growth independent of the compute tier’s growth is the point of separating them, and on a GPU pool the arithmetic is unusually stark. Based on publicly available information as of September 2026, simplyblock’s erasure coding lands near 1.25:1 raw to effective at FTT1 against roughly 3:1 for three-way replication in practice. These are our own comparison figures rather than independently verified benchmarks, so treat them as a starting point for a test on your own hardware. The direction is what matters here: every unit of protection overhead you avoid on a GPU host is the unit you did not buy at the highest price per slot in the estate.
What the storage stack costs the node in CPU. A GPU host’s cores are there to feed the accelerators. A storage data path that consumes host CPU to serve its own I/O is competing with the thing the node was bought for, and that competition shows up as lower accelerator utilisation rather than as a storage metric anyone is watching. Keeping the data path off the node, over the fabric, is what protects that budget. The same argument applies to the sustained write traffic these workloads generate, which is worked through in more detail for training checkpoints and model registries and for model weight loading on inference nodes.
The practical shape of this is a requirement list rather than a product preference. Ask a candidate storage layer how long a volume takes to become usable on a different node after the original one is gone, with the dataset sizes you actually run. Ask where the replicas or parity live and whether you can keep them off the GPU hosts. Ask how much host CPU the data path consumes under sustained write load. The answers to those three questions decide what a GPU node failure costs you, and they are answerable before anything is bought. The device pin is not negotiable, so the recovery path is the thing to engineer.
Questions and Answers
Can a VM with GPU passthrough live migrate on KubeVirt or OpenShift Virtualization?
No. A VM holding a host device, whether a full passthrough function or a mediated vGPU device, is reported as not live-migratable and migration requests against it are declined. Simplyblock’s recommendation is to stop treating this as a gap to close and design the restart path instead: put the VM’s volumes in a shared pool reachable from every GPU host, so recovery on a surviving node is an attach and a boot rather than a data movement. That is the part of the outcome you control, and on large datasets it is most of the elapsed time.
We adopted NVIDIA vGPU on vSphere specifically so those VMs could vMotion. Does that carry over?
Not as a platform capability. vGPU migration on vSphere was the GPU vendor implementing a migration path for its own mediated devices in that specific hypervisor, with matching drivers, profiles and hardware required on both hosts. The destination platforms treat any host device as a pin. Plan for the vGPU VMs to behave like the full-passthrough ones, and rebuild the maintenance procedure accordingly rather than assuming the old one transfers.
How do we patch hypervisor hosts if the GPU VMs cannot be drained?
With scheduled shutdown windows per node, which is the honest answer and the one worth socialising early. The variable you can improve is how long each window has to be. If the VMs restart on another GPU host with their volumes already available there, the window is a shutdown, a boot and a verification. If the data has to be rebuilt onto the target node first, the window is whatever that takes, and it scales with dataset size rather than with VM count. Sizing the GPU pool with at least one node of spare device capacity is the other half of this, and it is a decision taken at purchase time.
Should storage replicas live on the GPU nodes themselves?
Generally no, and the reason is commercial rather than technical. Protection overhead placed on GPU hosts is capacity bought at the highest price per slot in the estate, and it grows with the dataset rather than with the accelerator count, so the two scaling curves fight each other. Separating the storage pool from the GPU compute tier lets each one be sized for what it is for. This is also what keeps a GPU refresh from turning into a storage migration, which matters more than it sounds on hardware that is replaced on a faster cycle than the rest of the estate.
Is Ceph a reasonable choice behind a GPU node pool?
Ceph is a good option when performance is not the main goal, and it is a genuinely capable, well-understood platform. The two things to measure for this specific use are the raw-to-effective ratio at your chosen protection level, because that ratio is applied to datasets that are large by definition, and the host CPU its data path consumes on nodes whose cores are meant to be feeding accelerators. Both push in the same direction on a GPU pool. If the nodes are expensive and the write load is sustained, those two numbers are worth measuring on your own hardware before the design is fixed.
Does running a storage layer across GPU worker nodes change what we pay for it?
Simplyblock is priced like the platform it runs on: by the CPU capacity of your worker nodes, counted the same way your OpenShift or Kubernetes subscription counts them, and data volume does not change the price. Everything is included, so disaster recovery, replication, snapshots, encryption and QoS are not separate line items to add once the GPU estate is in scope. The design consequence worth noting is the one above: because capacity and compute scale independently, growing the dataset behind a GPU pool is not the same decision as growing the pool.
