Space Reclamation After the vSphere Exit: Who Gives Deleted Blocks Back

Three months after the last virtual machine leaves vSphere, someone opens the storage dashboard and finds a number that does not add up. The pool reports 41 TB consumed. Add up every volume the platform admits to owning and you get 12. Nothing has failed, no alert has fired, and the gap is not a dashboard rounding error. It is deleted data that never came back.

This is the quietest failure mode in a VMware® exit, and it is quiet for a structural reason: nothing breaks. Workloads keep running, writes keep landing, and the only symptom is that consumed capacity moves in one direction. It usually gets discovered by a purchase order rather than by monitoring.

It happens because vSphere had an apparatus for exactly this problem and the apparatus was close to invisible. It was a column, an alarm, and a per-datastore setting. No migration runbook lists them, because nobody configured them during a migration. They were already there, doing their job, before anyone on the current team arrived.

One boundary first, because this topic has a sibling that gets merged into it. This article is about the general estate: the chain that returns freed blocks to the pool, and who owns each link once the hypervisor that used to own all of them is gone. It is not about desktops. Non-persistent VDI is the one workload where this problem is acute rather than slow, because every single logoff frees a session’s worth of writes and every login allocates more, and that case has its own treatment in replacing VMware VDI storage on Kubernetes. Read that one if you are migrating a desktop pool. Read this one if you are migrating an estate.

Three things vSphere did about deleted space that nobody wrote down

Start with what you are actually losing, because the loss is not a feature. It is an operational habit that the platform made possible.

The provisioned-versus-used column. The datastore summary in the vSphere Client put Provisioned Space and Used Space next to each other. Two numbers, one screen, no query to write. The ratio between them was the over-provisioning position of that datastore, readable at a glance by anyone who happened to be looking, including people who were not thinking about capacity at all. That accidental visibility is what caught drift early.

The alarm. vCenter shipped a datastore usage alarm out of the box, configured to fire well before a datastore actually filled. Nobody set it up. It was simply present on every datastore in the estate, which means every team that ever ran vSphere has been protected by a capacity alarm they never wrote and may not know exists. After the exit it does not exist, and its absence is silent by definition, because the thing an alarm does when it is missing is nothing.

The UNMAP setting. This is the mechanism itself. VMFS6 reclaims freed blocks on its own, asynchronously, at a reclamation priority set per datastore. On VMFS5 the same work was a manual esxcli storage vmfs unmap run that somebody scheduled, which is why administrators who lived through that era remember reclamation as a chore and administrators who started on VMFS6 have often never thought about it. Either way, the hypervisor took responsibility for telling the array that blocks were free.

There was a second half to it, one level up. For a guest’s own deletions to reach the array, the virtual disk had to be thin and the datastore had to be VMFS6, and then a TRIM-aware guest filesystem could pass its discards down through the stack. When those conditions were met the whole chain worked end to end with no per-workload configuration, and the reason this matters now is not nostalgia. It is that the chain still has to exist afterwards, and it no longer has one owner.

The reclaim chain, link by link

After the exit the same job is done by four independent things that ship separately, are configured separately, and fail separately. A break in any one of them looks identical from the dashboard: consumed capacity that only goes up.

The guest filesystem. This is where deletion actually happens and where the information originates. A filesystem that deletes a file marks blocks free in its own metadata and, unless told otherwise, tells nothing underneath it. There are two ways to make it talk. Continuous discard, via the discard mount option, issues the request inline as files are deleted, which is simple but puts the work on the deletion path. Periodic trim runs fstrim on a schedule and batches the whole thing. Simplyblock’s documentation recommends the periodic approach for regular maintenance, with fstrim.timer named as the mechanism where a systemd timer is operationally appropriate, and the commands differ by filesystem: fstrim -v /mount/point on ext4, xfs_fsr -v /mount/point on XFS.

The virtual disk path. For a virtual machine on KubeVirt or Red Hat® OpenShift® Virtualization, the guest’s discard has to survive the trip through the virtual disk controller before it reaches anything real. This is the link that has no vSphere equivalent worth comparing, because in vSphere it was one setting on one virtual disk, and it is the link most likely to be skipped in planning because it sits between two teams. Whichever controller a VM is defined with, the question to ask of the VM definition is whether discard is passed through rather than silently dropped, and the answer belongs in the template every VM is created from, not in a per-VM fix applied later.

The CSI layer. The Container Storage Interface driver is the boundary between the platform and the storage system, and it either forwards discard to the backend or it does not. The API surface tells you nothing about this, which is a recurring theme in CSI and the reason CSI alone is not enough for stateful Kubernetes: identical calls, very different implementations underneath. Ask the question of the driver directly rather than inferring it from the fact that volumes provision and mount correctly. They will do that whether or not discard passes.

The pool. Finally the storage system has to act on what it receives. Simplyblock’s own documentation is explicit that this is not automatic from the filesystem’s point of view: in a thin-provisioned setup, deleting files in the filesystem does not by itself reclaim capacity on the storage backend, and trimming is what notifies the storage stack that blocks can be released. It is equally explicit that the outcome is conditional, stating that the effectiveness of reclaim depends on the filesystem, kernel support, and mount configuration. That is the honest version, and it is worth more than a claim of automatic reclamation, because it tells you which variables to check when a trim run frees less than you expected.

Table 1: The reclaim chain after a vSphere exit. Four links that used to have one owner, plus the alarm that used to be free.

The practical consequence of a four-link chain is that verification cannot be theoretical. Delete a large file inside a guest, run the trim, and read the consumed figure on the volume before and after. Simplyblock’s documentation asks for the same discipline on the verification side and names the pitfalls plainly: trimming the wrong mountpoint or an unmounted path, expecting reclaimed capacity to appear immediately in every dashboard and counter, and assuming all filesystems and environments behave identically for discard and reclaim. Those three cover most of the false conclusions a first test produces. Do the test on day one of the migration rather than in month three, because it costs an afternoon now and a hardware order later.

Not sure whether discard survives the trip from your guests to your pool? If you are planning a vSphere exit and want the reclaim chain verified against your actual guest images, VM templates, and CSI path before cutover rather than after, that is a conversation worth having early. Talk to a storage architect

Two different things are called reclaim, and only one of them frees blocks

There is a naming collision here that sends people looking in the wrong place, and it is worth spending a section on because the wrong place looks so convincingly like the right one.

Kubernetes has a reclaim policy. It is a field on a persistent volume, set through the StorageClass, and it takes Delete or Retain. It governs the lifecycle of the PV object: whether the volume survives the deletion of the claim that bound it. With Retain, deleting a PVC leaves the PV in a Released state with the storage intact and waiting for a deliberate cleanup, which is the correct setting when data needs review before it disappears.

Search for reclamation in a Kubernetes context and this is what you will find, and it has nothing to do with the problem in this article. The reclaim policy decides whether a volume continues to exist. Space reclamation decides whether the blocks inside a volume that still exists come back to the pool when the filesystem stops using them. A cluster can be perfectly configured on the first and completely broken on the second, and the first is the one that is documented, discussed, and set deliberately.

There is a similar trap in the word unmap. In CSI, map and unmap are stages of the attach and detach cycle by which a volume is presented to a node and taken away again, and that is the sense in which the term appears in most Kubernetes storage material, including our own writing on CSI provisioning latency and day-two operations. It is not SCSI UNMAP telling a backend that blocks are free. Same word, unrelated mechanism, and the ambiguity is one reason this whole area reads as covered when it is not.

Two consequences follow, and the second one is the expensive one.

The first is that Retain and space reclamation interact. A retained volume that nobody cleans up is capacity held by an object with no claim on it, which is a different leak from the block-level one but arrives at the same dashboard. Both need an owner.

The second is snapshots, and this is documented behaviour rather than a caveat we are inferring. Simplyblock’s documentation carries an explicit warning on volume removal: if snapshots or snapshot chains of the logical volume exist, the internal storage is not reclaimed until all of the snapshots are deleted as well. That is the correct and expected behaviour of any copy-on-write snapshot implementation, since a snapshot’s entire purpose is to keep the old blocks readable, but it has a direct operational consequence after a vSphere exit. If your backup product, your migration tooling, or your own automation left a snapshot chain behind, deleting the volume does not give the space back, and neither does trimming inside a guest that no longer exists. Snapshot hygiene is part of capacity hygiene, and it is usually the first thing to check when a pool refuses to shrink after a cleanup that should have worked.

Over-commit discipline when there is no datastore to alarm on

The last part is the one most vendor content skips, including some of our own. A thin pool is genuinely a better allocation model than carving fixed datastores, and the temptation is to state that and stop. One post on this site says that thin provisioning makes capacity planning irrelevant. That overstates it, and this article is the right place to qualify it rather than repeat it.

What thin provisioning removes is the need to size each volume correctly in advance. That is a real and substantial gain: volumes can be provisioned generously, they consume only what is written, and they can be expanded later without a migration. What it does not remove is the need to know the aggregate position, and it actually makes that position harder to see, because the reassuring per-datastore boundaries are gone. Over-commitment across a shared pool is a bet that the sum of what everyone writes stays below what you own. Thin provisioning does not change the odds of that bet. It changes who can see them, and by default the answer is nobody.

So the discipline moves rather than disappearing, and after the exit it needs building instead of inheriting:

• Alert on the aggregate, not on volumes. The failure mode is the pool, and a per-volume view will not show it. The number that matters is total pool consumption, which is exactly what the vCenter alarm was watching in its own scope.
• Track provisioned against consumed as a ratio over time. The absolute figures are less useful than the trend. A ratio that climbs steadily while workload count is flat is a reclamation problem, and it is visible in the trend months before it is visible in a threshold.
• Give trim a schedule and an owner. A periodic timer inside guest images, applied through the template rather than by hand, so that new workloads inherit it. Simplyblock’s guidance also suggests running large trim jobs during low I/O periods in production, which makes a scheduled window preferable to an ad-hoc run during business hours.
• Put snapshot expiry next to it. Retention policy is a capacity control here, not only a recovery control, for the reason in the previous section.
• Watch what the node sees, not only what the pool reports. Space that never returns eventually shows up as pressure on the node, which is a different and louder failure than a dashboard number, and one we have written about separately in node disk pressure and pod eviction.

There is one commercial note worth making, because capacity leakage has two different costs and teams often conflate them. Phantom usage costs you raw hardware: media you bought that is holding blocks nothing references. What it should not do is inflate your storage licence on top of that. Simplyblock is priced like the platform it runs on: by the CPU capacity of your worker nodes, counted the same way your OpenShift or Kubernetes subscription counts them, and data volume does not change the price. Reclamation is therefore a hardware efficiency problem to solve rather than a billing meter to fear, which is the right incentive to have while you are still learning where your own chain leaks.

The larger point is the one that generalises past this mechanism. A vSphere estate accumulated a lot of quiet competence over fifteen years, and very little of it is written down, because it was never a decision anyone made. It was a default. The provisioned-versus-used column and the datastore alarm are two of those defaults, they were doing real work, and the migration plan that lists operators and drivers and network policies will not mention either one. Reclamation is worth working through carefully partly for its own sake and partly as a reminder to go looking for the others.

Questions and Answers

Why does consumed capacity keep growing after a VMware exit even though workloads are stable?

Because deleting files inside a guest does not, by itself, return blocks to a thin-provisioned pool. Something has to tell the storage system those blocks are free, and after a vSphere exit that job is split across the guest filesystem, the virtual disk path, the CSI driver, and the storage system, instead of being handled by the hypervisor. If any link drops the discard, consumed capacity only ever rises. Simplyblock supports the full chain and documents the trim workflow that drives it, including the periodic fstrim schedule, the per-filesystem commands, and how to verify reclaimed capacity afterwards. Verify it on day one of the migration with a delete, a trim, and a before-and-after capacity reading, because it takes an afternoon to test and a hardware order to discover late.

Is the Kubernetes reclaim policy the same thing as space reclamation?

No, and confusing the two is the single most common wrong turn in this area. The reclaim policy is a StorageClass field taking Delete or Retain, and it decides whether a persistent volume object survives the deletion of its claim. Space reclamation decides whether the blocks inside a volume that still exists come back to the pool when the filesystem stops using them. They are independent. A cluster can have a correctly considered Retain policy and still leak capacity on every deletion.

Does trimming the filesystem always free capacity immediately?

Not always, and expecting it to is a documented pitfall. Simplyblock’s documentation notes that the effectiveness of reclaim depends on the filesystem, kernel support, and mount configuration, and warns specifically against expecting immediate visible reclaimed capacity in all dashboards and counters. The other frequent cause of a disappointing result is a snapshot: if snapshots or snapshot chains of a logical volume exist, the internal storage is not reclaimed until those snapshots are deleted too, which is inherent to how copy-on-write snapshots work rather than a limitation to work around. Check the mountpoint, the filesystem type, the timing of backend metrics, and the snapshot chain before concluding that discard is not passing.

Which filesystem trim approach should we standardise on, continuous discard or a periodic timer?

For most estates, a periodic timer. Continuous discard via the discard mount option is simpler to reason about but moves the work onto the deletion path, where it competes with production I/O. A scheduled fstrim batches the work and can be placed in a window, which matches the guidance to run large trim jobs during low I/O periods. The important part is less which one you pick than that it lives in the guest image or VM template every workload is created from, so new workloads inherit it instead of needing a manual fix.

Does this apply to containers as well as virtual machines, or only to migrated VMs?

Both, and the chain is shorter for containers because there is no virtual disk path in the middle. A container writing to a mounted volume frees blocks in exactly the same way and needs the same discard or trim behaviour to return them. Virtual machines on KubeVirt or OpenShift Virtualization add one extra link, which makes them the more common place for the chain to break after a VMware exit, but a container platform with no trim discipline leaks capacity just as steadily.