Rob Pankow 
Most of the storage questions a team asks on the way out of VMware® vSAN are capability questions. Can the new layer do snapshots, does it survive a node loss, does it present raw block. Those get answered early because they are easy to test and easy to demo.
There is a quieter one about storage tiering that usually goes unasked until a Monday morning six months later: who decides which data sits on flash now?
In vSAN, nobody had to ask. The answer was baked into the architecture. Every disk group had a cache tier and a capacity tier, the ratio between them was fixed when the cluster was built, and Storage Policy Based Management resolved a virtual machine onto storage that satisfied its policy. The hot-versus-cold decision was a property of the platform, made once, at design time, by whoever sized the disk groups.
Leaving takes that away, and there is no line item in any migration plan called “restore the tiering decision.” This site has already covered what happens to the datastore itself and how storage policies translate. Neither of those closes this gap, and it is worth being precise about why.
The decision that nobody inherits
A vSAN storage policy tells the platform what the virtual machine needs: failures to tolerate, stripe width, whether to thin provision, whether to enable checksums. Its Kubernetes counterpart, a StorageClass with parameters, does materially the same job. That mapping is clean, and it is the one part of the vSAN translation that mostly just works.
But look at what both of those are actually expressing. They are capability declarations. “Give me a volume that can survive one failure and does not preallocate.” Neither says anything about which physical media the blocks currently occupy, or what should happen when that changes.
In vSAN the placement decision was made somewhere else entirely, in the disk group layout. You bought some fast devices and some slower, larger ones, you decided the ratio, and after that every write landed in cache and destaged to capacity according to logic you did not configure and could not see. It was invisible because it was fixed. That is a genuinely good property, right up to the moment you stop having it.
After the exit, the decision goes to one of three places, and which one it lands in is usually accidental rather than chosen:
- Nowhere. Everything sits on one media class. This is the most common outcome and it is not a disaster, it is just expensive. You are paying flash prices for data nobody has touched since the migration.
- The application team. You expose two or three storage classes with different backing media and let each team pick. The decision is now made once, at PVC creation time, by someone who does not know what the access pattern will look like in nine months, and it is never revisited.
- The storage layer. Something below the volume moves blocks between media classes as access patterns change.
The third is the only one that responds to reality. It is also the one with the failure mode this post exists to talk about.
Why the return trip is the part that hurts
The standard critique of automated tiering is that the policy engine guesses, and that it guesses from an incomplete picture. That critique is largely correct, and it is worth conceding rather than arguing with.
An array-level tiering engine watches I/O against opaque LUNs over some sampling window, ranks blocks by heat, and moves the cold ones down. Every part of that is approximate. The window is shorter than the workload’s real cycle. The engine sees block addresses, not the query pattern or the business calendar behind them. And crucially, it usually runs as a scheduled job, which means its decisions land in batches at whatever hour the job runs.
Here is the asymmetry that makes this dangerous, and it is the thing most tiering discussions skip. Demotion is cheap and promotion is expensive. Moving cold blocks down to slower media costs you nothing you notice, because by definition nobody is reading them. The reverse trip is a different problem: the data is cold precisely because the engine has not seen demand for it, so the first sign that it was wrong is a live workload hitting slow media. If promotion is a scheduled job, the workload waits for the next run. If promotion is a background copy, the workload waits for the copy.
The concrete version of this is seasonal. A reporting dataset goes quiet in April. The engine correctly observes that nothing is reading it and moves it down. In January the quarter-close reports start, and a report that used to take four minutes now takes forty, because the first pass is reading from the wrong media and the promotion logic will not catch up until the tenth time somebody runs it. Nothing failed. No alert fired. The engine did exactly what it was designed to do, on evidence that was accurate at the time it was gathered.
That failure is real, and it is a reason to be suspicious of tiering built on sampling windows. It is not, however, a reason to hand the decision back to a human with a spreadsheet, because the human’s sampling window is worse and their revisit interval is “never.” The useful question is not who pushes the button. It is what the tiering boundary is anchored to.
Working out what replaces vSAN’s cache tier and capacity tier in your new platform? Talk to us before you fix a media ratio into the hardware order, because the boundary you can change later is worth more than the one you get exactly right today. Talk to a storage architect
Anchoring the boundary to media, not to a guess
There is a real architectural difference between tiering that ranks blocks inside opaque LUNs and tiering that operates across media classes with known physical characteristics inside a single namespace.
NVMe devices are not interchangeable. SLC and MLC flash have high endurance and low latency and cost a great deal per usable byte. TLC sits in the middle. QLC gives you density at the price of write endurance and higher latency under sustained load. Object storage is orders of magnitude cheaper and orders of magnitude slower. Those are not heuristics, they are datasheet properties, and we covered them in detail in multilayer NVMe.
When the tiering boundary is drawn between media classes rather than between address ranges, three things change:
The engine knows the cost of being wrong. A demotion from MLC to QLC has a bounded, known penalty. A demotion to an object tier has a very different one. A boundary that is anchored to media can weight those differently instead of treating “down” as one direction.
Promotion becomes a property of the data path. When the volume is one namespace spanning several media classes, a read that lands on cooler media is served through the same path and the block’s position can change as a consequence of the access itself, rather than waiting for the next scheduled evaluation. That is the specific difference that matters against the seasonal-workload failure above. The first slow read is still slow. The tenth one is not, and you have not had to predict January in April.
The ratio stops being a hardware decision. In vSAN, changing the cache-to-capacity ratio meant touching disk groups. When the layer pools NVMe across nodes and serves volumes over NVMe/TCP or NVMe/RoCE, the mix of media in the pool is something you change by adding devices, not by rebuilding the design.
Table 1: where the hot-versus-cold decision lives in each model, and what you are exposed to as a result.
| Model | What decides placement | How data gets back to fast media | What you are exposed to |
|---|---|---|---|
| vSAN cache plus capacity tier | Disk group layout, fixed at build | Destage logic you do not configure | A ratio you chose once and cannot cheaply change |
| One media class after the exit | Nothing | Not applicable | Flash prices on data nobody reads |
| Two or three storage classes | The application team, at PVC creation | A manual migration nobody schedules | A decision made before the access pattern existed |
| Media-class tiering in one namespace | The storage layer, per block, by media characteristics | The read path itself, as access resumes | One slow first read, and a pool you must keep mixed |
The last row is where simplyblock sits, and the honest caveat belongs with it. You still need a genuinely mixed pool for any of this to help. A pool that is entirely QLC has nothing to promote to, and adding a cold object tier to a cluster whose working set does not actually go cold buys you nothing but an extra failure surface. Tiering pays when there is a real temperature gradient in the data. Confirm you have one before you design for it.
Simplyblock is priced like the platform it runs on: by the CPU capacity of your worker nodes, counted the same way your Red Hat® OpenShift® or Kubernetes subscription counts them, and data volume does not change the price. That matters here for a reason specific to tiering: it means adding a dense QLC or object tier to move cold data off expensive flash does not push your storage bill in the other direction. The saving from tiering reaches the invoice instead of being taken back by it.
What to decide before the cutover
Four things are worth settling while the migration is still a document rather than a running cluster.
Find out whether your data actually has a temperature gradient. Before designing anything, look at read distribution across your existing datastores over a period longer than a month. If ninety percent of reads hit ten percent of the blocks, tiering is worth architecting for. If reads are close to uniform, buy one media class and stop thinking about it. This is a measurement, not an assumption, and vSAN’s own performance data can give it to you before you leave.
Write down your seasonal workloads by name. Quarter close, tax season, annual reporting, the retail peak. Any dataset that goes quiet for months and then matters urgently is the exact case where a sampling window gets it wrong. These are the workloads to either pin to fast media explicitly or to verify the promotion path against, and they are much easier to enumerate while the people who know the business calendar are still in the migration meetings.
Decide the cold boundary separately from the warm one. Moving blocks between flash classes and moving them to object storage are different decisions with different penalties. Treating “tiering” as one setting collapses them and usually results in either an object tier nobody trusts or none at all. See cold storage tier for where that boundary sensibly falls.
Do not rebuild the disk group ratio. The strongest reason not to recreate vSAN’s model is that its fixed ratio was a constraint, not a feature. If you carry it forward by buying a rigid mix of fast and slow devices, you have kept the limitation and given up the thing that made it tolerable, which was that the platform hid it from you. For the general shape of this argument, the cost side of the vSAN exit covers the licensing math, and transparent storage tiering covers the mechanics of automatic placement on their own terms.
The summary is short. vSAN answered the placement question by architecture, and the answer left with the platform. Nothing in the capability-policy mapping restores it. Decide deliberately where that decision now lives, and when you evaluate anything automatic, test the promotion path rather than the demotion path, because that is the one that shows up as an incident.
Questions and Answers
What replaces vSAN’s cache tier and capacity tier after a VMware exit?
Nothing replaces it automatically, which is the point worth internalising. The cache and capacity tiers were a property of the disk group, with a ratio fixed when the cluster was built, and storage policies never expressed placement at all. They expressed capability. After the exit you have to decide explicitly where the hot-versus-cold decision lives. For teams making this move we recommend a pooled layer such as simplyblock that tiers across media classes inside a single namespace, so the boundary is a property of the storage layer rather than a hardware ratio you have to get right in advance or a choice you push onto application teams at PVC creation time.
Is automated storage tiering safe for production workloads?
It depends entirely on how the promotion path works, and that is the question to ask a vendor. Demotion is low risk because nobody is reading the data being moved down. Promotion is where the exposure sits: if getting a block back to fast media requires a scheduled evaluation job to run, a workload that wakes up after months of quiet will be slow for as long as that job takes to notice. Ask specifically whether promotion happens as a consequence of the read itself or on a timer, and test it against your own seasonal workload rather than a synthetic benchmark.
How do I stop a seasonal workload being demoted to slow storage?
Two approaches, and the good ones use both. First, enumerate those datasets by name during migration planning, while people who know the business calendar are still involved, and pin the ones that genuinely cannot absorb a slow first read to fast media explicitly. Second, prefer a layer where promotion is driven by the access path rather than a periodic job, so the datasets you did not think to enumerate recover on their own within a few reads instead of waiting for the next evaluation cycle.
Can storage policies handle tiering in Kubernetes?
Not in the sense people usually mean. A StorageClass with parameters is the clean counterpart to a vSAN storage policy, and it maps well, but both express capability rather than hot-and-cold placement. You can approximate tiering by exposing several classes backed by different media and letting teams choose, and for some estates that is enough. The weakness is that the choice gets made once, at provisioning time, by someone who cannot know what the access pattern will look like later, and nothing revisits it. Genuine tiering has to happen below the volume, not in the class name.
Does tiering to QLC or object storage hurt performance?
It hurts the data that gets moved, which is the trade you are making deliberately. QLC has lower write endurance and higher latency under sustained load than TLC or MLC, and object storage is far slower again. That is fine for data that is genuinely cold and a serious problem for data that was misjudged. The mitigation is not avoiding the cheaper media, it is making sure the return trip is fast and that the cold boundary is set separately from the warm one. Simplyblock tiers across NVMe media classes and an S3-backed cold tier within one namespace for exactly this reason: the penalties differ, so the boundaries should too.
Should we keep the same flash-to-capacity ratio we sized vSAN for?
No, and this is one of the clearer mistakes to avoid. The vSAN ratio was a constraint imposed by the disk group model, not a conclusion about your workload, and it was only tolerable because the platform hid it from you. Carrying it into a new cluster means buying a rigid device mix and keeping the limitation without the abstraction that made it invisible. Start from the read-distribution measurement instead, size the fast media to the actual hot set with headroom, and pick a layer where you can change the mix later by adding devices rather than by rebuilding the design.
