vSAN Stretched Cluster Replacement: Two-Site Synchronous Storage After the VMware Exit

There is a particular kind of team that reads VMware® exit content and finds none of it addresses their actual problem. They run two datacentres, 30 or 40 kilometres apart, with a vSAN stretched cluster across them and a witness appliance in a third location. Their recovery point objective is zero, not by aspiration but by design, and it has been zero for years.

For that team, most migration guidance is about single-site availability. Node fails, volume re-attaches, workload restarts. That is a real problem and this site has covered it, in the post on replacing HA, DRS and Fault Tolerance. It is also not what a stretched cluster is for. A stretched cluster exists so that losing an entire building is survivable without losing a write.

The thing worth being precise about is that a stretched cluster is not one feature you swap out. It is four mechanisms that vSAN happened to deliver in a single bundle, and leaving unbundles them all at once.

The four things a stretched cluster was doing for you

Write them down before choosing a replacement, because a migration plan that only accounts for the first one will pass a demo and fail a site test.

One, synchronous mirroring across two fault domains. Every write is acknowledged only once it is durable at both sites. That is where the zero RPO comes from, and it is the piece everyone remembers.

Two, a witness that arbitrates. A third location holds witness components: metadata only, no virtual machine data. Its only job is to be a tiebreaker when the two data sites cannot see each other.

Three, site-local read affinity. Reads are served from the copy in the same site as the running virtual machine, so a read does not pay the inter-site round trip. Without this, half your reads cross a wide-area link and your latency profile quietly doubles.

Four, a per-object policy that separates the two failure scopes. Primary failures to tolerate, PFTT, governs protection across sites. Secondary failures to tolerate, SFTT, governs protection within a site. One policy object, two independent decisions, applied per object.

Those four are the specification. Everything that follows is about who takes over each line.

The witness is not a copy of your data, it is a vote

The most common misreading in a stretched-cluster migration is treating the witness as a third replica. It is not. A vSAN witness host stores witness components, which are small metadata objects that exist to make the number of votes odd. It holds no virtual machine data, which is why it can sit on a thin link with high latency tolerance while the two data sites cannot.

Once you understand it as a vote rather than a copy, the replacement becomes obvious in shape if not in product. Whatever runs your storage after the exit has to answer one question: when the inter-site link drops and both sites are alive and healthy in isolation, which side is allowed to keep accepting writes?

There are only a few honest answers, and each has a cost.

A third-site quorum member is the direct translation. It participates in the consensus that decides which site holds the writable copy, and like the vSAN witness it needs to be reachable rather than fast. This is the closest structural match to what you had, and it means the third site does not go away just because vCenter did.

A cloud-hosted arbitrator is the same idea with the third site rented instead of owned. It usually works, and the failure mode to test is the correlated one: if your inter-site link and your internet egress share a path or a provider, the arbitrator disappears exactly when you need it.

Manual intervention is the answer nobody chooses on purpose and many estates end up with anyway. It is defensible for a small number of workloads with a documented runbook and a person on call. It is not a stretched cluster, and calling it one is how a 90-second failover becomes a 40-minute conference call.

The trap to avoid is a two-site configuration with no third vote at all, protected only by a rule that says one site is always the winner. That does survive a link failure. It does not survive the loss of the designated winner, which is the event you built two sites for.

Running two datacentres with a zero RPO and no obvious replacement for the witness? Talk to us before the cutover design is fixed, while the third site and the read path are still decisions rather than constraints. Talk to a storage architect

Site affinity without a policy object

Read affinity is the piece that most often gets discovered late, because nothing breaks when it is missing. The cluster comes up, the mirror is healthy, the application works. It is just slower than it was, on every read, forever, and the cause is invisible unless somebody thinks to ask which site served the block.

In vSAN this was a policy setting on the object, resolved by the platform against the host the virtual machine was running on. There is no equivalent single knob after the exit. What you get instead is a set of mechanisms that have to line up:

• The storage layer has to know its own topology. Which nodes are in site A, which are in site B. If the layer treats all nodes as one flat pool, it cannot prefer a local copy because it does not know what local means.
• The scheduler has to agree with that topology. In Kubernetes this is zone and region labels on nodes plus topology-aware volume binding, so a pod and the preferred copy of its volume do not land in different buildings. Getting the storage side right and leaving the scheduler unaware produces a workload that runs in site B reading from site A.
• Something has to re-evaluate affinity after a move. A virtual machine or pod that migrates across sites has changed which copy is local to it. Whether the read path follows is a property of the storage layer, and it is worth testing explicitly rather than assuming.

This is also where the honest limits sit. Site affinity for reads is achievable and worth insisting on. Site affinity for writes is not a thing: a synchronous mirror means every write goes to both sites by definition, so the write path always pays the inter-site cost. Anyone who tells you otherwise is describing asynchronous replication with extra steps.

The latency budget you inherit when the write path spans two sites

vSAN published a supported limit of 5 milliseconds round trip between data sites for a stretched cluster, and up to 200 milliseconds to the witness. Those numbers were validated against a specific implementation, so do not carry them over as though they are physics. What does carry over is the shape of the constraint.

Light in fibre covers roughly 200 kilometres per millisecond, so a 40 kilometre site separation costs about 0.4 milliseconds of round trip in the glass alone. Switching, optical transport and any encryption or inspection on the path add more, and the total is what every synchronous write pays before it can be acknowledged.

That is a small number in absolute terms and a large one relative to the media. An NVMe device answers a write in tens of microseconds. Put an inter-site round trip in front of it and the fabric, not the flash, sets your write latency. Two consequences follow, and both belong in the design rather than in a post-incident review:

• Measure the actual round trip on the actual path, under load, before committing. Not the vendor’s distance rating, not the link’s specification. The number you care about is what a small synchronous write sees at the busiest hour, including anything the security team has inserted into the path.
• Decide which workloads genuinely need synchronous protection. A zero RPO across sites is the right answer for a transactional database of record. It is an expensive answer for a build cache, a scratch tier or a read-heavy analytics replica. In vSAN this was a per-object policy decision, and the equivalent granularity after the exit is per volume, expressed in a StorageClass or its platform equivalent, so the two-site cost is paid only where it buys something.

A useful way to hold all of this is one row per stretched-cluster mechanism, with what has to take it over and what the exposure is if nothing does.

A site partition is not a node failure

Single-site failure handling and site partition handling are different problems, and a platform can be excellent at the first while being undefined about the second. This is the single most useful thing to test before a stretched-cluster migration is signed off.

A node failure is unambiguous. The node stops answering, the rest of the cluster agrees it is gone, the volume re-attaches somewhere else. Every storage layer worth considering handles this, and it is what most proof-of-concept exercises actually exercise.

A site partition is ambiguous by construction. Both halves are alive. Both halves can serve their local applications. Both halves believe the other side is the one that failed. The only thing preventing two divergent copies of your data is the arbitration you set up in advance, and the questions that matter are specific:

• Who keeps writing, and how fast is that decided? The answer should be a bounded number of seconds derived from a quorum decision, not a human judgement call.
• What does the losing side do? It must stop accepting writes. If it keeps serving reads from a copy that is no longer being updated, you have replaced split brain with stale reads, which is harder to detect.
• What happens when the link comes back? Resynchronisation is where the real difference between architectures shows up. A layer that tracks changed blocks resynchronises the delta. A layer that does not may rebuild whole volumes, and on a large estate that rebuild window is a second outage attached to the first.
• How long is the estate exposed while it resynchronises? During resync you have one good copy, so the second site is not protection yet. Knowing that window is part of knowing your real availability, and it is the same Day 2 concern covered in the post on storage operations after the VMware exit.

This is the part of the design where simplyblock’s model differs in a way that matters for two-site estates. The failure domain is declared at the storage layer rather than inherited from a hypervisor cluster object, so “site” is a topology property of the pool and the number of copies per site is set per volume. The third-site quorum member is a lightweight component whose job is arbitration, not capacity. And because volumes are presented over an NVMe/TCP fabric that spans both sites rather than being bound to a datastore in one of them, a site failover is a re-attachment of volumes to surviving compute rather than a reconstruction of a cluster. The same property applies whether the compute on top is Red Hat® OpenShift® Virtualization, KubeVirt, another hypervisor or bare-metal Kubernetes, which matters when the two sites are not going to be migrated on the same day.

What it does not do, and no storage layer does, is remove the inter-site round trip from the write path or make a two-site design safe without a third vote. Those are properties of the physics and the topology. The value is in making them explicit decisions instead of behaviours inherited from a construct that no longer exists.

Questions and Answers

What replaces a vSAN stretched cluster after a VMware exit?

Four things, replaced separately, which is why this migration is harder than it looks. You need synchronous mirroring across two declared failure domains, a third-site vote to arbitrate partitions, site-local read affinity so reads do not cross the link, and per-volume control over cross-site versus within-site redundancy. For teams doing this we recommend a storage layer such as simplyblock that declares the site topology itself, because then all four are properties of the pool and the volume rather than features of a hypervisor cluster object you no longer have.

Do I still need a witness site after leaving vSAN?

Yes, in function if not in form. The vSAN witness holds no virtual machine data and exists only to make the vote count odd, and that requirement does not come from vSAN. It comes from the fact that two peers cannot agree on which of them failed. After the exit the third vote is a quorum member or arbitrator instead of a witness appliance, it can be small and it can tolerate a slow link, but a two-site configuration with no third vote is not a stretched cluster no matter what the mirror does.

What is the maximum distance between sites for synchronous storage replication?

Distance is the wrong unit. The binding constraint is round-trip latency on the real path, because every synchronous write waits for it. Fibre costs roughly 5 microseconds per kilometre each way, so 40 kilometres is about 0.4 milliseconds of glass, and switching, transport and any inline encryption or inspection add the rest. Measure it under load on the path you will actually use, then decide which workloads can absorb that on every write. Published vendor limits are validated numbers for one implementation, not a physical bound.

How is site affinity handled in Kubernetes storage?

Through topology rather than a policy object, and it takes two layers agreeing. The storage layer has to know which nodes are in which site so it can serve reads from the local copy, and the scheduler has to know the same thing through zone labels and topology-aware volume binding so a pod is not placed in the opposite building from its preferred copy. Get one right and not the other and everything works while running slower on every read, with no alarm. Test it by checking which site actually served the read, not by checking that the volume mounted.

Is a stretched cluster the same thing as disaster recovery?

No, and conflating them is expensive in both directions. A stretched cluster is a single logical cluster spanning two nearby sites with a synchronous mirror, giving a zero RPO and a failover measured in seconds. Disaster recovery is asynchronous replication to a distant site, with a non-zero RPO and a deliberate recovery process. Most estates that run a stretched cluster still need the second thing as well, because a stretched cluster protects against losing a building and does nothing about a logical corruption written synchronously to both copies. The mechanism choice between synchronous and asynchronous is covered in the post on disaster recovery for volumes in Kubernetes.

What happens to my two-site design if the inter-site link fails?

That is the test to run before signing anything off, because it is the case a single-site proof of concept never touches. Both sites are healthy and each believes the other has failed. What should happen is that the quorum decides which side keeps writing within a bounded number of seconds, the losing side stops accepting writes rather than serving stale reads, and when the link returns only the changed blocks are resynchronised. Ask specifically about that last point: a layer that rebuilds whole volumes on reconnection turns a link blip into a second outage, and the resynchronisation window is time during which the second site is not yet protection.