Raw Device Mapping and the Multi-Writer Flag: What Happens to Shared Disks After the vSphere Exit

Two virtual machines. One disk. Both of them writing to it, on purpose.

That sentence describes a configuration that has been running quietly in VMware® vSphere estates for fifteen years, and it is one of the reliable ways for a migration plan to come apart in its second week. The VMs move. The disk moves. Then the cluster service inside the guests refuses to come online, or, worse, comes online in two places at once and a filesystem that two operating systems both believe they own starts taking writes from both of them.

What makes this catch teams is that the shared disk does not look like a special case from the outside. It is a VMDK, or it is a LUN, and migration tooling treats it as one more piece of storage to copy. What it actually is: a promise about concurrency that three separate pieces of vSphere machinery were making together, and that no single Kubernetes object reproduces.

One distinction is worth making before anything else, because conflating these two is the most common way this goes wrong. A shared block device is not the same requirement as a shared filesystem. ReadWriteMany gets reached for here constantly, and in most migration plans it is being used as the live migration precondition it genuinely is, not as a way for two guest operating systems to write to the same raw device under arbitration. Different requirement, different failure mode. The access mode says how many nodes may open the volume. It says nothing at all about what happens when two of them write to the same block.

This post takes the three vSphere constructs apart, says where each lands on the destination platform, and is direct about the one that does not travel.

What the shared disk actually was

Three separate mechanisms sat behind “two guests share a disk” on vSphere, and they are not interchangeable. Estates usually contain all three, configured by different people in different years, and the migration inventory rarely distinguishes them.

The multi-writer flag is the simplest. Set the sharing mode on a VMDK and the hypervisor stops applying the file lock that normally gives one VM exclusive access to that disk file. Several VMs can then open it at once. That is the whole of it. The flag removes an obstacle, and it adds nothing: no arbitration, no fencing, no notion of which guest currently owns which blocks. It came with real constraints, eager-zeroed thick provisioning and no snapshots of the shared disk among them, and it was always documented as a facility for clustering software that manages its own access, with Oracle RAC as the canonical case. Give a multi-writer disk to two guests that each assume exclusive ownership and you get precisely the corruption you would expect.

Raw device mapping is two different constructs wearing one name, and the difference is the entire point of this post. An RDM is a small mapping file on a datastore that points at a raw LUN. In virtual compatibility mode the hypervisor stays in the path: it intercepts the guest’s SCSI commands and passes through essentially reads and writes, which keeps snapshots and the familiar disk features working. In physical compatibility mode, sometimes called pass-through, the hypervisor gets out of the way and hands the guest’s SCSI command set almost intact to the array underneath.

That second mode is the one that mattered, because of the third mechanism.

SCSI-3 persistent reservations are how a guest cluster decides who is allowed to write. Nodes register with the device, one of them holds a reservation, and the reservation type excludes the others from writing while it stands. On failover the surviving node preempts the dead one’s registration and takes ownership. This is the fencing primitive underneath Windows Server Failover Clustering with shared disks, and it is why the quorum or witness disk in those configurations was never an ordinary volume.

Here is the part that gets lost, and it determines everything that follows: vSphere never enforced those reservations. The array did. Physical-mode RDM was plumbing, a decision to stop interfering so that the guest could talk to a storage target that implemented the reservation semantics itself. The multi-writer flag, by contrast, never carried reservations at all, which is exactly why it was suitable for Oracle RAC, whose own layer does the arbitration, and unsuitable for failover clustering with shared disks, which needs the storage to do it.

Later vSphere releases blurred this by allowing reservation-capable clustered VMDKs on a datastore rather than requiring a raw LUN, which removed the operational awkwardness of RDMs for many teams. It did not change the architecture. The guarantee still came from the storage target. It only stopped being visible.

And one more thing disappeared quietly in the same move: VMFS itself was arbitrating access to the datastore for every VM on it, all the time. Nobody configured that. It was a property of the filesystem the estate happened to be standing on, and it is the reason concurrency almost never came up as an architectural decision on the old platform.

Where each construct lands

Taken apart this way, the three mechanisms land in three different places, and one of them does not land at all.

Table 1: Each shared-disk construct, where it lands on the destination platform, and which layer is responsible afterwards.

The first row is the clean one. volumeMode: Block with an RWX access mode gives you a raw device that more than one node is permitted to open, which is a faithful translation of what the multi-writer flag did, including the part where nothing protects you. KubeVirt adds a shareable marker on the disk that relaxes the exclusive locking QEMU would otherwise apply, which is the same idea one layer up. If the workload was Oracle RAC or a cluster-aware filesystem that brought its own locking, this is a real translation and the pattern survives the move.

The second row is a non-event, and it is worth saying so because virtual-mode RDMs are common and teams tend to assume all RDMs are hard. A virtual-mode RDM was a LUN being used as an ordinary disk with the hypervisor still mediating. It becomes an ordinary volume. Migrate the data and move on.

The third row is where the work is. KubeVirt can present a volume to the guest as a SCSI LUN rather than as a generic virtual disk, and it can be configured to pass persistent reservation commands through to the underlying device, with a helper process on the node handling the reservation traffic. That is the honest analogue of physical-mode RDM. It is also a feature-gated capability whose behaviour and support status have moved between releases, so treat the version in front of you as the source of truth rather than any summary, including this one.

The fourth row is the one to slow down on.

The shared-disk inventory is usually wrong in the same direction. If you have VMs with multi-writer disks or physical-mode RDMs and nobody can currently say which of them actually need reservation semantics, that question is worth answering before the migration design is fixed rather than during the cutover. Talk to a storage architect

The reservation is the part that does not travel

There is no access mode for this. There is no field in a PersistentVolumeClaim, no capability bit in the CSI specification, and no annotation that any two storage vendors would interpret the same way, that says “this volume honours SCSI-3 persistent reservations.” The access modes describe how many nodes may attach a volume and in what mode. They are a statement about attachment, not about arbitration, and they were never designed to carry this.

So the question cannot be answered at the Kubernetes layer. It has to be answered down the whole chain, and every link has to say yes:

The guest cluster software issues the reservation commands. The guest’s storage stack has to be able to emit them, which means the disk has to be presented as something that accepts a full command set rather than as a simplified virtual disk. The virtualization layer has to pass them down rather than absorb or emulate them. The CSI driver has to expose the volume in a form where that pass-through is possible. And the storage target at the bottom has to implement the reservation semantics and enforce them across every path by which the volume can be reached, which is the requirement people forget, because a reservation enforced on one path and ignored on another is worse than no reservation at all.

Break any link and the failure is not a clean error at deployment time. It is a cluster that forms, passes validation, runs in production for a while, and then fences incorrectly during the one event the cluster existed to survive.

Two further wrinkles are worth knowing about before you design around this.

The first is protocol. SCSI-3 persistent reservations are, as the name says, a SCSI construct. NVMe defines its own reservation model that covers the same ground with a different command set and a different vocabulary. Neither is a superset of the other in practice, and a guest cluster that speaks SCSI reservations is not automatically served by a target that implements NVMe reservations. On an NVMe fabric this becomes a translation question rather than a pass-through question, and the answer depends on the specific platform rather than on the fabric in general. Anyone who tells you the fabric settles it is skipping a step.

The second is that a shared block device and a stuck one look alarmingly similar from the outside. A volume that will not attach because a previous node has not released it produces a multi-attach error, and the first instinct when that appears is often to widen the access mode until the error stops. That converts a visible, recoverable attachment problem into an invisible, unrecoverable data problem. The error is the platform doing its job. The shared-disk case is the narrow, deliberate exception to it, and the two should never be resolved with the same change.

None of this means the destination platform is deficient. It means the guarantee was always supplied by the storage layer, and the exit is the moment that stops being hidden. Simplyblock’s position here is the same as its position on the rest of the shared-access question: the storage layer is where this belongs, it is what the array was quietly doing all along, and the useful thing to know is precisely which half of the requirement you have. For the shared-filesystem half, simplyblock serves RWX with pNFS parallel access from the same pool that serves the block volumes, with no separate filesystem stack beside the block storage. For the shared-block-with-reservations half, the honest answer is that this is a chain to verify end to end against your specific guest cluster software, not a checkbox, and that a large share of the workloads currently holding a shared disk should not be carrying one across the migration at all.

Which is the last question, and the one worth the most.

Translate it, or retire it

The instinct on a migration is to reproduce what exists. For shared disks that instinct is often wrong, because a meaningful share of these configurations exist for a reason that expired years ago: the application needed high availability, the platform of the day offered shared-disk clustering as the way to get it, and the configuration outlived the constraint. The exit is a reasonable moment to ask whether it still earns its place, and it is much cheaper to ask now than after the pattern has been rebuilt on a platform where it is harder to operate.

Retire the shared disk when the application has a modern replication-based clustering mode and you are only using shared-disk clustering out of habit. A number of widely deployed databases and application servers now offer a shared-nothing high availability mode where each node keeps its own copy and the cluster coordinates at the application layer. Those modes remove the reservation requirement completely, turn each node’s storage back into an ordinary volume, and generally fail over faster than a disk-ownership handover does. If that path is open, the migration is the right moment to take it, and the shared-disk question disappears rather than being solved.

Translate the shared disk when the application genuinely does concurrent block access as part of how it works. Cluster-aware databases and cluster filesystems are not using a shared disk as a high availability trick; concurrent access is the architecture, and the arbitration lives in the application itself. These translate cleanly, because what they need from the platform is the multi-writer property, not reservations: a raw device that more than one consumer may open, which the destination expresses directly.

Verify before you commit when the application needs the storage to arbitrate and has no replication-based alternative you are willing to adopt. Here the chain has to be proven end to end, on the actual platform version, with the actual driver and the actual target, and the proof has to include a failover test where a node is removed abruptly rather than shut down politely. The graceful path passes almost everywhere. The abrupt one is what the reservation exists for.

A note on scope, because it is easy to let this sprawl. The temptation is to turn the shared-disk question into a per-application project, one workstream for each database engine in the estate. It is not that. It is one storage question, asked once, whose answer happens to sort the workloads into three groups. Sort them first. The inventory is usually smaller than feared, and it is frequently dominated by a handful of VMs that no longer need the arrangement at all.

What you should not do is discover the answer during cutover. The shared-disk cases are the ones where the failure mode is silent, arrives late, and lands on the data rather than on the availability. Everything else in a migration announces itself. This one waits.

Questions and Answers

Can I just use ReadWriteMany for my shared disks after the migration?

Only for the multi-writer case, and only with volumeMode: Block. RWX means several nodes may open the volume; it does not mean anything arbitrates their writes. That is an exact match for what the multi-writer flag did, so it works for applications that bring their own locking, such as cluster-aware databases and cluster filesystems. It is not a substitute for SCSI-3 persistent reservations, and using it for a failover cluster that expects the storage to fence will produce a cluster that appears to work until the first ungraceful node loss.

What is the actual difference between virtual and physical compatibility mode RDM, for migration purposes?

Virtual mode is not a migration problem: the hypervisor was mediating the SCSI path, the guest was using the LUN as an ordinary disk, and it becomes an ordinary volume on the destination. Physical mode is the one to flag, because it exists specifically to pass the guest’s full SCSI command set to the array, which in practice means the guest cluster is using persistent reservations. Inventory the two separately. Treating every RDM as hard wastes effort, and treating every RDM as easy is how the reservation cases get missed.

Does simplyblock support SCSI-3 persistent reservations for guest clustering?

The accurate answer is that this is a chain, not a single product feature, and it should be verified end to end for your specific guest cluster software, platform version and driver rather than taken from any vendor’s summary. What simplyblock does provide directly is the shared-access half of the problem: RWX with pNFS parallel access served from the same erasure-coded pool as the block volumes, over the same fabric, with no second filesystem stack to stand up beside the block storage. For workloads whose real requirement is concurrent access rather than storage-enforced fencing, that is the complete answer. For workloads that genuinely need the storage to arbitrate, the conversation worth having is usually whether the application still needs shared-disk clustering at all.

We are on Ceph today. Does that change the analysis?

Not structurally. The same chain has to hold, and the same question applies to the client protocol and the driver in front of it. Ceph remains a good option when performance is not the main goal, and this is one of the areas where the question to ask is not whether it is capable but whether the specific path from your guest cluster software down to the target is a supported and tested one. The analysis in this post is protocol-agnostic on purpose: the layer that enforces the reservation is the storage target, whichever target that is.

How do we find every shared disk in the estate before we start?

Look for three things rather than one. Disks with the multi-writer sharing mode set, mapping files pointing at raw LUNs with their compatibility mode recorded, and, in later environments, datastore-resident disks flagged as clustered. Then cross-reference with the guests: the configuration that matters is a disk attached to more than one VM, and the compatibility or sharing setting tells you which of the three constructs you are looking at. It is worth doing this before the datastore replacement design is settled, because the answer occasionally changes which destination storage layout makes sense.