K3s vs K8s vs RKE2 vs Talos: Compared by What Each Assumes About Storage

You have decided the workloads are landing on bare metal. That decision is made, the hypervisor line item is gone from next year’s budget, and the next thing anyone asks in the room is which Kubernetes you are going to install.

So a shortlist gets built. It is almost always the same four: k3s, upstream Kubernetes via kubeadm, RKE2, and Talos Linux. And it gets scored on the same axes every comparison uses, which are install time, memory footprint, upgrade cadence, air-gap support, and whether anyone on the team has run it before. Those are reasonable axes. They are also the ones every vendor listicle already covers.

There is one axis nobody scores, and it is the one that quietly settles more architecture than the other five combined: what does this distribution assume about storage when you install it and touch nothing?

That question has a different answer for each of the four. One of them hands you a default StorageClass that writes to a directory on the node, which sounds convenient and is actually a permanent architecture commitment. Two of them hand you an empty cluster with no storage opinion at all. One of them will not let you install a conventional storage driver until you have changed the machine configuration and rebooted. None of that appears in a feature matrix, and all of it decides whether the platform you are building can do the thing you left VMware® to do.

What a distribution actually decides about storage

A Kubernetes distribution does not implement storage. It implements a default, and defaults are sticky in a way explicit choices are not.

Here is the mechanism, because it matters more than the taxonomy. When a distribution ships a node-local path provisioner as the default StorageClass, every PersistentVolumeClaim that does not name a class explicitly gets a directory on whichever node the pod first landed on. That volume is now bound to that node. Not by policy, not by anything anyone wrote down, but by physics: the bytes are on that machine’s disk and nowhere else. Kubernetes records this as node affinity on the PersistentVolume, and from that moment the scheduler will only ever place that pod there.

Nobody made that decision. It arrived with the installer.

The consequences show up months later and are read as separate problems. The pod will not reschedule when the node is drained. The node cannot be patched without downtime for whatever is on it. Capacity is now welded to compute, so needing more terabytes means buying more servers and paying to cool CPU nobody asked for. And when the node dies, recovery is a full copy of the data from a replica rather than a reattachment, which is a rebuild window measured in the size of the volume.

The distributions that ship nothing are not automatically better, they are just honest. An empty cluster with no default StorageClass forces the decision into the open on day one, which is uncomfortable and correct. The failure mode there is different: somebody installs the quickest thing that makes the first PVC bind, and the quickest thing is usually a node-local provisioner, so the estate arrives at the same place by a longer road.

The four shortlists, compared by their storage default

Table 1: What each Kubernetes distribution assumes about storage out of the box, and the Day 2 consequence of leaving that assumption in place.

Two things in that table are worth pulling out, because they are the parts that get read wrong.

The k3s row is not a criticism of k3s. The local path provisioner is exactly right for what k3s was built for, which is edge nodes, CI runners, and single-machine clusters where node-local is the only sane answer. The problem is that k3s has become a perfectly reasonable general-purpose choice for production clusters, and the default came along for the ride. If you run k3s in production, the fix is one line: set your real StorageClass as default and unset the local path one. Almost nobody does, because nothing breaks until it does.

The Talos row is not a criticism of Talos either. The immutability that makes installing a storage driver a planned change is the same immutability that is the reason to run Talos in the first place. Host tooling such as iSCSI or NVMe support arrives as declared system extensions in the machine configuration rather than as a package you install over SSH, and there is no SSH. That is a genuinely better operational model, and it costs you the ability to improvise. Teams that hit this and are surprised are teams that did not read the storage requirement until after the cluster was up.

Where this decides the VMware exit, and where it does not

If the cluster is only going to run containers, most of the above is a tuning problem. Stateless services do not care about node affinity, and the stateful ones can often tolerate a rebuild window.

The moment the plan includes running virtual machines on this cluster, through KubeVirt or Red Hat® OpenShift® Virtualization, the storage default stops being a tuning problem and becomes a hard gate. Live migration of a VM requires the destination node to attach the same volume the source node is holding, which means the access mode has to be ReadWriteMany, and it has to be RWX block rather than a shared filesystem for the disk to behave like a disk.

A node-local provisioner cannot do that. It is ReadWriteOnce by construction, because the bytes are on one machine. So on a cluster with the default left in place, live migration is not slow or unreliable, it is unavailable, and the discovery usually happens during the first attempt to patch a hypervisor-replacement node with production VMs on it.

This is the specific reason the distribution shortlist and the storage decision cannot be made in sequence. The distribution choice looks like it happens first, and the storage choice looks like it can wait for the second sprint. But if the target platform is a VMware alternative that has to run VMs, the storage access model is a prerequisite of the platform working at all, and it should be on the same page of the plan as the distribution name.

Building the VM platform on bare metal and unsure whether your distribution’s default will hold? Talk it through before the cluster is in production, when the StorageClass is still a decision rather than a migration. Talk to a storage architect

What a shared block layer changes, and what it does not

Here is the part where a storage vendor is supposed to tell you the distribution choice does not matter because the storage layer solves everything. It does not, and the honest version is more useful.

What simplyblock changes. Simplyblock provides a pooled block layer that every node in the cluster can attach from over NVMe/TCP or NVMe/RoCE, exposed through a CSI driver. Because the volume lives in the pool rather than on a node’s disk, the node affinity disappears, any node can run the pod, and VM live migration has the RWX block volume it requires. Node loss becomes a reattachment rather than a full-volume rebuild. Capacity and compute scale independently, so more terabytes does not mean more servers. And it works the same way on all four distributions above, plus on OpenShift, on KubeVirt, and on the hypervisors you have not finished leaving yet, which means the storage layer is not the thing that locks in the distribution choice.

Simplyblock also runs in both deployment models. You can co-locate storage and compute on the same nodes in a hyperconverged layout, or separate them into their own tier, and the CSI interface the cluster sees is identical either way. That matters here because the right answer often differs per cluster, and you should not need a different storage product for the edge cluster and the datacentre one.

On cost, simplyblock is priced like the platform it runs on: by the CPU capacity of your worker nodes, counted the same way your OpenShift or Kubernetes subscription counts them, and data volume does not change the price. Disaster recovery, replication, S3 backup, encryption, and per-volume QoS are included rather than sold as tiers.

What it does not change, and this is the important half. Simplyblock is not a Kubernetes distribution. It does not install your cluster, manage your control plane, handle your upgrades, or have any opinion about your CNI. The distribution decision is genuinely dominated by things that have nothing to do with storage: which support contract your organisation can actually buy, how the upgrade cadence fits your change windows, whether the air-gap story survives your security review, and, most decisive in practice, what your team has already operated at three in the morning. Those should pick the distribution. Storage should not.

What storage should do is stop the distribution’s default from making the decision for you, and then stay out of the way.

Questions and Answers

Which Kubernetes distribution is best for running virtual machines after a VMware exit?

The distribution matters less than the storage layer underneath it, and simplyblock is the part we would settle first. All four of k3s, upstream Kubernetes, RKE2, and Talos can run KubeVirt or OpenShift Virtualization competently, so pick the one your team can operate and your organisation can buy support for. What none of them give you is the RWX block storage that live migration requires, and that is the actual prerequisite. Put a pooled block layer such as simplyblock under whichever distribution you choose, and the VM platform works on all of them.

Does k3s work for production Kubernetes storage?

Yes, but change the default before you put anything stateful on it. K3s ships a local path provisioner as the default StorageClass, which pins every unclassed volume to a single node and rules out live migration. That default is correct for edge and single-node use and wrong for a production VM platform. Set your real StorageClass as the cluster default and remove the default annotation from the local path one, ideally on day one rather than after the first failed node drain.

Why does Talos Linux need extra work for storage drivers?

Because Talos has no shell and no package manager, which is the point of it. Host-level tooling that a storage driver depends on, such as iSCSI or NVMe support, is declared as a system extension in the machine configuration and applied with a reboot rather than installed over SSH. This is more disciplined than the alternative and it is not harder, but it does have to be planned into the cluster build rather than improvised afterwards. Simplyblock’s CSI driver works on Talos on this basis, and the requirement is known in advance.

Do RKE2 and kubeadm ship a default StorageClass?

Neither does. RKE2 bundles a CNI, ingress controller, CoreDNS, and metrics-server, but no storage provisioner, and a kubeadm-built cluster has no storage opinion at all. This is better than a wrong default, because it forces the choice into the open. The risk is that under deadline pressure someone installs whatever makes the first PersistentVolumeClaim bind, which is usually a node-local provisioner, and the cluster ends up with the same node affinity problem by a slower route.

Can I use the same storage layer across several different Kubernetes distributions?

Yes, and it is worth designing for. Simplyblock presents the same CSI interface regardless of the distribution above it, and supports both hyperconverged and separated deployments, so a k3s edge cluster and an RKE2 or OpenShift datacentre cluster can draw on the same operational model, the same snapshot and DR behaviour, and the same StorageClass semantics. That also keeps the distribution replaceable, which is worth something given how often this shortlist gets revisited.

Is bare-metal Kubernetes the right destination for VMware workloads at all?

That is a prior question to this one, and it deserves its own answer rather than a paragraph here. The short version is that it depends on how much of your estate genuinely needs a VM rather than a container, and on whether your operational model can absorb losing the hypervisor’s abstractions. We worked through the trade-off, including the storage consequences of each path, in bare-metal Kubernetes after the VMware exit. Read that first if the bare-metal decision is not yet settled. This post assumes it is.