Kubernetes Multi-Tenant Storage Isolation: Per-Tenant NVMe/TCP After VMware Migration

Enterprises migrating multiple business units from VMware to Kubernetes carry an isolation requirement that namespace boundaries alone cannot satisfy. On VMware, tenant isolation was enforced at three layers simultaneously: vSphere resource pools capped CPU and memory per tenant, NSX micro-segmentation controlled network traffic between tenant VMs, and vSAN storage policies assigned dedicated IOPS limits and separate datastore capacity per business unit. These three layers together gave each tenant a predictable, bounded environment even when running on shared physical hardware.

Kubernetes namespace isolation handles the first layer reasonably well: ResourceQuotas cap CPU and memory, and NetworkPolicies restrict pod-to-pod traffic. The third layer, storage isolation, is almost always missing. Most Kubernetes storage deployments use a single shared storage pool with a single StorageClass for all tenants. One tenant’s backup job, a bulk snapshot operation, or a high-ingest logging pipeline can saturate the shared storage I/O path and degrade every other tenant’s database or application. The noisy-neighbor problem that vSphere resource pools solved for compute migrates to the storage layer and stays unsolved.

This guide covers why hyperconverged storage cannot fully address this problem, how disaggregated NVMe/TCP isolates storage by design, and how to configure per-tenant StorageClasses on Kubernetes that enforce QoS at the storage layer rather than only at the scheduler.

The Storage Tenant Problem in Kubernetes

Kubernetes namespaces establish resource quotas and network isolation, but the storage subsystem is shared by default. A StorageClass is a cluster-wide resource: any namespace with permission to use it can provision volumes from the same storage pool, competing for the same IOPS budget, the same I/O queues, and the same network bandwidth.

Hyperconverged storage intensifies this problem rather than solving it. When storage software runs on the same nodes as workloads, a tenant’s compute activity directly competes with storage I/O for CPU cycles, memory bandwidth, and NIC queue depth. A backup process for one tenant’s namespace triggers CRC computation, compression, and network transfer on the node where another tenant’s latency-sensitive application is running. The resource pools that vSphere enforced at the hypervisor level have no equivalent in hyperconverged Kubernetes storage: the hypervisor and the storage stack are collapsed into the same Linux process space.

The organizational consequence is that platform teams cannot make credible per-tenant SLA commitments on a shared Kubernetes cluster using shared hyperconverged storage. Infrastructure teams that migrate multiple business units from a well-tuned vSphere environment are typically moving from a model with documented per-tenant performance guarantees to a model with no storage-layer isolation at all.

Per-tenant storage isolation in Kubernetes requires three things working together:

• Per-tenant QoS enforced at the storage layer, not just at the Kubernetes scheduler
• Physical or logical separation of the I/O path between tenants, so one tenant’s throughput cannot saturate another tenant’s bandwidth
• Independent storage credentials per tenant, so a compromised tenant workload cannot access another tenant’s volumes

Namespace-level ResourceQuotas do not provide any of these. They cap the total PVC storage capacity a namespace can request, but they do not cap the IOPS those volumes can consume, restrict the network bandwidth the storage protocol uses, or enforce any access control between volumes provisioned from the same pool.

How Disaggregated NVMe/TCP Creates Storage-Layer Isolation

Disaggregated storage solves the compute-storage coupling problem at the architectural level. When storage runs on dedicated hardware nodes connected over a standard network protocol, storage I/O never competes with workload CPU cycles on the same machine. A bulk backup operation running in one tenant’s namespace consumes bandwidth on the storage network and CPU on the storage node. The compute node running the other tenant’s database is not in the I/O path at all: it submits NVMe commands over the TCP connection and receives completions, with no local CPU involvement proportional to I/O volume.

NVMe/TCP enforces this separation at the protocol level. Each compute node connects to storage using standard NVMe host software over a TCP socket. The storage node handles command processing, data placement, and replication entirely on its own CPU and memory. The compute node’s CPU load from NVMe/TCP is proportional to connection setup and command serialization, not proportional to I/O volume. This architecture makes noisy-neighbor effects from one tenant’s I/O workload structurally impossible on the compute side: the noise happens on the storage node, which is dedicated storage hardware with no tenant workloads running on it.

Simplyblock implements per-tenant QoS at the NVMe namespace level. Each StorageClass maps to an NVMe namespace pool with configurable IOPS ceiling, throughput ceiling, and replication factor. When the storage node enforces QoS, it enforces it per namespace, per volume, and per initiator connection. A tenant’s high-ingest workload cannot borrow IOPS headroom from another tenant’s namespace because the NVMe queue scheduler on the storage node enforces the limit before I/O reaches the physical drive.

Migrating multiple business units from VMware and need per-tenant storage guarantees? Simplyblock’s disaggregated NVMe/TCP provides storage-layer QoS isolation between Kubernetes namespaces on commodity hardware, without proprietary storage arrays. Talk to a storage architect

Per-Tenant StorageClass Design on Kubernetes

StorageClass parameters are the Kubernetes-native mechanism for expressing per-tenant storage policy. Each StorageClass maps to a storage configuration that the provisioner enforces when volumes are created. With simplyblock, StorageClass parameters translate directly to NVMe namespace configuration on the storage node: IOPS limits, throughput limits, and replication factor are set per StorageClass and applied per volume provisioned from that class.

The following examples show a three-tier tenant configuration: production databases, standard application workloads, and development environments.

Production database tier (high-throughput, strict QoS):

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: simplyblock-tenant-prod
5
  annotations:
6
    storageclass.kubernetes.io/is-default-class: "false"
7
provisioner: csi.simplyblock.io
8
parameters:
9
  replication: "3"
10
  qos_rw_iops: "20000"
11
  qos_rw_mbytes: "800"
12
volumeBindingMode: WaitForFirstConsumer
13
allowVolumeExpansion: true

Standard application tier (balanced QoS):

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: simplyblock-tenant-standard
5
provisioner: csi.simplyblock.io
6
parameters:
7
  replication: "2"
8
  qos_rw_iops: "8000"
9
  qos_rw_mbytes: "300"
10
volumeBindingMode: WaitForFirstConsumer
11
allowVolumeExpansion: true

Development tier (resource-constrained):

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: simplyblock-tenant-dev
5
provisioner: csi.simplyblock.io
6
parameters:
7
  replication: "2"
8
  qos_rw_iops: "2000"
9
  qos_rw_mbytes: "100"
10
volumeBindingMode: WaitForFirstConsumer
11
allowVolumeExpansion: true

To complete the tenant isolation model, platform teams restrict StorageClass access per namespace using Kubernetes RBAC. A namespace-scoped Role grants create permission on persistentvolumeclaims and get permission on the specific storageclass objects that namespace is permitted to use. Combined with a LimitRange that sets minimum and maximum PVC sizes, this prevents tenant workloads from requesting volumes outside their assigned tier and from creating unbounded volumes that exhaust pool capacity.

Namespace RBAC for StorageClass access control:

1
apiVersion: rbac.authorization.k8s.io/v1
2
kind: Role
3
metadata:
4
  name: storage-tenant-prod
5
  namespace: team-finance
6
rules:
7
- apiGroups: [""]
8
  resources: ["persistentvolumeclaims"]
9
  verbs: ["create", "get", "list", "watch", "delete"]
10
- apiGroups: ["storage.k8s.io"]
11
  resources: ["storageclasses"]
12
  resourceNames: ["simplyblock-tenant-prod"]
13
  verbs: ["get", "list"]

This RBAC pattern ensures that a workload in the team-finance namespace can only provision volumes from simplyblock-tenant-prod. It cannot access volumes provisioned for other namespaces, and it cannot switch to a lower-cost StorageClass to circumvent capacity quotas. The combination of StorageClass QoS parameters and namespace RBAC provides the storage-layer equivalent of vSphere resource pool isolation.

Comparing Tenant Isolation Approaches

The table below evaluates four storage architectures against the isolation requirements enterprises carry from VMware environments.

Table 1: Tenant isolation capabilities by Kubernetes storage architecture.

Enterprises migrating from VMware find that the disaggregated NVMe/TCP approach is the closest equivalent to vSphere’s layered isolation model. Storage policy assignment per-VM in vSphere translates to StorageClass assignment per-namespace in Kubernetes, with QoS enforced at the same layer: the storage software itself, not the compute hypervisor. The key structural difference from proprietary SAN alternatives is that disaggregated NVMe/TCP works on commodity hardware without per-site vendor licensing. The same storage pool serves all tenant namespaces, with isolation enforced in software at the NVMe namespace level. Simplyblock licenses per usable storage TB, so cost scales with provisioned capacity, not with the number of teams using the platform or the number of CPU cores on compute nodes.

Questions and Answers

Why doesn’t Kubernetes namespace isolation solve the storage tenant problem?

Kubernetes namespace isolation controls CPU and memory via ResourceQuotas and network traffic via NetworkPolicies, but it does not enforce any I/O rate limits on persistent volumes. Two pods in different namespaces provisioned from the same StorageClass compete for the same storage pool’s I/O capacity with no scheduling constraint between them. A bulk import job in one namespace can exhaust available IOPS and degrade a latency-sensitive database in another namespace. Solving this requires per-StorageClass QoS limits enforced by the storage backend, not by the Kubernetes scheduler.

How does simplyblock enforce per-tenant QoS in Kubernetes?

Simplyblock enforces QoS at the NVMe namespace level on dedicated storage nodes. Each StorageClass maps to a configuration with qos_rw_iops and qos_rw_mbytes parameters. When the simplyblock CSI driver provisions a volume from that StorageClass, the storage node applies those limits to the NVMe namespace backing the volume. A tenant workload that issues more I/O than the limit allows sees requests queued or throttled on the storage node, without affecting other tenants’ namespaces. The QoS enforcement happens in the storage layer, not at the Kubernetes scheduler, so it applies regardless of what the compute node is doing.

What storage architecture should enterprises use when migrating multiple teams from VMware?

Enterprises migrating multiple teams from a well-tuned vSphere environment need storage that can enforce per-team SLA commitments without proprietary hardware dependencies at every site. Disaggregated NVMe/TCP with per-tenant StorageClasses maps directly to the vSphere model: storage policies per-VM in vSphere become StorageClasses per-namespace in Kubernetes, with QoS limits enforced by the storage software. Simplyblock runs on commodity NVMe hardware and works with any Kubernetes distribution, so the same storage architecture applies across the entire post-migration environment without re-purchasing vendor storage arrays for each new deployment site.

Can simplyblock enforce per-tenant storage quotas in Kubernetes?

Yes, at two levels. First, per-volume QoS caps control the maximum IOPS and throughput any single volume provisioned from a StorageClass can consume. Second, namespace-level Kubernetes ResourceQuotas cap the total storage capacity a namespace can provision across all PVCs. The combination of QoS caps and capacity quotas gives platform teams the equivalent of vSphere datastore limits (capacity) plus storage policy IOPS limits (throughput), both enforced in standard Kubernetes primitives without custom controllers.

How does disaggregated NVMe/TCP prevent the noisy-neighbor problem in multi-tenant Kubernetes storage?

Hyperconverged storage, where the storage software runs on the same nodes as workloads, means that a tenant’s I/O-intensive job competes for CPU cycles and memory bandwidth with other tenants’ pods running on the same node. Disaggregated NVMe/TCP places storage software entirely on dedicated storage hardware. Compute nodes submit NVMe commands over a TCP socket and receive completions. The storage node handles all I/O processing independently. One tenant’s backup job consumes CPU and memory on the storage node, not on the compute node where another tenant’s database is running. The compute workloads are structurally isolated from each other’s storage I/O by the network boundary between compute and storage tiers.