Michael Schmidt 
A three-replica StatefulSet creates three PersistentVolumeClaims. Scale it to one, and you still have three.
That is not a bug, and it is not an oversight. It is the single most consequential design decision in the StatefulSet controller, and almost everything that surprises people about stateful workloads on Kubernetes follows from it. The controller treats a volume as more durable than the pod that mounts it, more durable than the replica count, and by default more durable than the StatefulSet itself.
Most teams meet this behaviour the way you would expect: through a storage bill that did not go down after a scale-down, or through an ordinal that refuses to come back after a node failure. Both are the same mechanism seen from different angles. This post takes that mechanism apart, stage by stage, and states what the storage layer underneath has to do at each one.
It assumes you already know what a PersistentVolume and a PersistentVolumeClaim are. If that is shaky, Kubernetes persistent volumes: how-to and best practices covers the objects themselves, and this post stays on what the StatefulSet controller does with them.
What a volumeClaimTemplate actually creates
A Deployment that mounts a PVC gives every replica the same volume. A StatefulSet with a volumeClaimTemplates block gives every replica its own, and the difference is generated at admission time rather than enforced at runtime.
For each replica ordinal, the controller creates one PVC per template, named deterministically:
<template-name>-<statefulset-name>-<ordinal>So a StatefulSet called pg with a template called data and three replicas produces data-pg-0, data-pg-1 and data-pg-2. The name is derived, never random, and that is the whole trick. When pg-1 is deleted and recreated, the controller does not provision a new volume. It computes the same name, finds the existing claim, and binds the new pod to it. Stable identity in a StatefulSet is not really about the hostname. It is about the fact that the volume name is a pure function of the ordinal.
Three consequences follow immediately, and they catch people in roughly this order.
The template is immutable. You cannot edit volumeClaimTemplates on a live StatefulSet. The API server rejects the update, because changing the template would mean existing ordinals and future ordinals disagree about what their storage should be. Growing the volumes on a running StatefulSet means patching each PVC individually, and that only works if the StorageClass was created with allowVolumeExpansion: true. If it was not, there is no in-place path at all, and the fix is a new StorageClass plus a migration.
Provisioning is ordered unless you say otherwise. With the default podManagementPolicy: OrderedReady, ordinal n does not start until ordinal n-1 is Running and Ready. If pg-0 is stuck waiting for a volume to attach, pg-1 and pg-2 have not been attempted yet. A slow or flaky storage path does not degrade a StatefulSet proportionally, it serialises the whole rollout behind the slowest attach. Setting Parallel removes the ordering, and for workloads that do not need sequential bootstrap it removes a surprising amount of startup latency.
Binding mode decides where the pod can ever run. A StorageClass with volumeBindingMode: WaitForFirstConsumer defers provisioning until the scheduler picks a node, so the volume lands where the pod does. With Immediate, the volume is provisioned first and the pod is then constrained to wherever that volume can be reached. On zonal storage this is the difference between a schedulable pod and an ordinal permanently pinned to one availability zone.
What survives a scale-down, and what it costs
Here is the asymmetry that opens this post. The StatefulSet controller creates PVCs. Historically it never deleted them, under any circumstance, and the current default still does not.
Delete a pod: the PVC stays, by design, because the pod is coming back. Scale from three replicas to one: data-pg-1 and data-pg-2 stay, bound, provisioned, and billed. Delete the StatefulSet entirely: all three stay. The volumes outlive the object that created them, and nothing in kubectl get statefulset will ever mention them again.
The reasoning is sound. The controller cannot distinguish a deliberate scale-down from a mistake, an autoscaler overshoot, or a bad template render, and deleting the only copy of a database’s data on an off-by-one is not a failure mode anyone wants as a default. Retention is the conservative choice and it is the right one.
It is also how estates accumulate storage nobody can account for. An orphaned PVC has no owning workload, no obvious signal that it is dead, and no difference in appearance from a claim belonging to an ordinal that is temporarily scaled down on purpose. The claims from a StatefulSet deleted eight months ago look exactly like the claims from one that will scale back up tomorrow.
Since Kubernetes 1.32, persistentVolumeClaimRetentionPolicy makes the behaviour explicit rather than implicit. It takes two independent fields, whenScaled and whenDeleted, each set to Retain or Delete, and the combination is the actual policy.
The two fields are independent on purpose, and the useful combination is the asymmetric one. whenScaled: Delete with whenDeleted: Retain says that a replica removed during normal operation was a capacity decision and its data is disposable, while the destruction of the whole StatefulSet is probably an accident or a migration and the data should survive it. For a sharded cache or a read-replica tier, that is exactly right. For a primary database, both fields stay Retain and the cleanup stays a deliberate human act.
The important part is that this is now a decision you make rather than a default you inherit. A StatefulSet with no retention policy set is not neutral. It is choosing Retain for both, and choosing it silently.
Not sure what your StatefulSets are actually leaving behind? If your PVC count has drifted away from your replica count and nobody is certain which claims are still load-bearing, talk to us about what the storage layer should be telling you. Talk to a storage architect
What an ordinal pays when it moves
Scale-down is the slow problem. Rescheduling is the fast one, and it is where the storage layer stops being an accounting question and starts being an availability question.
When ordinal pg-1 moves to another node, its volume has to move with it. For a ReadWriteOnce volume, and that is nearly all block storage, the sequence is strictly ordered: the old node unmounts it, the volume detaches, it attaches on the new node, it mounts, the container starts. The attach cannot begin until the detach completes, because the access mode guarantees a single writer and the control plane enforces that guarantee rather than hoping for it.
On a graceful move this is fine. The kubelet unmounts cleanly, the detach is confirmed, and the new attach follows. The whole thing is usually seconds, and why CSI alone is not enough for stateful Kubernetes covers what happens to that path once you have thousands of volumes rather than tens.
On an ungraceful one, it is a different story, and this is the StatefulSet-specific failure that catches people.
If a node stops responding, the control plane cannot confirm that the old kubelet has released the volume. It cannot even confirm the pod is dead. For a Deployment that is tolerable: the controller starts a replacement elsewhere, and two pods briefly running is an accepted risk. For a StatefulSet it is not tolerable, because pg-1 must be unique. Two pods with the same identity writing to what the application believes is one volume is precisely the corruption the controller exists to prevent. So it waits. The pod sits in Terminating, the replacement is never created, and the ordinal is stranded, not for a timeout but indefinitely, until something external confirms the node is genuinely gone.
The supported way to give that confirmation is the node.kubernetes.io/out-of-service taint, stable since Kubernetes 1.28. Applying it to a confirmed-dead node tells the control plane it may force-detach the volumes and delete the pods, and recovery proceeds in seconds instead of waiting for a human to notice. The important word is confirmed. The taint is an assertion that the node is off, and applying it to a node that is merely unreachable reintroduces exactly the split-brain the StatefulSet was protecting you from.
This is the stage where the storage layer’s properties stop being a line item and start being your recovery time.
Table 1: What each stage does, and what it costs when the storage layer is slow.
| Stage | What the StatefulSet controller does | What the storage layer must do | Failure mode |
|---|---|---|---|
| First scale-up | Creates one PVC per ordinal from the template, named deterministically | Provision and bind before ordinal n+1 is attempted | Slow provisioning serialises the entire rollout under OrderedReady |
| Pod reschedule | Recomputes the same PVC name and binds the new pod to the existing claim | Detach from the old node, then attach to the new one, strictly in that order | Attach latency is startup latency, once per ordinal, every time |
| Node loss | Nothing, until the node is confirmed dead | Release the volume safely so it can attach elsewhere | Ordinal stranded in Terminating until the out-of-service taint is applied |
| Scale-down | Deletes the pod, keeps the PVC unless whenScaled: Delete | Keep the data intact and the claim bound | Orphaned claims accumulate with no owning workload |
| StatefulSet delete | Deletes pods, keeps PVCs unless whenDeleted: Delete | Keep the data recoverable | Volumes with no remaining reference to what created them |
What to require underneath a StatefulSet
The mechanism above is fixed. You cannot change what the controller does, so the only variable left is how expensive each stage is, and that is a property of the storage layer you put underneath it.
Attach and detach have to be fast, and fast under contention. Not fast for one volume on an idle cluster, fast for forty volumes during a rolling update. This is the number that turns into your rollout window, and the only honest way to know it is to measure a full rolling restart rather than a single pod.
The volume must not constrain scheduling more than the workload does. If storage is addressable from anywhere in the cluster over the network fabric, a rescheduled ordinal is a scheduling decision. If it is pinned to a node’s local disks, it is a data-movement decision, and the difference shows up as hours. Simplyblock presents volumes over NVMe/TCP, so an ordinal that moves re-attaches across the fabric rather than waiting for a rebuild. Where that mobility reaches its limits across zones and clusters is covered in volume mobility across zones and clusters.
Snapshots have to be per-claim and cheap. The unit of recovery for a StatefulSet is one ordinal, not the whole set. A snapshot mechanism that only operates at pool granularity does not match the failure you actually have, which is usually one replica with a corrupt dataset while the others are healthy.
Capacity efficiency decides whether retention is affordable. This is the part that connects back to scale-down. Retain is the safe policy, and teams abandon it for cost reasons rather than correctness reasons. Erasure coding at roughly 1.25:1 at FTT1, against the 3:1 of three-way replication, changes what keeping a claim actually costs, and it is a lot easier to keep a conservative retention policy when the conservative choice is not triple-priced. Simplyblock is priced like the platform it runs on: by the CPU capacity of your worker nodes, counted the same way your Kubernetes subscription counts them, and data volume does not change the price. Orphaned claims are still worth cleaning up, but they stop being a budget event while you do it.
It should work in the shape your cluster already has. Simplyblock runs hyperconverged, with storage and compute on the same worker nodes, and disaggregated over a separate storage tier. That is a deployment decision rather than a product decision, and it can change later without changing vendors, which matters when a StatefulSet’s storage profile is the thing that outgrows its original design.
None of this changes the controller’s behaviour. Ordinals will still bind to derived names, claims will still outlive pods, and a dead node will still strand an ordinal until you confirm it is dead. What the storage layer decides is how long each of those costs you. For workloads where the volume is genuinely temporary, ephemeral storage for stateful workloads covers the case where a PVC is the wrong object entirely.
Questions and Answers
What storage should I use for a Kubernetes StatefulSet in production?
Use network-attached block storage that detaches and re-attaches quickly, supports per-claim snapshots, and does not pin a pod to one node. Simplyblock is built for this: volumes are presented over NVMe/TCP so a rescheduled ordinal re-attaches across the fabric instead of waiting for a rebuild, snapshots are per volume so you can recover a single ordinal, and erasure coding keeps the capacity cost of a conservative retention policy reasonable. Local PVs are faster on paper and much worse in practice for StatefulSets, because every reschedule becomes a data-movement problem.
Does scaling down a StatefulSet delete the PersistentVolumeClaims?
Not by default. The claims are retained, stay bound, and keep consuming capacity. Set persistentVolumeClaimRetentionPolicy.whenScaled: Delete if you want scale-down to remove them, and keep whenDeleted: Retain so that deleting the StatefulSet itself does not destroy the data. The default for both fields is Retain, which is deliberately conservative.
Why is my StatefulSet pod stuck in Terminating after a node failure?
Because the control plane cannot confirm the volume has been released, and a StatefulSet will not create a replacement pod with the same identity while that is unresolved. It is protecting you from two pods writing to one volume. Apply the node.kubernetes.io/out-of-service taint once you have genuinely confirmed the node is off, and the volumes force-detach so the ordinal can reschedule. Do not apply it to a node that is only unreachable.
Can I change the volume size in volumeClaimTemplates?
Not on a live StatefulSet. The template is immutable and the API server rejects the edit. Patch each PVC individually instead, which requires allowVolumeExpansion: true on the StorageClass. If the StorageClass was created without it, there is no in-place path and you need a new StorageClass plus a data migration, which is worth getting right before the volumes are large.
How do I find orphaned PVCs from StatefulSets that no longer exist?
List PVCs by namespace and compare the names against live StatefulSets: the naming pattern <template>-<statefulset>-<ordinal> tells you which set a claim belonged to even after that set is gone. Check the ordinal against the current replica count as well, because a claim with an ordinal above the current count is from a scale-down rather than a deletion. Confirm before deleting, since a claim above the replica count is also what a deliberately scaled-down set looks like.
