Michael Schmidt 
Ephemeral storage in Kubernetes is easy to reach for. emptyDir is one line in a pod spec, it lives on local node disk or memory, and it feels indistinguishable from a fast local SSD, right up until it isn’t there anymore. The failure mode is not a crash or an error message. It’s silence: the pod comes back, the application restarts, and the data it was relying on is simply gone, because nothing about ephemeral storage was ever designed to survive the pod that created it.
Many teams do not discover this the hard way until it happens in production, usually during a node drain, an autoscaling event, or a routine upgrade, not during a load test. This post covers what actually happens to ephemeral data on eviction and node failure, how to decide between emptyDir, hostPath, local persistent volumes, and network-attached options, and how disaggregated NVMe/TCP persistent volumes deliver the near-local latency that made ephemeral storage attractive in the first place, without the data-loss risk.
Why emptyDir Feels Like the Right Answer, Until It Isn’t
emptyDir exists for a real reason: it is fast, requires no provisioning, and matches the mental model of “temporary scratch space.” Build caches, sort spill files, session buffers, and intermediate query results all fit that description, and for genuinely disposable data, emptyDir is the correct tool.
The trouble starts when a team reaches for emptyDir because it’s convenient, not because the data is actually disposable. A stateful workload, a message queue’s local segment files, a cache that is expensive to rebuild, or a database’s data directory, ends up on node-local storage because it was the path of least resistance during a proof of concept, and nobody revisited the decision before the workload reached production. Kubernetes will not stop you from doing this. Nothing about the pod spec syntax signals that you have just made your data as durable as the node it happens to be scheduled on, which is to say: not durable at all.
hostPath compounds the same risk in a different way. It mounts a specific path from the host filesystem into the pod, which can survive a single pod restart on the same node, but ties the workload to that exact node and offers no protection against node failure, disk failure, or the node being drained and decommissioned. Both emptyDir and hostPath share the same underlying limitation: the data’s lifetime is bound to a single node, and Kubernetes treats nodes as disposable by design.
What Actually Happens to Ephemeral Data on Eviction, Reschedule, and Node Failure
Three routine cluster events all produce the same outcome for ephemeral storage: data loss.
Pod eviction. When a node hits disk pressure, memory pressure, or another eviction signal, the kubelet evicts pods to protect node stability. The evicted pod’s emptyDir contents are deleted as part of cleanup. This is not a bug. It is exactly how emptyDir is documented to behave, but the eviction itself is often triggered by conditions the application owner never sees: another pod on the same node filled the disk, or a batch job spiked memory usage cluster-wide.
Pod reschedule. Deployments reschedule pods constantly: rolling updates, HorizontalPodAutoscaler scale-down events, node cordoning ahead of maintenance, bin-packing by the scheduler. Each reschedule can land the pod on a different node, and emptyDir content never travels with it. A pod that depended on locally cached state starts cold every time, and a pod that depended on locally written durable state loses that state outright.
Node failure. Hardware failure, kernel panics, cloud provider maintenance events, and spot instance reclamation all remove a node from the cluster with no graceful pod termination at all. Whatever was in emptyDir or hostPath on that node is unrecoverable, full stop. There is no reattachment step to wait for, because there is nothing left to reattach.
The pattern across all three: Kubernetes correctly treats nodes as fungible infrastructure, which is the entire point of running a scheduler, but ephemeral storage inherits that fungibility for the data too. A persistent volume backed by a real storage layer breaks that coupling. The volume is not part of the node’s identity, so a reschedule or node failure becomes a reattachment event measured in seconds, not a data-loss event with no recovery path.
Ephemeral storage failure modes rarely show up in a load test. If your stateful workloads still depend on node-local storage, talk to us before the next node drain finds the gap for you. Talk to a storage architect
emptyDir vs hostPath vs Local PV vs Network PVC vs NVMe/TCP PVC: A Decision Table
Use this table to match the storage option to the durability the workload actually needs, not the durability it happened to get by default.
Table 1: Ephemeral and persistent storage options compared for stateful Kubernetes workloads.
| Storage type | Survives pod eviction | Survives node failure | Reattach time | Latency profile | Typical use case |
|---|---|---|---|---|---|
emptyDir | No, deleted on eviction | No, data is gone | N/A | Fastest (local disk/RAM) | Build caches, spill files, disposable scratch |
hostPath | No | No | N/A, node-bound | Fast (local disk) | Node-level debugging, single-node testing |
| Local persistent volume | Yes, if rescheduled to the same node | No | N/A, node-bound | Fast (local NVMe/SSD) | Latency-critical workloads that tolerate node pinning |
| Generic network PVC (CSI) | Yes | Yes, depending on driver | Minutes, varies by driver and replication | Adds network/replication overhead | General persistent workloads, moderate latency needs |
| simplyblock NVMe/TCP PVC | Yes | Yes | Seconds, fabric-attached reattach | Near-local, NVMe/TCP fabric | Databases, queues, and other latency-sensitive stateful workloads |
The gap in the middle of this table is where most teams get burned. emptyDir and hostPath are fast and simple, but offer zero durability. Generic network-attached PVCs restore durability but often reintroduce the latency and reattachment delay that pushed teams toward ephemeral storage to begin with. Closing that gap, durability without giving latency back, is the actual architecture problem.
How Disaggregated NVMe/TCP Persistent Volumes Close the Gap
The reason teams reach for ephemeral storage in the first place is almost always latency: local disk avoids the round trip to a remote SAN or a slow network filesystem. Disaggregated NVMe/TCP storage removes the tradeoff by keeping the volume off the node, on a fabric-attached pool, while still delivering latency close to local disk over standard Ethernet.
The mechanism is what makes the difference. Because the volume lives on the storage pool rather than on the node’s local disk, a pod eviction, reschedule, or node failure triggers a volume reattachment to wherever the pod lands next, typically a matter of seconds, rather than a total loss of the underlying data. The pod restarts, the CSI driver reattaches the existing volume, and the application resumes from where it left off. No rebuild, no cold cache, no missing data directory.
This architecture also decouples two decisions that ephemeral storage forces together: how fast the storage is, and how tightly it is bound to a single node’s fate. With simplyblock’s disaggregated NVMe/TCP (and NVMe/RoCE) design, compute and storage scale independently, so adding storage capacity does not mean adding compute nodes, and losing a compute node does not mean losing storage. Simplyblock is priced like the platform it runs on: by the CPU capacity of your worker nodes, counted the same way your OpenShift or Kubernetes subscription counts them, and data volume does not change the price. The same CSI driver and storage layer work across any Kubernetes distribution, so this is not a lock-in decision tied to a specific platform.
For workloads currently running on emptyDir because “it was fast and nobody got around to fixing it,” the practical migration path is straightforward: move the persistent volume claim to a StorageClass backed by NVMe/TCP, keep genuinely disposable scratch data on emptyDir where it belongs, and route anything that would hurt to lose onto the durable, fast tier.
Questions and Answers
What is ephemeral storage in Kubernetes, and why do teams use it for stateful workloads?
Ephemeral storage, most commonly emptyDir or hostPath, is pod-scoped or node-scoped storage that Kubernetes removes when the pod terminates or the node is lost. Teams reach for it because it is fast and requires no provisioning, but that same simplicity means it silently loses data during pod eviction, rescheduling, or node failure, events that happen routinely in any production cluster.
What happens to data in emptyDir when a pod is evicted or a node fails?
The kubelet deletes emptyDir contents as part of normal pod cleanup on eviction, and any data on a failed node is unrecoverable because there is no replication or reattachment mechanism. This is documented, expected behavior, not a bug, which is exactly why it catches teams off guard when a workload that needed durability was never actually configured for it.
Is hostPath ever safe for stateful workloads?
hostPath is appropriate for node-level debugging or single-node test environments, but not for production stateful workloads. It ties data to one specific node, offers no protection against node failure or disk failure, and blocks the pod from being rescheduled elsewhere without losing its data.
How is a network PVC different from ephemeral storage, and why is it still not always enough?
A generic network-attached persistent volume claim survives pod eviction and typically survives node failure, unlike emptyDir or hostPath. The tradeoff is that many network storage backends reintroduce latency and slower reattachment (often minutes) that pushed teams toward ephemeral storage in the first place, so durability comes back at the cost of performance.
How does simplyblock deliver near-local latency without the risks of ephemeral storage?
Simplyblock’s disaggregated NVMe/TCP architecture keeps persistent volumes on a fabric-attached storage pool rather than on node-local disk, so a pod eviction or node failure triggers a volume reattachment measured in seconds instead of permanent data loss, while NVMe/TCP keeps latency close to what local disk delivered.
Does moving off ephemeral storage always cost more?
Not necessarily, and the comparison depends on what you are pricing against. Simplyblock is priced like the platform it runs on: by the CPU capacity of your worker nodes, counted the same way your OpenShift or Kubernetes subscription counts them, and data volume does not change the price. Independent compute and storage scaling means you are not forced to add compute nodes just to gain storage capacity. For workloads where a single data-loss incident means a restore, a customer-facing outage, or lost work, the cost of durable storage is usually smaller than the cost of the incident it prevents.
