iowait is a Linux CPU utilization metric that measures the percentage of time a CPU core is idle while at least one I/O operation is pending for which that core is waiting. Unlike normal CPU utilization, iowait does not represent useful work: the processor is available, but the thread it needs to run is stalled waiting for a disk read or write to return. When iowait is high, applications are not CPU-bound. They are storage-bound, and the CPU is sitting idle as a consequence.
iowait shows up in the output of top as the wa field in the CPU line, and in iostat -c as the iowait column. It is a per-CPU metric: a system with 32 cores can have high iowait on cores that are blocked on storage while other cores run unrelated workloads normally. Persistent elevated iowait on application nodes is a reliable signal that the storage layer cannot keep pace with the workloads running on those nodes.
How iowait Is Measured
The Linux kernel tracks five CPU time states per core: user, system, idle, iowait, and steal. iowait increments when a CPU core is in the idle state and there is at least one I/O request outstanding from a process that would run on that core when the I/O completes.
Tools that report iowait:
top: thewapercentage in the CPU summary line (press1to see per-core breakdowns)iostat -c: theiowaitcolumn in the CPU utilization reportsar -u: historical CPU utilization including iowait- Prometheus
node_exporter:node_cpu_seconds_total{mode="iowait"}per core
In Kubernetes, node-level iowait is visible through Prometheus and Grafana dashboards. Per-pod I/O latency requires additional instrumentation at the storage layer: simplyblock exposes per-volume I/O metrics via its Kubernetes storage observability integration.
iowait vs CPU Utilization: Why the Distinction Matters
A system reporting 70% CPU utilization and 0% iowait is compute-bound: adding CPU or optimizing application code will help. A system reporting 15% CPU utilization and 35% iowait is storage-bound: the CPU is mostly idle, but applications cannot make progress because storage is not delivering data fast enough.
The two conditions look different in monitoring and require different responses. iowait-heavy workloads benefit from faster storage, not more CPU. Scaling Kubernetes pods horizontally does not help if all pods are waiting on the same slow storage backend. Adding more application replicas to a storage-bound cluster increases the number of threads waiting on I/O, which can actually increase storage pressure rather than relieve it.
For database workloads, the relationship is direct. PostgreSQL and MySQL commit latency is bounded by the fsync latency of the underlying block device. High iowait correlates with high commit latency, which limits transactions per second regardless of how much CPU or memory is available.
🚀 Seeing persistent high iowait on your Kubernetes database nodes? Simplyblock NVMe storage delivers sub-millisecond P99 latency, which eliminates iowait caused by slow storage backends. 👉 See NVMe storage cost and performance optimization
Diagnosing iowait in Kubernetes
When iowait is elevated on Kubernetes worker nodes, diagnosing the root cause requires correlating the CPU metric with storage I/O metrics:
- Identify which nodes show elevated iowait using
node_cpu_seconds_total{mode="iowait"}in Prometheus. A node with high iowait is a candidate for storage investigation. - Correlate with disk I/O metrics using
node_disk_io_time_seconds_totalandnode_disk_read/write_bytes_total. High disk utilization alongside high iowait confirms the storage device is the bottleneck. - Identify which pods are generating I/O using
kubectl top podsor container-level I/O metrics fromcAdvisor. The pods with the highest I/O throughput on the saturated node are the likely contributors. - Check storage device latency directly. If the underlying block device is showing P99 latencies above 5-10ms for random 4K reads or writes, that is the latency translated directly into iowait on the application node.
NVMe-backed volumes typically eliminate iowait caused by device latency. iowait that persists on NVMe-backed deployments usually indicates network congestion on the NVMe/TCP path or a software bottleneck in the storage stack.
How NVMe Latency Reduces iowait
The mechanism is direct: lower storage latency means threads wait less time for I/O to return, which means CPU cores spend less time in the iowait state and more time running application code.
A storage device with 5ms average latency holds each I/O thread blocked for 5ms per operation. A device with 0.3ms average latency holds the same thread for 0.3ms. For a database issuing thousands of random reads per second, this difference compounds across every operation and every core. The aggregate CPU time recovered from iowait at sub-millisecond latency is available for transaction processing, which increases throughput from the same hardware.
In multi-tenant Kubernetes clusters, the effect is also visible in scheduling density. Pods that are storage-bound hold their CPU allocation while waiting for I/O, preventing the Kubernetes scheduler from using that allocation for other work. Reducing iowait frees CPU quota faster, which allows more pods to be scheduled on the same node.
How Simplyblock Reduces iowait
Simplyblock’s NVMe/TCP storage delivers sub-millisecond P99 latency on commodity Ethernet, which directly reduces the time application threads spend in iowait. Teams migrating from SAN-attached spinning disk or slow cloud block storage to simplyblock typically observe significant iowait reductions on database nodes within hours of the migration.
The per-volume IOPS and throughput limits in simplyblock also prevent individual workloads from saturating the storage layer, which prevents the secondary iowait spikes that occur when noisy workloads monopolize storage I/O and force all other pods into extended wait states.
Related Terms
iowait sits at the intersection of Linux performance diagnostics and storage architecture.
- Storage Latency
- P99 Storage Latency
- Storage Metrics in Kubernetes
- Database Performance vs Storage Latency
- What Is NVMe Storage
Questions and Answers
What is a normal or acceptable iowait percentage?
There is no universal threshold, but sustained iowait above 5-10% on application nodes is generally worth investigating. Brief spikes during heavy I/O operations are normal. Persistent elevated iowait during steady-state operation indicates the storage layer is a bottleneck. For latency-sensitive workloads like databases, even moderate iowait can cause P99 latency spikes that violate SLOs.
Can high iowait exist alongside low CPU utilization?
Yes. iowait is counted as idle CPU time from the scheduler’s perspective, so a system with 80% idle CPU and 15% iowait shows only 5% active utilization in a naive CPU usage reading. Tools like top report these separately. A system that appears “underutilized” by CPU may be fully storage-bound, and adding more compute capacity will not improve performance.
How do you reduce iowait in a Kubernetes cluster?
The primary lever is storage latency. Replacing slow block storage (spinning disk, high-latency cloud volumes) with fast NVMe-backed storage is the most direct intervention. Secondary levers include adding storage QoS limits to prevent noisy workloads from saturating the storage layer, distributing I/O across more storage nodes, and tuning application I/O patterns (batch size, queue depth, sync behavior).
Is iowait a per-CPU or per-system metric?
iowait is tracked per CPU core. top shows a system-level average by default; pressing 1 in top shows per-core breakdowns. It is possible for some cores to show high iowait while others are busy, depending on which cores are running I/O-heavy threads. Prometheus node_exporter exposes per-core iowait via node_cpu_seconds_total{mode="iowait",cpu="N"}.
Does iowait appear in container or pod metrics?
iowait is a node-level metric, not a container-level metric. Kubernetes does not expose iowait directly for individual pods. However, correlating node iowait with pod I/O metrics from cAdvisor (container_fs_reads_bytes_total, container_fs_writes_bytes_total) can identify which pods are contributing to the node-level iowait. Storage platforms like simplyblock expose per-volume I/O latency metrics that provide more precise per-workload visibility.