Where the Latency Number Came From: esxtop, DAVG and KAVG After the vSphere Exit

GAVG/cmd 41.7.

Read off an esxtop screen at two in the morning, that figure was the beginning of a diagnosis rather than the end of one. Forty-one milliseconds is bad. But the same row carried two more numbers, and those two said whose fault it was: whether the device at the far end was genuinely slow, or whether the host had been sitting on commands it had not yet managed to send. One number was the symptom. The other two were the attribution.

After a VMware® vSphere exit, the symptom survives and the attribution does not. The application still reports that its writes are slow, and the person holding the pager still has to decide which layer to wake somebody up about. What has gone is the thing that made that decision cheap: a single vendor owning the host, the queue and the path to the device, and therefore able to timestamp every boundary in between.

This is not an article about what latency is. Simplyblock has written that one, and the metrics themselves are defined in IOPS, throughput and latency explained and in the glossary entries on NVMe latency, p99 storage latency and tail latency. This is about the decomposition: how one number was split into three, and where each of those three lives now.

What the three numbers actually were

The counters come from esxtop’s disk views, and the relationship between them is arithmetic rather than approximate.

GAVG/cmd was guest average latency: the round trip as the virtual machine experienced it, measured at the point where the guest’s I/O left the virtual SCSI or NVMe controller and came back. This is the number that correlates with a user complaining, and on its own it is almost useless for diagnosis. It is a total.

DAVG/cmd was device average latency: the time from the moment the VMkernel handed a command to the storage adapter until the completion came back. Everything outside the host lives in this figure. The HBA or initiator, the cable or the switch fabric, the target port, the array’s own controller and cache, and finally the media. A high DAVG meant the problem was somewhere the host could see but not fix.

KAVG/cmd was kernel average latency: time the command spent inside the VMkernel before it ever reached the adapter. In a healthy system this is supposed to be close to zero, conventionally under a millisecond or two. KAVG is mostly queue time, and its dominant component was surfaced separately as QAVG.

And the relationship: GAVG = KAVG + DAVG. That identity is the whole point. A guest complaining about 41.7 milliseconds with DAVG 40.9 and KAVG 0.8 has a storage problem. The same 41.7 with DAVG 3.2 and KAVG 38.5 has a host problem, and it is usually queue depth: the adapter or the device queue is full, commands are backing up in the kernel, and the storage at the far end is answering perfectly well but is not being asked fast enough. Two adjacent columns separated an array escalation from a queue-depth setting, and they did it in one screen without instrumenting anything.

Two more counters completed the picture. QUED and %USD reported how much of the available queue depth was in use, which is what turned “KAVG is high” into “and here is why.” ABRTS/s counted commands the host gave up on after their timeout expired, which is a path or target problem rather than a slowness problem, and it meant something entirely different from a large DAVG.

Why the split existed at all

It is worth being precise about why this decomposition was available, because that is what determines whether anything can reproduce it.

The split was not clever instrumentation. It was a consequence of ownership. One vendor’s kernel sat between the guest and the adapter, so it could stamp a timestamp when the guest’s command arrived and another when it was dispatched to hardware. The difference is KAVG by construction. The same kernel owned the adapter driver, so it could stamp dispatch and completion, and the difference is DAVG by construction. Every boundary that mattered was a boundary inside code that one party wrote.

That is a structural property, and it does not survive disaggregation of the stack. After the exit, the path from an application to persistent media crosses a guest kernel, a virtualization layer, a container runtime, a CSI-provisioned block device, a node’s own block layer, a network, and a storage system, and those pieces come from different projects and different vendors. Nobody is positioned to timestamp all of them. The honest conclusion, and the one worth stating early, is that no storage vendor restores the old single-screen picture by itself, and any vendor claiming otherwise is describing their own half and calling it the whole.

What can be rebuilt is the attribution, from more than one instrument, provided each layer reports its own portion truthfully rather than collapsing it into a total.

Rebuilding the latency picture on a platform you are still designing? Talk to us before the storage layer is chosen, while what it exposes is still a selection criterion rather than something to work around. Talk to a storage architect

Where each number goes after the exit

The destination’s default instruments are good, and they are arranged differently. The Linux block layer reports per-device latency through iostat -x as r_await and w_await, queue occupancy as aqu-sz, and saturation as %util. In-guest, the same tools work inside a KubeVirt virtual machine exactly as they did inside a vSphere one.

The trap is that the node-side numbers do not mean what the host-side numbers meant. On ESXi, DAVG stopped at the adapter and everything past it was one opaque figure, but that figure covered a fabric most teams could reason about because it was short and dedicated. On a node attached to network storage, await on that block device includes the node’s own queueing, the network round trip, queueing at the storage node, and the device service time, all added together. The destination’s most obvious instrument collapses precisely the distinction that DAVG and KAVG existed to draw.

Mapped counter by counter:

Table 1: The esxtop decomposition and its destination-side equivalents.

Two rows deserve emphasis. The device and fabric row is where the loss is concentrated, because one ESXi counter becomes three independent questions and the node can only answer them as a sum. And the CSI layer appears nowhere in this table at all, which surprises people. CSI is a control-plane interface: it provisions, attaches and detaches volumes. No I/O passes through it and it has no latency to report. If a runbook says “check the CSI driver” when an application is slow, that runbook is pointing at the wrong component.

The network is the genuinely new instrument. On a dedicated block fabric, congestion was somebody else’s department. Over TCP, retransmissions and congestion control appear at the application as latency with no storage-side signal whatsoever, so connection-level counters and NIC statistics become part of the storage diagnosis rather than adjacent to it. The fabric trade-offs behind that are covered in NVMe over TCP vs iSCSI, and path behaviour under failure in NVMe multipath and Kubernetes failover.

What a storage layer can honestly tell you

Given that no component sees the whole path, the useful question when evaluating a storage layer is narrow: does it report its own portion as separate figures, or as one?

A storage system that terminates NVMe/TCP at its own nodes is measuring, on its side of the network, the things a host cannot see: how long a command waited in its queues, and how long the media actually took to service it. Those are the two halves of what DAVG used to bundle, and they are separable at the storage nodes because that is where both events occur. Publishing them as distinct series means an operator can subtract: if guest-visible latency is 40 milliseconds, node queueing is small, and the storage system reports 2 milliseconds of device service time and 1 millisecond of internal queueing, then roughly 37 milliseconds is in the network, and that is a fabric investigation rather than a storage escalation. The subtraction only works if each side publishes its own figure honestly instead of reporting a client-side total.

There is a second, less obvious effect, and it is architectural rather than a matter of which metrics get exported. Where a hyperconverged design runs the storage stack on the same cores as the workloads it serves, host CPU contention and storage latency become the same number again. A busy node makes its own storage slow, the storage reports that it is slow, and nothing in either figure distinguishes “the media is saturated” from “the storage stack could not get scheduled.” The decomposition collapses for structural reasons, in the same way it did on vSphere and for the opposite cause.

This is where storage efficiency turns into an observability property rather than only a cost one. Simplyblock’s own capability comparison puts write IOPS per CPU core at roughly 100,000 against about 5,000 for Red Hat® OpenShift® Data Foundation, and raw-to-effective capacity at about 1.25:1 at FTT1 against roughly 3:1 for three-way replication. Those figures are from simplyblock’s battlecard and are based on publicly available information as of September 2026, so treat them as our reading rather than as independently verified. The relevant consequence here is simply that the less host CPU a storage layer consumes per unit of work, the less often contention on the node is a plausible explanation for a latency number, and the fewer candidate causes an operator has to eliminate. Ceph remains a reasonable choice when performance is not the main goal, but its CPU cost per operation makes contention a standing hypothesis rather than an unusual one.

Both deployment models are available, and the choice is worth making with this in mind rather than only on cost. Separating storage onto its own nodes buys attribution: the storage layer’s numbers are then measured somewhere the workload cannot perturb. Converged deployment gives that up in exchange for fewer machines, which is often the right trade, but it should be a decision rather than a default. The decision is reversible later without changing vendors, which is more than could be said for the platform being left.

Three things are worth requiring from any candidate, in the order they matter. Device service time and internal queueing must be published separately, not as one storage latency figure. The figures must be per volume, because a cluster-wide average hides exactly the one noisy neighbour that prompted the investigation. And percentiles must be available rather than means alone, because a mean will not show the behaviour that actually breaks applications. What to do with those percentiles once you have them, and how per-volume limits keep one workload from becoming another’s latency, is the subject of storage quality of service.

None of this is the forecasting problem. Whether next quarter’s workloads will fit is a different loop on a different timescale, handled in VMware capacity planning after the exit. This is the problem of an estate that is slow right now, and of an operator who no longer has two adjacent columns telling them who to call. The columns are rebuildable. They just come from three places instead of one, and it is worth knowing which three before the first incident rather than during it. The wider set of post-migration operational habits is covered in day 2 storage operations after the VMware exit, and the datastore-layer decisions that precede all of this in replacing the VMware datastore on Kubernetes.

Questions and Answers

What replaces esxtop after a vSphere exit?

Nothing replaces it as a single screen, and a vendor promising otherwise is describing their own layer. The practical replacement is three instruments read together: in-guest or node-side iostat -x for the total and for node queueing, connection and NIC counters for the network, and the storage system’s own per-volume metrics for queueing and device service time. Simplyblock’s recommendation is to make the third of those a selection criterion: choose a storage layer that publishes device service time and internal queue time as separate per-volume series, because that is the one part of the old decomposition an operator cannot reconstruct from the node.

Is high await on a Kubernetes node the same thing as a high DAVG?

No, and assuming so is the most common misreading. DAVG stopped at the storage adapter and covered a short, dedicated path. Node-side await on a network-attached device includes node queueing, the network round trip, remote queueing and media service time in a single figure. It is closer to GAVG in scope than to DAVG. To get back to something DAVG-like you have to subtract the storage system’s reported figures from the node’s, which is only possible if the storage system publishes them.

Does the CSI driver affect storage latency?

Not on the data path. CSI is a control-plane interface that provisions, attaches, detaches and snapshots volumes. Application I/O does not traverse it. CSI problems show up as volumes that will not attach, pods stuck pending, or slow snapshot operations, never as slow reads and writes on an already-attached volume. Runbooks that list “check CSI” as a latency step should be corrected, because that step costs time during an incident and cannot succeed.

Why does a converged deployment make latency harder to attribute?

Because the storage stack and the workloads compete for the same CPU. When a node is busy, its storage gets slower, and both the guest-visible number and the storage system’s own number rise together with nothing to distinguish media saturation from the storage stack failing to get scheduled. Separating storage onto its own nodes means the storage layer’s figures are measured where the workload cannot perturb them. Converged deployment is frequently the right trade for hardware reasons, but the attribution cost should be a known cost rather than a surprise during an incident.

Which metrics should we require from a storage layer during evaluation?

Three, in order. Device service time and internal queue time published separately rather than combined into one latency figure. Per-volume granularity, since a cluster-wide average conceals the single noisy volume that triggered the investigation. And percentiles rather than means alone, because means do not show the tail behaviour that actually breaks applications. Ask to see these in a running system during the evaluation, not in a datasheet, since which figures are actually exported per volume is where products differ most.