How to Benchmark Block Storage Performance: fio and Network Tuning Best Practices

When a distributed block storage benchmark comes back lower than expected, the storage is rarely the first thing at fault. Far more often the test itself is measuring the wrong thing: a cold volume, a single thread that never fills the pipe, or a network stack left at defaults that caps throughput long before the drives are busy.

We see this constantly during proof-of-concept work. A cluster of NVMe drives that benchmarks at tens of GB/s locally will appear to plateau at a fraction of that over the network, and the instinct is to blame the storage layer. The real cause is usually a combination of a missing warm-up phase, a workload that is latency-bound rather than throughput-bound, and an untuned host network.

This guide walks through how to benchmark distributed block storage correctly: how to configure fio, why preconditioning matters, how to interpret mixed read/write workloads, and the specific NVMe/TCP network settings that let a cluster reach the throughput its hardware is capable of. The examples assume a simplyblock deployment on Red Hat® OpenShift®, but the methodology applies to any distributed block storage test.

Why Block Storage Benchmarks Mislead

Before changing any settings, it helps to understand the three things that most often make results look artificially low.

Cold volumes and the warm-up problem

On a freshly provisioned thin volume, the first write to any region triggers allocation. The storage layer has to journal metadata and map new extents before the data lands. This first-touch penalty inflates write latency for the entire duration of a test if the test dataset has never been fully written.

A short ramp_time in fio does not solve this. Ramp time only discards the first N seconds of measurements; it does not guarantee the whole dataset has been allocated. If your runtime is shorter than the time it takes to touch every extent once, you are measuring allocation overhead, not steady-state performance.

The fix is an explicit preconditioning pass: write the entire test dataset sequentially once, then run the measured test against that already-allocated data. Most teams running IO-heavy workloads such as databases exclude the warm-up entirely, because in production the working set is allocated long before peak load arrives.

Mixed read/write is bound by the slower operation

A randrw workload with a 35% write mix sounds read-dominated, but total throughput is gated by the slower operation, which is almost always the write. fio keeps the read/write ratio fixed throughout the run, so if writes slow down (for example because of the cold-volume penalty above), reads are throttled to maintain the 35/65 split. One slow path drags the whole number down.

When you want to understand the true ceiling of a cluster, test pure read and pure write separately first, then layer in the mixed profile that matches your production workload. The separate runs tell you where the real bottleneck is; the mixed run tells you what a real application will feel.

IO amplification and realistic network ceilings

Data protection multiplies the bytes that actually cross the wire, and how much depends on the protection model. With single mirroring (a 1+1 scheme) every write becomes two writes; with the three-way replication that many distributed storage systems default to, it becomes three. Erasure coding, which is what simplyblock uses, keeps that multiplier far lower (more on that below). For a 35/65 write/read mix with roughly 10% protocol and metadata overhead, the effective amplification under 1+1 is:

1
(0.35 × 2 + 0.65) × 1.1 ≈ 1.4×

So 1 GB/s of application IO can mean roughly 1.4 GB/s on the network. On top of that, no TCP link saturates at 100%. A realistic sustained ceiling is around 75% of nominal line rate once you account for headers, congestion control, and host CPU. A “disappointing” benchmark is frequently just these two factors working as designed.

Latency-bound is not the same as capacity-bound

You cannot see the storage engine’s true CPU load from outside the SPDK process. A test running a single thread at a low queue depth produces latency-bound behavior: each IO waits for the previous one to complete, so you measure round-trip latency, not the aggregate IOPS the cluster can actually deliver. Low memory and CPU usage on the storage nodes during a “maxed out” test is the tell-tale sign you are latency-bound, not capacity-bound.

Seeing lower-than-expected numbers in a storage POC? If your cluster plateaus well below its hardware ceiling, a storage architect can review your fio profile and network config and pinpoint the bottleneck. Talk to a storage architect

Designing a Representative fio Workload

A good distributed-storage benchmark mimics production concurrency. The most common mistake is scaling the number of worker pods while leaving each worker single-threaded at a shallow queue depth. That measures how fast one client can talk to the cluster, not how much the cluster can do.

Generate enough outstanding IO

Distributed block storage needs many concurrent IOs in flight to keep every drive and every CPU core busy. There are two levers:

1. Spread across many volumes and subsystems. As a rule of thumb, target at least 6 volumes (PVCs) per storage node, each on its own NVMe-oF subsystem, so queue pairs are distributed rather than concentrated. For a 5-node cluster, that means roughly 30 PVCs across 30 subsystems. Alternatively, raise the number of queue pairs per volume (for example from 3 to 9) so fewer workloads still generate enough parallelism.
2. Raise queue depth per job. Keep numjobs at 8, but run at an iodepth of at least 32. A single job at iodepth=8 cannot fill a 100G pipe no matter how many pods you run.

Always precondition first

Run a sequential write across the full dataset before the measured run, and point the real test at the same file:

1
fio --name=prewrite \
2
    --filename=testfile \
3
    --size=5G \
4
    --rw=write \
5
    --bs=1M \
6
    --direct=1

For an even more thorough warm-up on each volume, write the entire dataset once with a 128K sequential write before the real test begins.

A representative main run

The following profile keeps the realistic mixed workload but removes ramp_time in favor of explicit preconditioning, and uses a queue depth that actually exercises the cluster:

1
[global]
2
refill_buffers
3
directory=/mnt
4
filename=testfile
5
size=5g
6
direct=1
7
time_based=1
8
ioengine=libaio
9
group_reporting
10

11
[workload]
12
blocksize=24k
13
runtime=300
14
iodepth=32
15
numjobs=8
16
rw=randrw
17
rwmixwrite=35

A few notes on block size: at 24K, a mixed workload will tend to saturate the network and the drives before it saturates IOPS, which makes it a good throughput test. If you specifically want worst-case IOPS, use 4K random; if you want to validate streaming bandwidth, use 128K or 1M sequential. Match the block size to the question you are actually asking.

Table 1: Match your fio configuration to the question you are trying to answer.

How to read the fio output

fio prints three numbers that matter: bandwidth (bw), IOPS, and completion latency (clat). Which one you optimize for depends on the test, but a few rules keep the output trustworthy:

• Read latency as percentiles, not averages. A healthy mean clat with a bad p99 or p99.9 means tail latency from queueing or contention. Tail latency is what production workloads actually feel.
• Find the knee of the queue-depth curve. Re-run the same workload at increasing iodepth. While IOPS and bandwidth keep climbing as depth rises, you are latency-bound and have not found the ceiling yet. When they flatten and only latency climbs, you have hit capacity. That plateau is your real number.
• Cross-check the storage nodes, not the client. Low host memory and CPU during a “maxed out” run is the signature of a latency-bound test. True saturation shows up as busy SPDK reactor cores on the storage nodes, which you cannot see from the fio host.
• Compare against an amplification-adjusted baseline. Take the single-drive local result, multiply by the number of drives, divide by your protection amplification, and apply the ~75% network factor. If the cluster lands near that figure, it is healthy, not slow.

Tuning the Network for NVMe/TCP

When initiators (clients) and targets share the same nodes, the bottleneck is rarely the physical link. It is the host TCP stack and the software-defined overlay network (OVN on OpenShift) competing for CPU with the storage engine. The following changes consistently move the needle for NVMe/TCP.

Use jumbo frames

Set the storage NICs to an MTU of 9000. This is the single most important network change. Standard 1500-byte frames force the CPU to process far more packets per gigabyte, and on a 100G link that packet rate becomes the limiting factor long before bandwidth does. Every device in the storage path, NICs and switches, must agree on MTU 9000.

Separate storage traffic and use active/active multipathing

A bond in balanced-SLB mode behaves, for a single host-level flow, much like an active/standby pair. So a “2x100G” bond often delivers only 100G of usable storage bandwidth. The fix is to stop bonding the storage path and instead:

• Create two VLANs, one on each physical NIC port, each with its own IP address.
• Configure NVMe-oF multipathing so both 100G ports are used active/active.

This isolates storage networking from compute networking, lets both links carry storage traffic simultaneously, and allows you to control storage bandwidth independently later. On OpenShift, a NodeNetworkConfigurationPolicy provisions exactly this: two VLAN interfaces at MTU 9000 with static addresses on separate subnets.

Enable hardware offload and zero-copy

Push as much per-packet work as possible into the NIC. Enable segmentation and receive offloads, scatter-gather, and checksum offload; leave LRO off because it can hurt latency for storage traffic. Also grow the ring buffers and spread interrupts across queues so a single core does not become the bottleneck:

1
ethtool -K "$IF" tso on gso on gro on lro off sg on tx on rx on tx-nocache-copy off || true
2
ethtool -G "$IF" rx 8192 tx 8192 || true
3
ethtool -L "$IF" combined 16 || true
4
ethtool -X "$IF" equal 16 || true

Apply the TCP sysctl tuning set

Storage traffic benefits from large socket buffers, a modern congestion control algorithm, and queueing discipline tuned for throughput. These are best applied cluster-wide through a tuning profile (for example a Tuned resource on OpenShift) so every worker node is consistent:

1
[sysctl]
2
net.core.default_qdisc=fq
3
net.core.somaxconn=4096
4
net.core.netdev_max_backlog=8192
5
net.ipv4.tcp_max_syn_backlog=16384
6
net.core.rmem_max=268435456
7
net.core.wmem_max=268435456
8
net.core.optmem_max=268435456
9
net.ipv4.tcp_mem=67108864 67108864 67108864
10
net.ipv4.tcp_rmem=8192 1048576 33554432
11
net.ipv4.tcp_wmem=8192 1048576 33554432
12
net.ipv4.tcp_window_scaling=1
13
net.ipv4.tcp_sack=1
14
net.ipv4.tcp_congestion_control=bbr
15
net.ipv4.tcp_slow_start_after_idle=0
16
net.ipv4.tcp_no_metrics_save=1
17
net.ipv4.conf.all.rp_filter=2
18
kernel.numa_balancing=0

The large rmem/wmem values keep the TCP window open enough to fill a 100G fat pipe, bbr handles high-bandwidth links better than the default congestion control, and disabling tcp_slow_start_after_idle stops throughput from collapsing between bursts. If your fabric supports RDMA, NVMe/RoCE sidesteps most of this host-CPU cost entirely, at the price of a lossless network and RoCE-capable NICs.

Grouped by what they do, the settings above break down like this:

Table 2: What each group of TCP sysctls contributes to NVMe/TCP throughput.

Leave enough cores for the kernel network stack

On a node where the storage engine pins cores, the kernel still needs CPU for TCP processing. Leave roughly 6 to 7 cores non-isolated (a little over 10% on a typical node) for kernel-side networking. Going as low as 4 cores, one of which is shared with system tasks, can be too tight depending on how much TCP traffic the deployment ultimately carries. RDMA reduces this requirement.

Choosing a Cluster Topology and Protection Scheme

The protection scheme you choose changes both the usable capacity and the performance profile of the cluster, so it belongs in the benchmark plan, not as an afterthought. When creating the cluster, enable node affinity so volumes and their primary paths land predictably; this makes results reproducible and keeps the initiator close to its data.

Table 3: Three-way replication versus simplyblock’s erasure-coded schemes, and the effect on capacity, write traffic, and fault tolerance.

Among simplyblock’s schemes, 1+1 gives the best raw IOPS on a first 5-node cluster, particularly for 4K random, while 2+1 delivers comparable real-world performance at much better capacity efficiency because pure 4K random is uncommon in production. A 2+2 scheme lets you take two nodes down at once without losing IO, but it is around 30% slower on writes and is best run on 6 or more nodes so the cluster can rebalance between failures rather than during them.

Why erasure coding beats 3x replication

Most distributed storage systems protect data by keeping whole copies of it, and three-way replication is the common default. It is simple, but it is expensive in exactly the dimension that limits a fast cluster: every logical write becomes three physical writes, so it carries 3.0× write amplification and consumes 3× the raw capacity for the data you actually use.

Simplyblock takes a different approach. It protects data with distributed erasure coding, splitting each write into data chunks plus parity rather than storing full copies. A 2+1 scheme tolerates a node failure at 1.5× amplification, and a 2+2 scheme tolerates two concurrent node failures at 2.0×, which is the same durability three-way replication provides at 3.0×. That is a one-third reduction in both write traffic and raw capacity for the same fault tolerance, and a larger reduction still against three-way replication when you only need single-failure protection.

This gap matters most on a network-bound NVMe/TCP cluster. As the amplification math earlier showed, write traffic, not raw link speed, is usually what saturates the network first. Putting a third to a half less data on the wire per useful write directly raises effective throughput, and because erasure coding spreads each volume across more drives, large reads also pull from more devices in parallel. The net effect is more usable capacity per drive and higher throughput per usable byte than a replication-based system, with no loss of fault tolerance.

The trade-off is small random writes. A write that does not fill a full stripe forces a read-modify-write to recompute parity, so pure 4K random write favors a plain mirror, which has no parity to maintain. That is why 1+1 wins on 4K random while erasure coding matches or beats it on the throughput-oriented workloads most databases and analytics actually run. Reach for a mirror only when you are chasing the lowest possible small-random-write latency; for everything else, erasure coding is the better architecture.

A Repeatable Benchmarking Checklist

Put together, a defensible distributed block storage benchmark looks like this:

1. Precondition every volume with a full sequential write before measuring.
2. Test pure read and pure write separately to find the true ceiling, then run the production-like mixed profile.
3. Generate enough concurrency: 6+ volumes per node across separate subsystems, numjobs=8, iodepth=32 or higher.
4. Account for amplification: multiply application IO by your protection factor plus overhead before comparing to line rate, and expect to top out near 75% of nominal bandwidth.
5. Tune the network before blaming the storage: MTU 9000, active/active multipathing on separate VLANs, hardware offload, and the TCP sysctl set.
6. Leave headroom for the kernel: 6 to 7 non-isolated cores for TCP processing.
7. Re-run and compare against the local single-drive baseline, adjusted for amplification, to confirm the result is sane.

Follow this sequence and the numbers usually converge on the hardware’s real capability. When they do not, you have isolated the variable that matters, which is exactly what a benchmark is supposed to do.

Questions and Answers

Why is my distributed block storage benchmark slower than the local drives?

The most common causes are a cold volume that has never been fully written (so you are measuring first-touch allocation), a single-threaded fio job at low queue depth that is latency-bound rather than capacity-bound, and an untuned host network. Add data-protection amplification and the realistic ~75% TCP saturation ceiling, and a result that looks like a large shortfall is often the test design, not the storage. Precondition the volumes, raise concurrency, and tune the network before concluding the storage is slow.

Should I use ramp_time or a warm-up run in fio?

Use an explicit warm-up run, not ramp_time alone. Ramp time only discards the first few seconds of measurements; it does not guarantee the entire dataset has been allocated on a thin volume. Run a full sequential write across the dataset first (for example --rw=write --bs=1M over the whole file), then point the measured test at that same file so every extent is already allocated.

What fio queue depth and job count should I use for distributed storage?

For capacity testing, keep numjobs around 8 and set iodepth to at least 32, spread across at least 6 volumes per storage node on separate NVMe-oF subsystems. A single job at a shallow queue depth measures latency, not throughput, and will never fill a 100G link regardless of how many pods you launch. Use shallow depth and a single job only when you are specifically measuring latency.

Why does MTU 9000 matter so much for NVMe/TCP?

At a 1500-byte MTU, a 100G link forces the CPU to process roughly six times as many packets per gigabyte as it does at 9000 bytes. On fast links the per-packet CPU cost, not raw bandwidth, becomes the limiter. Jumbo frames cut packet rate dramatically and free CPU for the storage engine. Every NIC and switch in the storage path must agree on MTU 9000 for it to take effect.

Does a 2x100G bond give me 200G of storage bandwidth?

Usually not. A bond in balanced-SLB mode behaves like active/standby for a single host-level flow, so storage traffic effectively uses one link. To use both 100G ports simultaneously, stop bonding the storage path and instead configure two VLANs on separate IPs with NVMe-oF multipathing in active/active mode. That also isolates storage from compute traffic and lets you manage storage bandwidth independently.

Which protection scheme should I benchmark on a 5-node cluster?

Start with 1+1 for the best raw IOPS (especially 4K random) or 2+1 for much better capacity efficiency with comparable real-world performance, since pure 4K random is rare in production. Choose 2+2 only when you need to tolerate two simultaneous node failures; it costs about 30% on write performance and is best run on 6 or more nodes so it can rebalance between failures rather than during them.

Why does simplyblock use erasure coding instead of 3x replication?

Three-way replication, the common default in distributed storage, stores every block three times: 3.0× write amplification and 3× the raw capacity for the data you actually use. Simplyblock instead protects data with distributed erasure coding. A 2+2 erasure-coded scheme tolerates two concurrent node failures at 2.0× overhead, the same durability three-way replication gives at 3.0×, and a 2+1 scheme tolerates a single failure at just 1.5×. On a network-bound NVMe/TCP cluster, where write traffic is what saturates the link first, that lower amplification means less data on the wire per useful write and therefore higher effective throughput, plus far more usable capacity per drive. The one exception is pure small random writes (such as 4K random), where a plain mirror avoids the parity read-modify-write and wins on latency.