We Break Our Storage So You Never Have To

Here’s a confession most storage vendors won’t make: testing storage is genuinely hard, and most teams don’t do nearly enough of it. Not because they don’t care. The bugs that matter most are just genuinely hard to reproduce.

Those bugs don’t live in deterministic logic paths. They live in the timing between a network partition and a write acknowledgement. They appear when an SPDK process restarts mid-write and a replica must decide what to accept. They show up when three things fail in a specific sequence and the cluster must determine which writes were committed and which were not.

The only way to find them is to run the failures on real hardware, under real concurrent I/O load, repeatedly, and check whether any data disappeared. Not as a testing philosophy. As the only viable option.

That’s exactly what simplyblock does, automatically, before every release ships. Not in a demo environment. Not on idle clusters. On real NVMe hardware, with real FIO workloads running, with real failures injected at the worst possible moments.

Why Storage Is Harder to Test Than Almost Anything Else

Testing a web application is straightforward. Write unit tests, add integration tests against a real or mocked database, spin up a CI environment, call it tested. The bugs that escape this process are generally embarrassing but rarely catastrophic.

Storage failure bugs are environmental by nature. You can’t mock an NVMe device failing mid-write in a way that exercises the actual recovery path. A unit test for a failover handler doesn’t test whether the host-side NVMe driver correctly reconnects to the secondary path when the primary disappears. The only test that covers that path is a live cluster, real hardware, a real failure injected at the right moment.

The timing problem compounds this. Many storage bugs aren’t bugs in any single component but in the interaction between components at specific moments. A snapshot operation running concurrently with a node going offline creates a situation no single component owns. Finding that bug requires thousands of failure injections across many hours, because the vulnerable window may be narrow and occur only once every few hundred cycles.

The stakes differ by domain. The same storage failure has different consequences depending on what’s stored and what happens when it can’t be accessed:

Table 1: Domain-specific failure modes and corresponding test coverage in the simplyblock E2E suite.

How Simplyblock Runs on Kubernetes

There’s a meaningful difference between software that “supports Kubernetes” and software that’s designed as a Kubernetes-native system. That distinction matters operationally, and it matters for how effectively a storage system can be tested.

Simplyblock is the latter. The Kubernetes deployment uses six Custom Resource Definitions managed by an operator: SimplyblocStorageCluster for cluster lifecycle, SimplyblockPool for storage pools with QoS and RAID configuration, SimplyblockLvol for individual volumes with encryption and host access control, SimplyblockStorageNodes for node enrollment and device management, SimplyblockDevices for physical NVMe device operations, and SimplyblockSnapshotReplication for cross-cluster snapshot replication. Your application uses standard PersistentVolumeClaims and VolumeSnapshots. Platform teams can manage storage through the same GitOps pipelines they use for everything else.

This architecture directly shapes how we test. Our Kubernetes test track (K8sNativeFailoverTest) uses Kubernetes APIs exclusively for all data-plane operations: volumes created via PVC manifests, snapshots via VolumeSnapshot objects, FIO workloads as Kubernetes Jobs with ConfigMaps. The only time we drop to sbcli is for outage injection and diagnostics. This means we’re exercising the actual path your application takes, including the operator reconciliation loop and CSI driver, not a test-only shortcut around them. If there’s a bug in how the operator reconciles a failed node with pending PVCs, this test will find it.

The test suite runs against k3s (on-premise), Red Hat® OpenShift® OCP (with OVN-Kubernetes CNI and stricter RBAC policies), GCP, and AWS on EC2 ARM64 instances. Platform differences are real: a bug in the OpenShift networking configuration wouldn’t surface on k3s. Running on four platforms means a bug must survive four independent checks before it can reach your cluster.

Storage reliability on your specific Kubernetes platform is a testable claim, not an assumption. If you want to understand how simplyblock’s testing maps to your workload and platform requirements, the engineering team can walk through it. Talk to a storage architect

Four Failure Modes Injected Under Live Load

All failure types are injected while 20 storage volumes run FIO read/write workloads continuously. FIO never pauses during a failure injection. Any I/O error immediately fails the test.

container_stop

The SPDK process inside a storage node is killed abruptly. On Docker, docker stop. On Kubernetes, kubectl delete pod on the snode-spdk pod, triggering an automatic DaemonSet restart. This simulates the most common real-world failure class: process crashes from memory pressure, kernel bugs, or OOM kills.

The test validates that connected NVMe clients detect the failure, switch to the secondary path, and continue I/O without errors while the primary node restarts. If there’s one failure scenario you need confidence about before running production databases on distributed storage, this is it.

graceful_shutdown

The storage node shuts down in an ordered sequence, equivalent to a planned maintenance window or rolling upgrade. The cluster must migrate all LVOLs from the departing node to secondary nodes before the node goes offline. Clients see zero I/O interruption.

For healthcare and fintech environments where maintenance windows are expensive or simply not available, this test carries more weight than any marketing claim about zero-downtime operations. The answer is either yes, proven by the test, or it’s a claim.

interface_partial_NW

One of the storage node’s network interfaces is blocked with iptables while others remain available. The node is reachable through some paths but not others. This is arguably the most dangerous failure mode in distributed systems: the cluster must make correct decisions about what’s committed and what’s not, under genuine ambiguity.

Partial network partitions are where split-brain scenarios, stale leaders, and partial write acknowledgements live. The bugs this test surfaces are the most dangerous category precisely because they’re the hardest to reason about before they occur.

interface_full_NW

All network traffic to and from the storage node is blocked. Complete isolation. The test validates that the cluster detects the isolation, marks the node offline, migrates all affected LVOLs to surviving nodes, and that FIO continues throughout. After recovery, data integrity is verified by checksum. Any write that was acknowledged before the isolation and is missing after recovery fails the test immediately.

The 100-Hour Stress Loop

Why 100 hours? Because that’s the threshold where timing-dependent bugs reliably appear.

Scenario tests catch most bugs. They don’t catch timing-dependent bugs that only appear after the 47th failover, or when a specific failure type hits during a specific phase of a migration task, or when three things fail in a particular order at a particular moment in the write cycle.

The stress test runs a continuous failure loop: pick a random failure type, inject it, verify FIO shows zero errors, wait for recovery, validate migration completion, verify checksums, repeat. It runs until something breaks or the 72-hour GitHub Actions timeout expires. Both the Docker and Kubernetes tracks must complete 100+ continuous hours before a release ships.

We’ve found bugs at hour 62 that never appeared in the first 20. We’ve also caught bugs that only trigger when a specific outage type hits during a snapshot operation, which occurs roughly once every few hundred cycles. 100 hours provides enough cycles to surface that category of bug before it reaches a production cluster.

Also running in parallel: a separate LVOL continuous test runs create, snapshot, clone, resize, and delete operations for 20+ hours on both Docker and Kubernetes. This catches a different class of bugs: resource leaks, capacity accounting errors, and orphaned snapshots after failed clone operations. These are lifecycle bugs that accumulate over time and are invisible in short test runs.

The erasure coding striping configuration also factors into the test matrix. Simplyblock uses NDCS (data chunks) and NPCS (parity chunks) parameters to define fault tolerance and performance profiles. All four standard combinations must pass the full E2E suite:

Table 2: All four NDCS/NPCS striping configurations are required to pass the full E2E suite before each release.

Five Gates Before Every Release Ships

Every release passes five sequential quality gates. Each gate is a hard blocker: failure resets the clock.

Gate 1: Clean E2E pipeline (~2 hours) The full functional test suite on Docker and Kubernetes. Covers the complete API surface: LVOL create/connect/resize/migrate, QoS enforcement, snapshot and clone lifecycle, DHCHAP authentication, encryption, host access control, S3 backup/restore, and HA variants. Runs automatically on every push to main and release branches.

Gates 2 and 3: 100-hour stress runs (Docker and Kubernetes, run in parallel) Gate 2 runs RandomMultiClientMultiFailoverTest on Docker. Gate 3 runs K8sNativeFailoverTest on Kubernetes. Same pass criteria: 100+ continuous hours with zero data errors. Both run on dedicated lab hardware isolated from each other and from the E2E pipeline.

Gate 4: LVOL continuous operations (20+ hours) Non-stop LVOL, clone, and snapshot operations on both Docker and Kubernetes. Lifecycle durability testing for resource leaks, capacity accounting drift, and edge cases in the deletion path when clones still exist.

Gate 5: Configuration matrix and upgrade Full E2E across all four NDCS/NPCS configurations. Major version upgrade with FIO running on an active cluster. Standalone backup and security test suite passes. This gate covers the full range of supported configurations, not just the default.

Total automated test time before every release: approximately 220+ hours on real NVMe hardware.

Test Coverage Map

The table below shows what the current suite covers, what’s partial, and what’s not yet included. We publish the gaps rather than have you discover them during an incident.

Table 3: Test coverage by scenario and track. “Full” means both tracks pass across all supported platforms and NDCS/NPCS configurations.

The complete test suite is publicly available at github.com/simplyblock/sbcli/tree/main/e2e.

We don’t claim our system never fails. Distributed systems fail. What we claim, and what we test, is that when hardware fails, your data is safe and your application keeps running. Getting to that confidence takes 220+ hours of automated chaos engineering before every release.

What you get when you deploy simplyblock:

• Every write acknowledged before a failure is present and correct after recovery, verified by checksum
• Zero I/O errors during planned maintenance and rolling upgrades
• Storage that scales independently of compute, with no vendor lock-in
• A system tested against your specific platform: k3s, OpenShift, GCP, AWS ARM64
• Full test suite source code, publicly available

Questions and Answers

How long does simplyblock’s complete test suite take to run before each release?

Approximately 220+ hours. The E2E functional suite completes in about 2 hours. The Docker and Kubernetes stress runs (100+ hours each) run in parallel. The 20-hour LVOL continuous test runs alongside. The configuration matrix and upgrade tests in Gate 5 add the remaining time. All runs execute on dedicated lab hardware isolated from engineering workstations and from each other.

What is the difference between the Docker and Kubernetes test tracks?

Both tracks run the same test scenarios with the same pass criteria, but the infrastructure layer is completely independent. The Docker track uses SSH-based operations and sbcli. The Kubernetes track uses kubectl exec and Kubernetes-native APIs: PVC manifests, VolumeSnapshot objects, and FIO as Kubernetes Jobs. A bug that appears only on Kubernetes due to operator reconciliation behavior or CSI driver state wouldn’t be caught by Docker testing alone.

What does “zero data errors” mean in the stress test?

Every write that’s acknowledged before a failure injection must be present and correct after recovery. The stress test verifies this with checksums after every failure cycle. A single missing or corrupted byte fails the entire run. Zero data errors is a binary pass/fail, not a percentage threshold or statistical measure.

Does running 220+ hours of tests slow down the release cadence?

It does. The alternative is shipping timing-dependent bugs that only surface in production after many hours of operation under real failure conditions. We’ve found bugs at hour 62 that didn’t appear in the first 20. Those bugs would otherwise have been found by your cluster. The release cadence reflects that tradeoff directly.

Does simplyblock test specifically against OpenShift?

Yes. The Kubernetes test track runs against OpenShift OCP with the platform-specific configuration it requires: OVN-Kubernetes CNI, the routingViaHost patch, and OpenShift’s stricter RBAC policies. A test run that passes on k3s but fails on OpenShift catches bugs that would otherwise only appear in regulated enterprise deployments, which is exactly the context where those bugs matter most.