Raunak Jalan 
The problem with storage reliability claims is that they’re easy to make and difficult to verify. A vendor can publish a “99.999% availability SLA” by defining availability in ways that exclude the failure modes you actually care about. They can run customer success stories about uptime without describing the test conditions under which that uptime was verified.
For platform engineers who own the stateful layer in production, this matters. Storage failures have consequences that web application failures generally don’t: if your database storage loses an acknowledged write during a node failure, you’re not debugging a 500 error, you’re investigating data corruption. The asymmetry between a misleading reliability claim and the incident it enables is significant.
This post is a practical framework for evaluating distributed storage reliability: specific questions to ask vendors, evidence to request, and signals that indicate a vendor is conflating marketing with actual test coverage.
The Core Problem: Verifiable Claims vs. Asserted Claims
There are two types of reliability claims. Verifiable claims are backed by reproducible tests on real hardware with documented failure conditions, workload parameters, and pass criteria. Asserted claims are statements about availability or durability without corresponding test methodology.
Most storage vendor documentation consists almost entirely of asserted claims. “Our system maintains data integrity during node failures” is an assertion. “We inject four failure modes on real NVMe hardware under live FIO workload, verify checksums after every recovery cycle, and require 100+ continuous hours without a data error before shipping a release” is a verifiable claim.
The distinction matters because distributed storage bugs are not found by assertion. They’re found by running failures repeatedly, at the right moments, under concurrent I/O load. A vendor who hasn’t done this doesn’t know whether their system has these bugs. They simply don’t know, and their reliability claims reflect intent rather than test results.
Five Questions That Reveal Real Reliability
1. What failure modes do you test, and exactly how are they injected?
This question surfaces whether the vendor tests realistic failure scenarios or sanitized versions. Useful follow-ups:
- Do you test SPDK or storage process crashes, not just clean shutdowns?
- Do you test partial network partitions (some interfaces blocked, others reachable) or only complete node isolation?
- When a failure is injected, is I/O running concurrently, or is the cluster idle?
Testing only graceful shutdowns when real failures are OOM kills or process crashes misses the failure modes that cause data loss. Partial network partitions are particularly important because they’re the scenario where split-brain can occur. A vendor who hasn’t tested this doesn’t know whether their consensus mechanism handles it correctly.
2. What hardware does the test run on?
Virtualized or cloud-based test environments are not equivalent to real NVMe hardware for storage reliability testing. NVMe behavior under failure, including how the host-side driver responds to a failed device, how reconnect timing interacts with in-flight writes, and what the latency profile looks like under partial path availability, differs materially between real hardware and emulated environments.
A vendor who runs all reliability tests on VMs or shared cloud instances has not validated the behavior their customers will observe on bare metal or dedicated NVMe nodes. The specific failure bugs that matter for production deployments live in the interaction between real NVMe drivers, real hardware, and real network conditions.
3. How long does the test run continuously?
This is the question that filters for timing-dependent bugs. Timing bugs are the most dangerous class of storage reliability issue: they don’t appear in short test runs, they appear at hour 47 when the 200th failure injection happens to land during a snapshot operation at exactly the wrong moment.
A vendor who runs a 10-minute chaos test before shipping has not found this class of bug. They haven’t run the test long enough to surface it. Useful calibration: a test run that doesn’t reach 50-100 continuous hours of failure injection provides limited confidence about timing-dependent bugs.
4. How is data integrity verified after recovery?
The pass criterion for a storage reliability test should be binary: every byte that was acknowledged before a failure is present and correct after recovery. Checksum verification is the only way to confirm this.
Vendors who describe their test pass criterion as “the cluster recovered without errors” or “all nodes came back online” are measuring infrastructure recovery, not data integrity. Recovery without data loss verification tells you the system recovered, not that the data it recovered was correct.
5. What gaps does your test coverage have, and how do you communicate them?
A vendor who claims complete coverage is telling you something important: they haven’t looked carefully for gaps, because complete coverage of distributed storage failure scenarios is not possible with any finite test suite. A vendor who can describe specific, named gaps in their current coverage is telling you something different: they’ve mapped their test coverage honestly and can give you an accurate picture of what’s validated and what isn’t.
This question also surfaces whether the vendor separates test environment coverage from production coverage. A system tested only on one Kubernetes distribution may behave differently on another. Platform coverage matters.
Red Flags in Vendor Responses
“We have 99.99x% uptime SLAs.” SLAs describe financial liability, not test methodology. An SLA that excludes data loss from force majeure, hardware failure, or customer configuration tells you about the contract, not about the reliability of the storage system under those conditions.
“Our architecture uses X replication factor, which provides Y durability.” Architectural claims describe what the system is designed to do, not what it has been tested to do. A 3-replica architecture provides durability only if the failure handling logic is correct, and that logic can only be verified by testing actual failure scenarios.
Case studies about uptime without description of the workload. “Customer X ran 18 months without downtime” is not a reliability claim. It’s a deployment history. It says nothing about whether data integrity has been verified under failure conditions.
Chaos testing described in marketing terms only. Any vendor can describe chaos testing in a blog post. The question is what specific failures were injected, on what hardware, with what running workload, for how long, and with what pass criteria. Absence of those specifics indicates the testing was either not done or not done rigorously.
What Good Evidence Looks Like
Strong evidence for storage reliability includes:
- Published test methodology with specific failure types, workload parameters, and hardware
- Continuous test runtime measured in hours (50+), not minutes
- Checksum-based data integrity verification as the primary pass criterion
- Platform coverage across multiple Kubernetes distributions, not just one
- Published test coverage gaps (this indicates honesty, not weakness)
- Publicly available test suite source code, so the methodology can be independently examined
simplyblock publishes all of this. The complete E2E test suite is available at github.com/simplyblock/sbcli/tree/main/e2e. The chaos engineering process is documented in detail, including the exact failure types, hardware, workloads, pass criteria, and current coverage gaps.
The framework above applies to any storage vendor. Apply it to simplyblock too.
Want to walk through how simplyblock’s testing maps to your specific requirements? The engineering team can run through the test methodology and coverage gaps for your use case directly. Talk to a storage architect
Questions and Answers
Is a published SLA a reliable indicator of storage reliability?
No. SLAs define financial liability, not test coverage. A 99.999% SLA describes the compensation model if uptime falls below that threshold. It does not describe what tests the vendor ran, what failure modes were exercised, or whether acknowledged writes survive hardware failures. The SLA and the underlying reliability claim are separate things.
Should I run my own chaos tests before selecting a storage system?
Yes, where possible. A proof-of-concept that includes actual failure injection, not just functional testing, is the most reliable way to evaluate storage behavior under your specific workload and platform. A vendor who is confident in their test results will support this. A vendor who wants to limit your evaluation to a demo environment may be limiting it for a reason.
How can I verify that a vendor’s test methodology is rigorous without running the tests myself?
Ask for the test source code. A vendor with rigorous test coverage will have a real test suite with readable code, not a marketing description of testing. If the test suite is public, you can read what it actually does. If it’s private but shareable under NDA, you can review it. If neither is offered, treat the reliability claims with appropriate skepticism.
What is the minimum continuous test duration that provides real confidence?
The honest answer is that it depends on the failure injection rate and workload complexity. simplyblock requires 100+ continuous hours because experience showed bugs at hour 62 that didn’t appear in the first 20. A rough heuristic: if a vendor’s longest continuous test is under 24 hours, they’ve likely not found their timing-dependent bugs yet.
