Chris Engelbert 
Storage vendors frequently claim live upgrade support. The claim is almost always true in the narrow sense: the upgrade process doesn’t require a full cluster shutdown. What the claim often obscures is the difference between an upgrade procedure that minimizes downtime and one that has been tested to produce zero I/O errors throughout, verified by running a concurrent workload during the entire upgrade process.
For teams in healthcare, fintech, and other environments where maintenance windows are expensive or contractually limited, the distinction is the entire question.
This post covers what live upgrade technically requires, what the graceful shutdown sequence looks like for a single storage node, what can go wrong at each step, and how to verify that a storage system’s live upgrade claim is backed by real test evidence.
Why Storage Upgrades Are Uniquely Risky
Upgrading a web application involves restarting containers with a new image. The application layer is stateless or nearly so; a brief restart produces a brief error for in-flight requests, and everything recovers. The risk is contained.
Storage upgrades are different. The storage layer holds persistent state, and the upgrade process has to move that state safely while applications continue writing to it. A rolling upgrade of a distributed storage system involves, for each node:
- Migrating all active volumes off the node
- Shutting down the storage process
- Upgrading the storage software
- Restarting the storage process with the new version
- Allowing the cluster to rebalance volumes back to the node
Each of these steps has failure modes. A migration that completes partially before the storage process stops leaves volumes without their expected redundancy for the duration of the upgrade. A version mismatch between upgraded and non-upgraded nodes can cause replication protocol failures if the upgrade changes the wire protocol. A restart that fails silently reduces cluster capacity without triggering alerts.
The Graceful Shutdown Sequence
simplyblock’s graceful shutdown sequence is the mechanism behind the graceful_shutdown test in the chaos engineering suite. The sequence for a single node upgrade looks like this:
Step 1: Suspend new LVOL placements on the node. The scheduler stops placing new volumes on the node being upgraded. Existing volumes remain in place for now.
Step 2: Migrate active LVOLs to secondary nodes. The cluster moves each LVOL’s primary replica to another node that already holds a secondary copy. This is a metadata update that changes which node is the Optimized path for each volume. Hosts connected via NVMe multi-path complete an ANA path transition: the host’s NVMe driver sees the path change and switches I/O to the newly Optimized path on the secondary node.
During this step, every application writing to any LVOL on this node goes through an NVMe path switch. The same mechanism as an unplanned failure, but controlled and announced in advance. Any I/O error during this migration is a bug.
Step 3: Verify all LVOLs have migrated. The cluster confirms no LVOLs remain with this node as their Optimized path before proceeding. This gate prevents proceeding with an upgrade while volumes are still being actively served from the node.
Step 4: Shut down the storage process. The SPDK process stops. No LVOLs are active on this node, so there are no in-flight writes to protect. The node goes offline cleanly.
Step 5: Upgrade the software and restart. The storage software is updated and restarted. On Kubernetes, this is a DaemonSet update: the new pod image is pulled and the pod restarts. The cluster registers the node as healthy once the SPDK target is back online and passes its health checks.
Step 6: Allow rebalancing. The scheduler begins placing new LVOLs on the upgraded node and migrating replicas back if the cluster prefers a balanced distribution. This step happens in the background and doesn’t affect ongoing I/O.
What Can Go Wrong
Migration stalls under high I/O load. If the cluster is under heavy write load during step 2, migration completion may take longer than expected. The migration algorithm must wait for write quorum to re-establish through the new primary before declaring the migration complete. Long migration windows increase the time applications spend in NVMe path-switch state, which increases P99 write latency during the window.
Protocol version mismatch during mixed-version operation. Between steps 4 and 5, the cluster is temporarily running one version on the upgraded node and the previous version on remaining nodes. If the upgrade changes any replication protocol or metadata format, mixed-version operation can cause replication failures. simplyblock manages this through strict compatibility constraints: protocol-breaking changes require a two-phase upgrade with an intermediate version that both old and new nodes understand.
Node doesn’t rejoin after restart. If the storage process fails to start on the upgraded version (incompatible kernel modules, missing dependencies, configuration format change), the node stays offline. The cluster continues operating with reduced capacity, but the upgrade is now blocked. Monitoring node health post-upgrade before proceeding to the next node is the safeguard against cascading capacity reduction.
Snapshot operations during migration. Snapshots that start during step 2 (while an LVOL is mid-migration) create a consistency challenge: the snapshot must capture state at a moment when the volume’s primary is changing. A storage system without explicit handling for this case can produce a snapshot that’s inconsistent across the migration boundary.
Monitoring blind spots. The upgrade process reduces redundancy for each node during its upgrade window. If two nodes are upgraded simultaneously, or if a hardware failure occurs while another node is mid-upgrade, the cluster may be in a degraded state without sufficient replicas. Sequencing upgrades one node at a time and confirming full replication before proceeding to the next node is the conservative path.
What the graceful_shutdown Test Actually Validates
The chaos engineering graceful_shutdown test runs the full sequence above while FIO operates on all 20 test volumes. The pass criterion is zero I/O errors throughout, including during every NVMe path switch in step 2.
This test is not run once before a release. It’s run as part of the 100-hour stress loop, where simplyblock’s continuous test alternates between the four failure types. graceful_shutdown is exercised hundreds of times per stress run. The reason is timing: a bug in the migration-while-snapshot scenario described above has a narrow timing window. It won’t appear in the first five graceful shutdowns. It may appear at the 200th, when a snapshot happens to start at exactly the wrong moment during migration.
The upgrade test in Gate 5 of the release pipeline goes further: it runs a major version upgrade on an active cluster with FIO running, not just a graceful shutdown of a single node. This is the test that validates protocol compatibility handling and the full upgrade sequence across all nodes, not just the single-node drain behavior.
Full test coverage by gate:
| Gate | Test | Pass Criterion |
|---|---|---|
| Gate 1 | Full E2E functional suite | All API operations pass, including snapshot and encryption |
| Gates 2–3 | 100+ hour Docker and Kubernetes stress loops | Zero data errors, zero I/O errors throughout |
| Gate 4 | 20+ hour LVOL continuous operations | No resource leaks, no capacity accounting errors |
| Gate 5 | Configuration matrix + live upgrade | All NDCS/NPCS configurations pass; FIO shows zero errors during full version upgrade |
Total test time before each release: approximately 220+ hours on dedicated NVMe lab hardware.
Verifying a Live Upgrade Claim
If a storage vendor claims live upgrade support, these questions distinguish a real claim from a marketing assertion:
- Is the upgrade tested with a running application workload, or is the cluster idle during the test?
- What is the pass criterion: “no downtime” (meaning the cluster stayed up) or “zero I/O errors” (meaning the application saw no errors)?
- How is the pass criterion verified: manual observation, automated test, or checksum-verified data integrity?
- Does the test cover protocol-version transitions between old and new node versions?
- How many times has the upgrade test been run, and on what Kubernetes platforms?
For healthcare environments running HIPAA-covered workloads and fintech systems where maintenance windows have business continuity implications, “we’ve done live upgrades” and “we run 100+ hours of chaos testing that includes graceful shutdown hundreds of times before each release” are not equivalent claims.
Running storage upgrades on a regulated workload? The simplyblock engineering team can walk through the upgrade sequence and test evidence for your specific platform and compliance requirements. Talk to a storage architect
Questions and Answers
How long does a single-node upgrade take with simplyblock?
Migration time in step 2 depends on the number of LVOLs on the node and the current I/O load. Under light load, migration completes within minutes. Under heavy write load, migration may take longer. The upgrade process doesn’t proceed to step 4 until all migrations are confirmed complete, so the total time scales with the actual migration workload. Scheduling upgrades during low-traffic periods reduces the migration window.
Does simplyblock support rolling upgrades without any maintenance window?
Yes. The graceful shutdown sequence migrates all active volumes before the node goes offline, and the NVMe multi-path mechanism handles the resulting path switches transparently. Applications don’t require a maintenance window for node-by-node rolling upgrades. The Gate 5 upgrade test verifies this before each release ships.
What happens if a node fails during an upgrade before migration completes?
If a node fails before completing step 3 (migration verification), the cluster handles it as a standard unplanned failure: the volumes that were migrating are reconstructed from their replicas on surviving nodes. The upgrade process pauses until the unplanned failure is resolved. This is the conservative path: it doesn’t proceed through the upgrade with a failed node.
Can we upgrade simplyblock without Kubernetes operator involvement?
No. The upgrade process on Kubernetes is driven by a DaemonSet update through the simplyblock operator. This is by design: the operator coordinates the migration sequence, health checks, and gate verification that make live upgrade safe. Manual node upgrades outside the operator’s control bypass those gates.
How does upgrade sequencing interact with erasure coding configurations?
Erasure coding configurations with parity (NDCS=2, NPCS=1 or NPCS=2) require a minimum number of available nodes to maintain fault tolerance. Upgrading one node at a time ensures the remaining nodes can maintain the required parity during the upgrade window. simplyblock’s release testing covers all four standard NDCS/NPCS configurations in Gate 5 to validate that upgrade sequencing preserves fault tolerance across all configurations.
