Rob Pankow 
Modernizing an edge fleet means settling one small site design and repeating it many times. The design has to keep a store or a warehouse running when a server fails or the WAN link drops, on two servers in a wiring closet, managed by a team that is never on site. Decide that design, and what it needs centrally, before the pilot starts.
The guide is for infrastructure leads who run many small sites on two-host VMware® clusters and have to decide what replaces them: retail stores, warehouses and distribution centers, plants, and branch offices. It covers what stays on site, the choice between one server and two, how a two-server site stays available without a storage witness at the site, and what the pilot has to prove. Simplyblock on Red Hat® OpenShift® is the worked example, based on its edge cluster documentation.
What has to stay on site
Keep on site what the site cannot run without and what cannot wait for the WAN. In a warehouse, that is conveyor control, the warehouse management system, and the palletizers. In a store, it is the point of sale, video, and inventory. These systems talk to equipment on the floor, so they need short response times, and they have to keep working through a WAN outage.
Everything else can run in a central data center or a public cloud. The sites should carry only what has to be local.
Three facts shape the site platform. Most of these applications come from vendors and run in VMs today, while newer ones arrive as containers, so the same servers have to run both for years. The space is often half a rack in a wiring closet, with no room for a separate storage system. And nobody on site manages infrastructure, so every routine task has to work remotely.
A simplyblock edge cluster fits that shape. It consists of one or two storage nodes on the Kubernetes cluster of the site, and it serves volumes from the local NVMe devices or other Linux block devices in those servers. The site runs no storage control plane of its own. Edge storage for remote sites describes the model, and data storage for warehouse and logistics IT applies it to distribution centers.
One server or two: what each site design survives
The number of servers decides which failures a site rides through. Two is the default for a site that cannot stop, and one is a valid choice where a pause is acceptable.
| Site design | Survives | Does not survive | Choose it when |
|---|---|---|---|
| One server | A drive failure, with an erasure coding scheme that has parity, and a WAN outage | The loss of the server: its volumes are unavailable until it is back | The site tolerates downtime, and backups or application-level replication cover the loss of the server |
| Two servers | A drive failure, the loss of either server, and a WAN outage | A server failure while the hub is unreachable. Without host fencing, a broken link between the two servers fences both | The site has to keep running through a server failure |
One exception applies at a one-server site: with a dedicated journal device, which Linux block devices always use, the journal is not mirrored, and a failure of that device stops the node, whatever the scheme.
A two-server site uses the 1+1 scheme: each server holds a copy of the data and of the journal. Usable capacity is therefore half of the raw capacity, before reserves. Every volume has a primary on one server and a secondary on the other, and a workload connects to both over two NVMe-oF paths.
Both servers run workloads. Size them so that one server can carry the whole site, because that is the state after a failure.
How a two-server site stays available without a storage witness at the site
Two nodes alone cannot tell a failed peer from a broken link between them. If both kept writing, their data would diverge, so a third vote has to decide. Many two-node designs place that vote in a witness that somebody has to host and maintain for every site.
In a simplyblock edge cluster, the third vote is the central control plane on a hub cluster, which also manages every site. No storage witness is installed at the site. When a server fails, the surviving node holds its writes for a moment and reports to the hub, and the hub grants it solo operation. With the hub reachable, I/O pauses for about 0.5 to 2 seconds and then continues on the surviving server.
Volume I/O stays at the site throughout. Volume data never crosses the WAN, and both nodes keep serving I/O while the WAN link to the hub is down. Management operations, such as creating a volume, wait until the link is back.
The design has a limit that belongs on the table before the pilot. A failover needs a reachable hub. If a server fails while the hub is unreachable, the surviving node fences itself, and the volumes are unavailable until the hub is reachable again. The survivor cannot tell a failed peer from a broken link, so it stops instead of risking two writers.
Two rules follow from that limit. The hub has to run highly available, or in a third failure domain that shares no power, network, or location with the two edge nodes. At the site, the link between the two nodes and the uplink to the hub must not share a switch, a cable, or a network interface, so that one failure cannot cut both.
Host fencing completes the picture. With it, Kubernetes restarts the failed server’s VMs and pods on the surviving server within seconds. Host fencing is off by default and is turned on per cluster. It also decides what a broken link between the two nodes does: without host fencing the hub fences both nodes, and with it the hub keeps one node.
The OpenShift cluster on the two servers needs a tie-breaker for its own etcd as well: either fencing through the baseboard management controllers (BMCs) of the two servers, or an arbiter node, which is a third, small machine at the site. That is independent of the storage arbitration on the hub.
What the pilot has to prove
A pilot should prove the hard parts, not the installation. Four tests cover them.
- Failures. Power off one server. Cut the WAN link. Then cut the WAN link and power off a server, and record what the people at the site see. The third test shows the documented limit once, under your control.
- Sameness. Build the second site from the same declarations as the first. With simplyblock, clusters, nodes, pools, and operations such as restarts and upgrades are declared as custom resources and run by the operator, and the hub shows alerts, events, and capacity for every site. Fleets fail through drift: small differences between sites that surface during an outage.
- The migration window. Sites that run around the clock still have quiet hours. Decide whether a site converts on temporary hardware that is rolled in, or in place on its own two servers, and how long it runs without redundancy during the change. With simplyblock, plan the storage step from the documentation: a two-node edge cluster is created with both servers from the start, and a one-server cluster cannot be changed into a two-server cluster later.
- Vendor support. Ask every application vendor to confirm in writing that they support their software on the new virtualization platform. A support ticket is the wrong place to find out.
Questions and Answers
Where should the hub run?
In a location that does not fail together with a site. The simplyblock documentation places the central control plane on a hub cluster in the data center, and it can be the same cluster that runs the simplyblock disaster recovery hub. Two-server sites also need two-node arbitration support in the control plane, the operator, and the storage nodes.
What happens at a site while a storage node is in maintenance?
At a two-server site, the other node continues and takes over the volumes while one storage node is shut down or restarted. At a one-server site, the volumes are unavailable during the maintenance, so the workloads are stopped first.
How are the volumes of a site backed up?
Simplyblock backs up volumes to Amazon S3 or S3-compatible object storage as chains of incremental snapshots, with the schedule and the retention set by a policy. A backup is restored into a new claim. For a one-server site, backups, replicated snapshots, or application-level replication are the protection against the loss of the server.
