Edge Fleet Rollout Safety Control Plane
Edge Fleet Rollout Safety Control Plane is release operations software for platform and IoT teams that ship signed software to devices with unreliable connectivity. It replaces delivery-centred confidence with evidence-centred promotion: each wave advances only after assigned devices report the expected generation, artifact digest, and acceptable health. A frozen benchmark later tested that rule against a fleet-wide installation failure. Operators retain local pause, abort, and rollback controls even when telemetry, notification, or workflow adapters are unavailable. This is a shipped public specification, not a deployed customer service.
Situation
A command queue can say that an install message was delivered. It cannot say that the target device installed the right bytes, remained healthy, or reported during the current observation window. That gap grows across fleets where devices disconnect, retry old reports, or run different hardware and processor architectures.
This project began as a greenfield specification for that gap. The brief required tenant isolation, signed artifacts, deterministic cohorts, fixed rollout stages, health gates, two-person approval, simulation, replay, benchmarks, an operator console, a pull protocol for devices, and immutable evidence. It also required a Docker-free path for local evaluation and a PostgreSQL production contract.
The safety rule was plain: no stage promotion from command acknowledgement alone. Promotion needs observed generation and digest convergence, bounded offline exposure, fresh health evidence when policy requires it, and a completed observation window.
Cost of Doing Nothing
The measurable cost in this specification is exposure, not money. Measured customer savings remain €0 because the ecosystem registry marks the project as not deployed. That is an evidence boundary, not an estimate of incident cost. The source material supports blast-radius measurements from a deterministic corpus; it does not support a claim about customer revenue, labour savings, or avoided downtime.
Doing nothing also leaves an audit gap. Without frozen membership and append-only evidence, an operator cannot prove which devices belonged to a wave, which inputs a gate saw, or whether a later inventory change rewrote the apparent history of a release. A green delivery queue would still leave the operator unable to distinguish receipt from healthy installation.
What I Built
The system is a C++23 control plane with a tenant REST API, an operator console, pull-based device endpoints, background workers, deterministic simulation and replay, benchmark analysis, and evidence export. SQLite WAL supports standalone operation. PostgreSQL 16 implements the production storage contract. DuckDB handles benchmark analysis without becoming the source of release truth.
System Flow
The default cumulative stages are 1, 5, 20, 50, and 100 percent. A keyed deterministic planner preserves hardware and processor representation, then freezes the resulting membership and its digest. A 10,000-device plan remained byte-identical across 100 input shuffles.
Devices receive a one-time provisioning credential, then authenticate their own polls. Each report carries sequence and observation state. A worker issues commands, waits for the policy's observation end, gathers fresh health evidence, and calls a gate evaluator with fixed decision precedence. Rollback checks the rollback artifact and signing key again before issuing commands.
Data Model
Tenant identity appears in operational relationships, and the schema rejects cross-tenant device placement. Release membership, stages, assignments, commands, reports, health samples, gate evaluations, approvals, simulations, replays, benchmarks, evidence, notices, adapter configuration, outbox delivery, and job leases are persisted as separate facts.
Evidence events form an immutable per-tenant hash chain. PostgreSQL serializes sequence allocation with a tenant-scoped advisory transaction lock. SQLite uses an immediate transaction and WAL. Both storage adapters implement the same application-facing interface, while each adapter owns dialect-specific representation rules.
Architecture Layers
The domain layer decides cohort membership, gate outcomes, simulation events, replay divergence, and benchmark results. Application jobs decide when to run those operations and how to persist their consequences. Infrastructure adapters translate storage and HTTP details. External systems receive frozen facts through an outbox boundary and cannot alter a release.
Decision Log
| Decision | Alternative Rejected | Why |
|---|---|---|
| Let devices pull desired state | Require the service to connect to every device | Private networks, disconnections, and changing addresses make inbound reachability a poor safety dependency. |
| Freeze fixed release waves | Resize waves while a release is active | Operators must approve one stable population, and replay must evaluate the same exposure boundaries later. |
| Require observed convergence | Promote from delivery receipts | A delivered command does not prove that the expected software is installed or healthy. |
| Keep standalone and production storage profiles | Require the production database for local evaluation | SQLite keeps the complete journey portable; PostgreSQL receives separate contract tests for the heavier concurrency boundary. |
| Record append-only evidence | Keep an editable activity log | Operators need proof of ordering and mutation, not a history that later edits can make look cleaner. |
| Isolate optional systems behind an outbox | Call external systems inside release transitions | Telemetry, notification, or workflow downtime must not take local safety controls away. |
| Compare strategies against frozen scenarios | Use live operational data as the benchmark baseline | A fixed corpus makes safety claims repeatable and keeps analysis separate from release truth. |
One decision earned its place through failure. PostgreSQL trims trailing zeroes from fractional TIMESTAMPTZ values. The first contract run hashed the pre-storage timestamp spelling, then failed verification when the database returned the same instant with fewer characters. The adapter now normalizes PostgreSQL's UTC representation before hashing and insertion.
Another limit surfaced under load. The reference workload passed at its bounded concurrency profile, while a higher 64-client experiment exposed SQLite contention. The project records that as a single-machine storage limit instead of presenting SQLite as the production scaling answer.
Ecosystem Integration
When a policy requires outside health evidence, the control plane reads it from the IoT Sensor Data Aggregator and blocks promotion if the sample is stale or unavailable. Frozen release events can pass to the Event-Driven Notification Hub, while non-authoritative playbook requests can go to the Workflow Automation Engine. Local notices and release controls remain inside the control plane, so an external outage delays a side effect without changing release truth. All three integrations are feature-flagged, and the system runs standalone with no ecosystem dependencies.
Results
The implementation ledger closed at 323 of 323 items. Docker-free CTest completed 63 of 63 tests. The production image completed 64 of 64, with two documented environment-dependent skips in that image run. Six browser journeys passed across desktop Chromium and an iPhone 13 profile.
The strongest end-to-end journey registered 100 devices, created signed release inputs, completed all five stages, issued 100 install commands, recorded five gate evaluations, converged every device, and finished with a valid evidence chain. The deterministic planner produced byte-identical 10,000-device cohorts over 100 shuffled input orders. In the frozen universal 20 percent failure scenario, staged exposure remained below the 2.5 percent ceiling and measured 0.505 percent, while all-at-once exposure reached 100 percent.
These are source-backed specification results. There is no live URL and no production-customer outcome. The shipped artifact demonstrates a bounded release-safety model whose claims can be replayed, inspected, and challenged without confusing delivery with convergence.