Architecture of a deterministic edge rollout control plane
A rollout controller cannot treat command delivery as proof of installation. Edge devices disappear, reconnect, repeat reports, and observe desired state in an order the server did not choose. The architecture therefore starts with a stricter constraint: a stage may advance only when stored observations prove that its assigned devices reached the expected generation and artifact digest within the policy's health bounds.
That constraint shapes every boundary in the Edge Fleet Rollout Safety Control Plane. Cohort membership is frozen before execution. Devices pull signed desired state instead of accepting inbound control connections. Gate evaluation reads observations and health samples, not dispatch acknowledgements. Evidence is append-only and hash-chained. External systems may add telemetry or request follow-up work, but none of them can mutate release truth.
System Topology
The operator console and device protocol meet in one control-plane service. That choice keeps release transitions, evidence writes, and local operator notices inside one transaction boundary. Background jobs handle scheduling, stage progression, delivery attempts, simulation, replay, benchmarks, exports, and adapter outbox work. They share the same storage contract as synchronous API actions.
The edge boundary is pull-based. A device receives a one-time provisioning secret, authenticates subsequent polls, and asks for its desired state. It then reports sequence, observed generation, observed artifact digest, health, and installation outcome. This direction fits intermittently connected fleets because the server does not need routable inbound access to every device.
The release path
The default policy divides a release into fixed cumulative stages of 1, 5, 20, 50, and 100 percent. The planner filters eligible devices, preserves representation across hardware and processor groups, and applies a keyed deterministic ranking. The same frozen inputs produce the same membership without storing or redrawing a random sample.
The plan is frozen as canonical bytes and a membership digest. Later inventory changes do not rewrite an active release. A 10,000-device cohort remained byte-identical across 100 shuffled input orders, which confirms that database row order does not leak into the plan.
Each stage has an observation window. Workers issue commands, but promotion waits for device reports that match both the desired generation and artifact digest. The gate evaluator applies a fixed precedence: rollback, pause, insufficient evidence, then pass. Offline fraction, telemetry freshness, convergence, and policy health thresholds are part of the decision. An acknowledgement can prove receipt; it cannot prove convergence.
Infrastructure Decisions
- Compute: One C++23 service owns synchronous API work and scheduled release jobs. I chose this over separately deployed API and worker services because release transitions, local notices, and evidence writes need one visible transaction boundary in this specification. Leases and outbox rows leave a clear split point if independent workers become necessary.
- Data layer: SQLite WAL and PostgreSQL implement one storage contract. I chose this over requiring PostgreSQL for every run because the complete release journey must work without Docker, while the production profile still needs tenant-scoped locking and a real PostgreSQL contract.
- Device boundary: Signed outbound polling carries desired state and observations. I chose this over server-initiated device connections because private networks and intermittent connectivity make inbound reachability an unsafe fleet assumption.
- Analytics: DuckDB reads frozen benchmark outputs. I chose this over analytical queries against operational release tables because benchmark work must not contend with control-plane writes or redefine release truth.
- External delivery: Durable outbox records isolate optional adapters. I chose this over synchronous telemetry, notification, or workflow calls because an adapter outage may delay a side effect but cannot take pause, abort, or rollback away from the operator.
Decision Log
| Decision | Alternative Rejected | Why |
|---|---|---|
| Freeze cohort membership before execution | Recalculate membership from live inventory at each stage | Approval, replay, and audit need one stable population even when devices join, leave, or change state during a release. |
| Promote on observed convergence | Promote on command delivery or acknowledgement | Receipt does not prove that the expected generation and artifact are running within the current health window. |
| Keep fixed cumulative stages | Resize stages during the active release | Fixed boundaries keep simulation, approval, gate evidence, and replay comparable after exposure begins. |
| Store canonical append-only evidence | Maintain mutable audit rows | Sequence links and event hashes expose deletion, reordering, and field changes that an editable log could hide. |
| Normalize values inside each storage adapter | Make the shared JSON serializer reinterpret typed strings | The adapter knows how its database renders typed values; the serializer cannot safely guess which strings are timestamps. |
| Recheck rollback artifact integrity at execution | Trust validation performed when the release was drafted | A later key compromise or missing artifact must block rollback before another command is issued. |
Storage and integrity boundaries
The storage interface exposes tenant, fleet, device, policy, release, evidence, query, execution, and transaction operations. SQLite enables foreign keys and WAL, uses an immediate transaction for evidence appends, and serializes access through a recursive mutex. PostgreSQL uses the same domain contract and takes a tenant-specific advisory transaction lock before assigning the next evidence sequence.
Evidence hashes cover the event identifier, sequence, tenant, aggregate, event type, actor, payload, occurrence time, trace identifier, and previous hash. JSON object keys are sorted before serialization. This byte-level contract matters across both database engines: PostgreSQL trims trailing fractional zeroes in TIMESTAMPTZ, so its adapter normalizes the timestamp before hashing and insertion. Hashing a representation that the database later rewrites would make a newly stored event fail verification.
Artifacts and rollback targets have separate integrity checks. A rollback command is not issued until the stored artifact and key material satisfy the required digest and signing constraints. Evidence exports are immutable outputs, while traces and benchmark corpora remain bounded files outside the transactional database.
Failure containment
The control plane owns local pause, abort, and rollback decisions. The IoT adapter can supply fresh health evidence when a policy requires it, but missing required evidence blocks promotion. The Notification Hub receives frozen release events. The Workflow Engine receives non-authoritative playbook requests. Adapter responses are classified, retried through outbox records, and cannot edit a release.
This boundary was tested against a universal 20 percent install-failure scenario. The frozen acceptance rule allowed at most 2.5 percent unhealthy exposure for the staged strategy, compared with 100 percent exposure for the all-at-once baseline. The recorded benchmark result was 0.505 percent. That number describes a deterministic corpus, not a customer production claim.
Deployment shape and limits
Standalone mode runs the C++23 service with SQLite and local filesystem paths. It supports the complete 100-device, five-stage journey without Docker. Production mode packages the same service with PostgreSQL 16 and a reverse-proxy boundary. Configuration chooses the storage backend, worker timing, body and artifact limits, HTTP adapter capacity, device-count bounds, and whether each external adapter is enabled.
The single-process shape is deliberate for this specification project. It keeps the release state machine and evidence transaction visible in one codebase, while leases and outbox records mark seams that can later move to separate workers. The evidence covers deterministic planning at the tested ceiling, a 100-device end-to-end journey, and the bounded reference workload. It does not establish a multi-region control plane or an unlimited fleet size.
Verification architecture
The evidence ladder mirrors the boundaries. Domain and component tests cover cohort determinism, gate precedence, replay, storage isolation, immutable evidence, and adapter behavior. Docker-free CTest completed 63 of 63 tests. The production image completed 64 of 64 tests, with two documented environment-dependent skips in that image run. Six Playwright journeys covered the operator console in desktop Chromium and an iPhone 13 profile.
The strongest journey registered 100 devices, created signed artifacts and policy, ran all five stages, issued 100 install commands, recorded five gate evaluations, converged every device, and finished with a valid evidence chain. The implementation ledger closed at 323 of 323 items. Those checks do not turn a public, undeployed specification into a live service. They show that its safety claims are tied to repeatable code paths and recorded test evidence.