2026-08-29 14:03:12.120000+00 and 2026-08-29 14:03:12.12+00 describe the same instant. SHA-256 does not care. It sees different bytes.
That distinction became a release-safety problem in the Edge Fleet Rollout Safety Control Plane. The service records every significant release action in an append-only evidence chain. Each event includes the hash of the previous event, and verification recomputes every hash from stored fields. The first PostgreSQL contract run appended an event and then immediately declared that same event invalid.
Nothing had been tampered with. PostgreSQL had only rendered the timestamp in its normal form.
Why a formatting detail reached the safety boundary
An edge rollout has several kinds of truth that can drift apart. The server may have issued an install command. A device may have acknowledged it. The device may still be running the old artifact. A health sample may belong to the previous observation window. The control plane handles those gaps by treating observations, gate evaluations, approvals, and control actions as evidence.
The evidence table is immutable. A tenant's first event starts with 64 zeroes as its previous hash. Each later event points to the hash before it. Verification reads events in sequence order, reconstructs the canonical JSON object, calculates its digest, and checks both links. A changed payload breaks the event hash. A removed or reordered event breaks the previous-hash link.
The event hash covers more than its payload. It includes the identifier, sequence number, tenant, aggregate type and identifier, event type, actor, occurrence time, trace identifier, and previous hash. That breadth makes the chain useful, but it also means every covered representation is part of the protocol.
The JSON serializer already sorts object keys recursively. I assumed that was the end of canonicalization. I was wrong about where canonicalization ended. Sorting keys gives stable JSON structure; it does not give a timestamp string a stable database representation.
The failing sequence
The PostgreSQL adapter generated a UTC timestamp with six fractional digits. It assembled the event object, serialized it, calculated the digest, and inserted the timestamp and digest in one row. PostgreSQL accepted the TIMESTAMPTZ value, then returned a shorter fractional component when the row was read for verification.
The append path had hashed this value:
2026-08-29 14:03:12.120000+00
The verification path reconstructed the event with this value:
2026-08-29 14:03:12.12+00
Both strings represented one instant. They produced different canonical JSON bytes, so the newly appended event failed at sequence one. The hash chain was doing its job. The input contract was inconsistent.
The discrepancy also threatened every later event. Once one stored digest differed from its recomputed value, the next event would inherit a previous hash that verification could no longer trust. A lexical mismatch at the head of the chain contaminated the meaning of the tail.
This was not an abstract cross-platform concern. The PostgreSQL storage checkpoint recorded the defect during the first contract run against an isolated cluster. The test exercised migration reruns, tenant isolation, evidence append and verification, and database triggers that reject updates and deletes. The append succeeded. The following verification exposed the mismatch.
The production normalizer
The fix lives in src/infrastructure/postgres_storage.cpp. This is the exact function used by the PostgreSQL adapter:
std::string canonicalPostgresTimestamp(std::string timestamp) {
const auto fractional = timestamp.find('.');
auto timezone = timestamp.find('+', fractional == std::string::npos ? 0 : fractional);
if (fractional == std::string::npos || timezone == std::string::npos) return timestamp;
while (timezone > fractional + 1 && timestamp[timezone - 1] == '0') {
timestamp.erase(timezone - 1, 1);
--timezone;
}
if (timezone == fractional + 1) timestamp.erase(fractional, 1);
return timestamp;
}
The append path first converts the service timestamp from an ISO T and trailing Z to PostgreSQL's UTC spelling with a space and +00. The function then removes trailing zeroes from the fractional part. If every fractional digit disappears, it also removes the decimal point. The adapter hashes that normalized value and inserts that same value.
The important property is not the loop. It is the timing. Normalization happens before the event object is serialized and before the row is inserted. The bytes selected for hashing already match the representation that verification will read.
An alternative would insert the row, query it back, hash the returned fields, and update the digest. That path conflicts with the table's immutability rule and makes the append transaction more elaborate. Another option would exclude occurrence time from the hash, which would weaken the event envelope for the sake of a formatting bug. The implemented boundary keeps time covered and makes the storage adapter responsible for its dialect's spelling.
Canonical JSON was necessary but insufficient
The shared canonical JSON code recursively sorts object keys and emits strict compact JSON. That removes variation from map iteration order. It also makes replay digests stable when the same logical object arrives with keys in a different order.
String values remain strings. The serializer cannot know whether a value contains a timestamp, a device key, an artifact digest, or ordinary text. Teaching a generic JSON layer to reinterpret selected strings would hide a database rule inside a shared primitive. The PostgreSQL adapter knows the column type and the database's output form, so it owns this normalization.
SQLite shows why the boundary belongs there. Its adapter stores a six-digit UTC string ending in Z and reads that text back unchanged. PostgreSQL stores a typed timestamp and may shorten the fractional part. Both adapters satisfy the same evidence operation, but they need different lexical preparation to preserve the operation's invariant.
The shared invariant is precise: the event bytes hashed during append must equal the event bytes reconstructed from persisted fields. It is not “all databases must print time the same way.”
Concurrency is part of the same contract
Canonical bytes would still be useless if two workers could claim the same next sequence. The PostgreSQL append transaction takes an advisory transaction lock derived from the tenant identifier before reading the latest event. It then chooses the next sequence, uses the previous event hash, inserts the new event, writes any local operator notice, and commits.
SQLite reaches the same boundary with BEGIN IMMEDIATE and process-level serialization. The mechanisms differ because the deployment shapes differ. SQLite supports the local, Docker-free mode. PostgreSQL supports the production storage contract. In both cases, sequence allocation and evidence insertion share a transaction.
This pairing matters. A hash chain has two dimensions of determinism: stable bytes inside an event and stable order between events. Timestamp normalization fixes the first. Tenant-scoped serialization protects the second.
The test that became more valuable after it failed
The PostgreSQL component test does not inspect the normalizer directly. It asks the storage contract for an event, then asks the same contract to verify the tenant's chain. It also attempts to update the stored event and expects the database trigger to reject the mutation. The final verification remains valid after that rejected attempt.
That test caught a defect an isolated unit test for SHA-256 would never see. The digest function produced the correct digest for the bytes it received. The JSON serializer produced stable output. PostgreSQL stored the correct instant. The defect existed only where those correct components met.
The broader evidence ladder kept the fix in context. The Docker-free build completed 63 of 63 CTest cases. The production image completed 64 of 64, with two documented environment-dependent skips in that image run. The PostgreSQL contract was also run with its database URL supplied, and the build journal records the timestamp defect and corrected pass. The implementation ledger closed at 323 of 323 items.
Those counts do not prove that SHA-256 is unbreakable or that every database version formats every temporal type identically. They prove that this adapter's append, read, trigger, and verification path was exercised as a connected contract.
What I changed in my design habit
I used to treat canonicalization as a serializer feature. The useful unit is larger: producer, serializer, storage type, database output, and verifier. If any member rewrites a covered value, the hash protocol must account for that rewrite before the digest is committed.
The same question now applies to every field in a signed or hashed record:
- Will the database change its lexical form?
- Will a driver coerce its type?
- Will Unicode normalization differ between producer and reader?
- Will numeric precision survive a round trip?
- Will an export path preserve the same bytes?
This does not mean converting every value to a string. It means naming the representation boundary and testing a full round trip through the real storage engine.
The surprise was useful because the chain rejected a harmless semantic equivalence. Safety systems need that severity. If the verifier silently accepted alternate spellings, it would need a second, fuzzier definition of equality for every covered field. That would turn evidence verification into interpretation.
The cleaner rule is exact: normalize at the boundary that knows the representation, hash once, store those bytes, and verify the bytes read back. In this project, removing four zeroes restored that rule. The size of the patch said nothing about the size of the invariant it protected.