When a Redis Payload Became a Pointer

{
  "runId": "run-identifier",
  "pkgKey": "object-key",
  "schemaVersion": "simulation-package.v2",
  "theaterCount": 1,
  "generatedAt": 1756378800000
}

That object is valid JSON. It is not a simulation package.

WorldMonitor originally placed the complete package in Redis. The gateway could read the value, parse it, validate the scenario fields, and create work. The publisher later moved the larger object to R2 and left a pointer in Redis. The key name stayed the same. The transport meaning changed underneath it.

If I had patched the parser to accept pointer fields, I would have mixed storage lookup with domain validation. Direct JSON ingestion would become harder to reason about, manual ingestion would inherit R2 concerns it did not need, and retries could create duplicate scenarios once the fetch finally succeeded.

I put one resolution step between JSON parsing and package parsing. It accepts either representation and returns one package shape. Everything after that boundary remains unaware of Redis pointers and object storage.

Transport is not the scenario contract

The gateway receives WorldMonitor data through two routes. A poller reads the latest value from WorldMonitor's Redis instance. An authenticated API route accepts a package directly. Both routes eventually call parseSimPackage() and persist the same scenario fields.

That shared parser is important because WorldMonitor's current package shape is not identical to the gateway's stable shape. The current feed can express simulationRequirement as an object. The gateway stores it as text. Theater fields can be nested differently. Constraints arrive as maps and become arrays. Ranking and strength values are clamped to the accepted range, HTML tags are stripped from text, and the entity list is capped at 20 before persistence.

Those are domain normalization rules. Fetching an object from R2 is a transport rule. I kept them in separate modules because they fail differently.

A pointer with missing R2 configuration is an operational error. A remote object that returns a non-success HTTP status is also an operational error. The object fetch has a 30-second timeout. A retrieved package with the wrong schema is a validation error. The first category may succeed on the next polling cycle. The second category should not enter the simulation queue at all.

One extra branch at the boundary is the cost. Every caller must pass optional R2 configuration even when direct packages are common. That branch buys a single downstream contract and keeps object-storage credentials away from the parser and manual-ingestion route.

Redis is a notification surface, not the package archive

Once the payload moved to R2, Redis became a statement about the latest available run. The pointer carries enough metadata to identify that run and locate its object. It does not become authoritative scenario data until the object is fetched and passes the package parser.

I resisted storing the pointer itself as rawPackage. Doing that would make every later replay depend on a remote object that can be moved, expired, or replaced. The gateway stores the accepted package content in PostgreSQL with the scenario. A report can then be traced to the input the gateway actually validated, even if the publisher's latest pointer changes.

The cost is duplicated storage. R2 keeps the publisher's object and PostgreSQL keeps the gateway's accepted copy. For these scenario documents, reproducibility matters more than avoiding a modest JSON copy. If packages grow large enough to pressure database backups, I would store a content-addressed archive and retain its hash with the scenario. I would not point historical simulations at a mutable “latest” object.

The poller closes its WorldMonitor Redis connection in a finally block after every cycle. That favors isolation over connection reuse. A fresh connection pays setup cost on each poll, but a dead external Redis client cannot linger inside the service and consume retries between scheduled runs. At the current polling rate, predictable cleanup wins.

One poll cycle, one normalization path

The poller connects to a Redis service owned by WorldMonitor, not the Redis instance used by BullMQ. It disables reconnect loops for a single poll, allows one request retry, and gives the connection five seconds. If that cycle fails, the scheduled job can try again later without leaving a reconnecting client behind.

The core path looks like this:

parsed = await resolveSimulationPackage(
  parsed,
  worldMonitorR2Config(),
);

const pkg = parseSimPackage(parsed);

const existing = await db
  .select()
  .from(scenarios)
  .where(
    and(
      eq(scenarios.tenantId, tenantId),
      eq(scenarios.worldmonitorRunId, pkg.runId),
    ),
  );

if (existing.length > 0) {
  return { ingested: false };
}

const [inserted] = await db
  .insert(scenarios)
  .values({
    tenantId,
    worldmonitorRunId: pkg.runId,
    title: pkg.title,
    theaters: pkg.selectedTheaters,
    entities: pkg.entities,
    eventSeeds: pkg.eventSeeds,
    constraints: pkg.constraints,
    simulationRequirement: pkg.simulationRequirement,
    source: SCENARIO_SOURCE.POLLER,
    rawPackage: pkg,
  })
  .returning({ id: scenarios.id });

The excerpt comes from pollWorldMonitor(). Resolution happens before normalization. Duplicate detection happens after normalization has established a trustworthy runId and before any queue message is created.

I also kept the normalized package in rawPackage. The named scenario columns support ordinary product queries, while the JSON copy preserves the accepted input for later diagnosis. Calling it raw is slightly historical because it contains the normalized package rather than the original pointer. That is a naming debt I would fix before exposing the field outside internal tooling.

What surprised me was how easily valid JSON could be mistaken for valid domain data. JSON.parse() answered only whether bytes formed an object. It said nothing about whether that object was a scenario, a pointer, or stale metadata. Once the publisher introduced R2, treating JSON parsing as ingestion success became an attractive lie.

Idempotency needs the tenant and the run

The poller wakes on a schedule. BullMQ can retry. An operator can submit the same package manually after a poll. A network timeout can happen after PostgreSQL commits but before the caller sees success. Duplicate delivery is normal under those conditions.

WorldMonitor's runId gives the gateway a stable source identity, but it is not globally sufficient. Two tenants may deliberately consume the same WorldMonitor run. The idempotency key is the pair of tenantId and worldmonitorRunId.

The application performs a readable duplicate check and returns without enqueueing another simulation. The database schema backs that check with a unique constraint on the same pair. I wanted both. The early query gives a quiet, expected path for the poller. The constraint closes the race where two workers check before either one inserts.

Manual ingestion uses the same parser and key but reports a conflict instead of silently skipping. That difference is about caller intent. A poller repeatedly seeing the latest run should remain quiet. A person or service explicitly submitting a duplicate deserves a clear response that no new scenario was created.

There is a trade-off in binding idempotency to runId. If WorldMonitor republishes corrected content under the same ID, the gateway will keep the first accepted scenario. That is safer than silently changing an input after a simulation may have started. A correction needs a new run ID or a separate revision contract. Mutable source identities would make reports impossible to reproduce.

Validation decides what never reaches the queue

The parser accepts the legacy package and the current version-two package, then returns the gateway's stable SimPackage. It does not let every downstream consumer carry version branches.

The current feed can hold theaters under a simulation context. The parser maps them into the gateway's selected-theater records. A requirement object is joined into one textual instruction. Constraint maps become flat lists. Missing ranking scores or entity strengths receive a neutral value of 0.5, and supplied values cannot escape the zero-to-one range.

The entity cap of 20 is a conscious loss of input. It limits prompt and graph pressure before MiroFish receives the seed document. Keeping the first 20 entries is simple and deterministic, though it assumes WorldMonitor orders them by relevance. If that ordering contract weakens, I would rank explicitly or reject oversized packages. Quietly taking an arbitrary subset would be indefensible.

Sanitization removes HTML tags from text fields because the same data reaches seed documents and a browser-facing product. This is not the only output safety boundary, but it stops source markup from becoming part of the simulation instruction by accident.

Invalid JSON, a missing key, or a package that fails schema validation produces no queue work. The poller logs the condition and returns { ingested: false }. Connection and fetch failures also avoid the queue, while the per-tenant failure tracker can emit an outage event after three consecutive operational failures. A malformed package and an unavailable source are not reported as the same incident.

That distinction keeps alerting useful. An empty feed may be normal between WorldMonitor runs. A broken Redis connection repeated across cycles is an availability problem. Schema drift is a data-contract problem. One generic “ingestion failed” counter would hide which team needs to act.

The package can move without moving the gateway

The current reader supports complete JSON in Redis and a pointer whose object lives in R2. The rest of the pipeline receives the same scenario either way. It records one tenant-scoped source identity, queues one simulation, and retains the accepted package beside the fields the application queries.

I would add versioned correction semantics before allowing an existing run to change. I would also record a content hash beside the source ID so an operator can distinguish a harmless duplicate from conflicting content published under the same identity. Neither addition is needed to understand the present contract.

The important boundary sits earlier: storage location may change, while package meaning and idempotency stay under gateway control.