Architectural brief: Swarm Intelligence Gateway
WorldMonitor produces a current scenario package. MiroFish turns that package into a multi-agent simulation, then returns a report, agent activity, graph data, and profiles. The gateway is the part that makes the handoff repeatable. It receives a package, assigns it to a tenant, runs one simulation at a time, stores the raw and parsed result, and exposes the result to a frontend or another service.
The important boundary is between the gateway and MiroFish. MiroFish remains an external self-hosted service with its own lifecycle and data provider. The gateway does not pretend that a single HTTP request creates a finished forecast. It records each phase, polls the upstream status shape it actually returns, and keeps enough state to explain a failure.
System topology
WorldMonitor is read as a batch source. The poller checks its configured Redis key, accepts either a package or a pointer to an object in R2, validates the result, and creates a scenario only when the tenant and run ID have not already been seen. Webhook and Workflow Engine calls enter through the API and still pass through tenant checks before a job is queued.
The queue has a deliberately narrow role. Redis holds the BullMQ job state and the worker claims one simulation at a time. That limit came from the actual local run, where a parallel attempt ended with process exit -9. Simulations ask the MiroFish service to build a graph, run agents, and write a report. Running two at once would make the cost and memory shape hard to reason about on a small host.
PostgreSQL stores the gateway's durable view of the run. Scenarios and simulations retain upstream IDs and phase status. Graph nodes and edges mirror the returned MiroFish graph. Agent episodes store action logs with embeddings. Profiles, reports, and parsed predictions give the frontend queryable data without asking MiroFish to rebuild the run. The database also holds tenant ownership on every row that can be queried through the API.
Infrastructure decisions
-
Fastify with TypeScript over a separate API gateway. The service needs a small set of authenticated routes beside a worker and a poller. Keeping those pieces in one typed process makes configuration and failure handling visible in the same codebase. A separate gateway would add another deployment, another health check, and another place to duplicate tenant rules.
-
PostgreSQL with pgvector over Zep for gateway-owned memory. The gateway needs graph rows, edges, episodes, profiles, and predictions in the same tenant-scoped database. pgvector supports episode search beside those tables, while Drizzle keeps the schema in the application repository. Zep would have introduced a second memory system and a separate retention boundary. MiroFish remains responsible for its own upstream graph provider during graph construction, but the gateway owns the copy needed for API reads and audit work.
-
BullMQ on Redis over an in-process promise. A request should not stay open while a simulation runs for minutes. BullMQ gives the gateway a durable job record, retry attempts, and a worker boundary. An in-process promise would disappear on restart and would make a web process responsible for long-running work.
-
One worker at a time over unconstrained concurrency. The first live attempt exhausted the available memory and was killed. The worker is configured with concurrency one so the host runs one full MiroFish lifecycle at a time. Higher concurrency might improve throughput on a larger host, but it would change the resource envelope and make local verification less repeatable.
-
REST status polling over a streaming protocol. WorldMonitor packages arrive in batches and MiroFish exposes phase-specific status endpoints. The client polls task, preparation, simulation, and report status with bounded timeouts. A streaming protocol would require upstream support for reconnects and event ordering that the service does not provide.
-
Gateway-owned lifecycle state over a single upstream status field. MiroFish needs ontology generation, graph build, simulation creation, preparation, start, report generation, and data reads. The gateway records those phases in its simulation row and maps upstream responses that arrive under nested data objects. Treating the service as one opaque request would hide the exact phase that failed.
-
Direct package support plus R2 pointer resolution over one storage format. Local verification can use the package already in Redis. Larger deployments can pass a pointer and let the package reader fetch the object with a bounded request and bearer token. Supporting both keeps the ingestion contract useful without copying every package through the gateway process.
-
Feature-flagged ecosystem events over a hard notification dependency. Completed and failed simulations can publish the standard event envelope to Notification Hub. The publisher checks the feature flag and required credentials at startup. With the flag off, the gateway can run with only its own API, database, queue, and MiroFish service.
Constraints
- The gateway does not own the WorldMonitor or MiroFish repositories. Their APIs, response wrappers, status names, and local service settings are external inputs.
- MiroFish can return a task ID before the useful result exists. Graph preparation and report generation have their own waits, so each phase has its own polling function and timeout.
- Local simulations consume enough memory that concurrency must remain one for the documented verification path.
- Each protected query must carry a tenant condition. API key authentication resolves an active tenant before route handlers query scenarios, simulations, predictions, actions, profiles, or reports.
- Package deduplication is based on tenant and WorldMonitor run ID. A repeated Redis value must not create another simulation.
- Prediction confidence is stored as a decimal string so the database value does not lose precision when it crosses the API and frontend boundaries.
- The gateway can record a report with no structured predictions when the parser finds no supported prediction shape. A report is not silently rewritten to make the parser look successful.
- Deployment, DNS, TLS, image publishing, and hosted upstream services remain outside this local-only project completion.
Decision log
| Decision | Alternative rejected | Reason |
|---|---|---|
| Store graph nodes, edges, episodes, profiles, reports, and predictions in PostgreSQL | Keep only MiroFish IDs and fetch results on demand | The frontend needs stable reads after a run completes. A database copy also gives the gateway tenant filters, retention jobs, and an audit trail when the upstream service is unavailable. |
| Accept WorldMonitor R2 pointers | Require every package to be embedded in Redis | Large packages do not need to pass through Redis as one value. Pointer resolution keeps the Redis message small while direct JSON remains convenient for local runs. |
| Poll each MiroFish lifecycle phase separately | Use one long request with a fixed sleep | The upstream contract exposed different response wrappers and status fields for tasks, preparation, simulation, and reports. Separate polling makes those differences explicit and gives each phase a bounded timeout. |
| Queue simulation work | Run orchestration inside the API handler | A report can take minutes. The queue lets the API acknowledge the job and lets the worker record retries and permanent failure without holding an HTTP connection. |
| Keep worker concurrency at one | Start multiple MiroFish runs in parallel | A real local attempt was killed with exit -9 under memory pressure. One worker keeps the verification path inside the available host budget. |
| Publish completion events only when enabled | Make Notification Hub mandatory | The gateway is useful as a standalone service. Feature flags allow event-driven consumers to opt in without making their availability a prerequisite for simulation completion. |
The resulting shape is intentionally plain. WorldMonitor provides a package, BullMQ gives it a durable job, the worker completes a known MiroFish lifecycle, and PostgreSQL keeps the result queryable. Each boundary has a named failure mode and a local check that can prove whether the handoff worked.