Making a swarm forecast accountable after the simulation ends
Swarm Intelligence Gateway is a local-first bridge for teams that monitor geopolitical, market, or social signals and need a repeatable way to run scenario analysis. The buyer is an intelligence product team. The operator is the engineer or analyst who owns the WorldMonitor and MiroFish services. The trigger pain is the handoff: WorldMonitor has a current scenario package, MiroFish has a long multi-agent lifecycle, and neither one gives the consuming product a tenant-scoped record of what finished, what failed, and which predictions came out of the run. This gateway accepts the package, runs the lifecycle through a queue, stores the evidence, and exposes the result for a frontend or another service.
The situation
WorldMonitor and MiroFish solve different halves of the problem. WorldMonitor watches live signals and prepares a simulation package. MiroFish takes that package through ontology generation, graph construction, simulation setup, agent execution, and report generation. A product team still needs a service between them.
Without that service, the handoff is a collection of scripts and assumptions. A package may arrive as direct JSON or as a pointer to object storage. A MiroFish response may put its useful data under a nested response.data object. Creating a simulation does not mean it is ready to start. Preparing it does not mean the report exists. When those facts stay in an operator's head, a failed forecast looks like a missing result rather than a named phase with a known cause.
The gateway gives the handoff a durable owner. It validates a package, checks the tenant and WorldMonitor run ID, creates a scenario, queues the work, and records every important result in PostgreSQL. The API returns the stored report, parsed predictions, actions, and profiles without making the frontend reconstruct the upstream lifecycle.
The cost of doing nothing
The honest direct hosting cost for the current proof is US$0 per month. The project has no public deployment and no hosted runtime bill. That number is a boundary, not a return-on-investment claim.
The cost still shows up in work. An operator who keeps the handoff manual must watch a multi-minute run, remember which upstream IDs belong together, and check several response states before deciding whether the forecast is complete. The latest local run took just under eight minutes. It persisted 12 graph nodes, 12 graph edges, 37 agent episodes, two profiles, an 8,366-character report, and four parsed predictions. Without the gateway's stored result, those facts remain scattered across upstream responses and process logs.
The first local attempt also ended with process exit -9 when too much work competed for memory. The next run stopped 21 nonessential containers and kept simulation concurrency at one. That sequence is the cost of an undocumented resource assumption. It was found in a local proof, not hidden behind a green mock test. A hosted operator would pay for the same uncertainty in engineer time, failed runs, or a larger machine. This repository does not assign a made-up annual saving to that uncertainty.
What I built
The gateway is a TypeScript Fastify service with three durable concerns. First, it owns a tenant-scoped record of scenarios and simulations. Second, it runs MiroFish from a BullMQ worker instead of holding an HTTP request open. Third, it stores the data that a consuming product needs after the upstream service has moved on.
The WorldMonitor poller reads the latest simulation package from Redis. It accepts direct JSON and resolves a pointer when the package lives in R2. Validation happens before a scenario is inserted. A tenant and WorldMonitor run ID form the deduplication boundary, so the same latest value does not become a new simulation on every poll.
The worker completes the MiroFish lifecycle in explicit phases. It generates an ontology, builds a graph, creates a simulation, prepares it, starts it, waits for completion, asks for a report, and then fetches action logs, profiles, and graph data. The client knows that task status may be nested, that a simulation can require a separate start call, and that a report must be generated before it can be read. Each phase has a bounded wait.
The gateway copies graph nodes and edges into its own PostgreSQL schema. It stores agent actions as episodes with a source key derived from stable action fields. The source key prevents duplicate log rows when the upstream data is read again. Local embeddings live beside those episodes in pgvector, so a tenant can search an agent's prior activity without handing the query to a second memory product.
The report parser extracts supported prediction shapes and keeps confidence as a decimal value. It does not turn an unrecognised sentence into a fake prediction. The frontend can show the upstream report while the API exposes the parsed subset for filters and panels.
System flow
Data model
Architecture layers
The decision log
| Decision | Alternative rejected | Why it matters |
|---|---|---|
| Put simulations behind BullMQ | Keep the full run in an API request | MiroFish work takes minutes. A queue gives the API a job boundary and gives the worker retry state without asking a browser to keep a connection open. |
| Keep concurrency at one | Run every queued simulation in parallel | The first local attempt was killed with exit -9 under memory pressure. One worker gives the documented local setup a known resource shape. |
| Mirror MiroFish data in PostgreSQL | Read every report and action from upstream on demand | The frontend needs stable data after completion, and the gateway needs tenant filters that apply to reports, graph rows, episodes, profiles, and predictions. |
| Use a source key for action episodes | Insert each upstream action on every read | MiroFish data can be fetched more than once. A hash built from stable action fields prevents one read from becoming duplicate memory. |
| Parse predictions after report storage | Treat every report sentence as structured data | The report is the evidence. The parser stores only fields it can identify and leaves the original text intact when extraction is uncertain. |
| Accept direct packages and R2 pointers | Force one WorldMonitor storage mode | Local runs are simple with direct JSON. Larger packages can remain in object storage while Redis carries a small pointer. |
| Keep Notification Hub optional | Make event delivery a completion prerequisite | The gateway can complete a local run without another service. When enabled, the standard event envelope gives downstream automation a clear hook. |
Ecosystem integration
The gateway fits the portfolio as an event-producing intelligence service. Webhook Engine can provide an inbound trigger when another system wants to submit a scenario. Workflow Engine can start a tenant run on a schedule or an HTTP step. Notification Hub can receive a completed or failed simulation event for delivery rules. The event publisher is feature-flagged, so the gateway still runs standalone with no ecosystem dependencies.
The upstream WorldMonitor and MiroFish repositories stay outside the portfolio's project pages. They are dependencies in the local proof, not claims of ownership. The gateway's public repository documents the boundary and the local run without implying a hosted product.
Results
The finished local proof crossed the boundary that matters. A real WorldMonitor package reached the gateway, the worker completed a real MiroFish lifecycle, and PostgreSQL retained data that the frontend can query later. The run produced 12 graph nodes, 12 edges, 37 agent episodes, two profiles, an 8,366-character report, and four parsed predictions.
That result does not turn a local build into a deployed service. It does show that the core product contract works: a monitored scenario can become a tenant-scoped, inspectable simulation result with a named queue, a bounded upstream lifecycle, and durable evidence after the process finishes.