Architectural Brief: Multi-Agent RAG Platform

One user testing a prototype hit $47 in API charges before lunch. No cost tracking, no model routing, no budget enforcement. The production system had to do two things at once: deliver accurate, grounded answers from uploaded documents, and prevent the LLM bill from outrunning the value. Every decision in this brief traces back to cost. Route queries to the cheapest model that can handle them. Track every cent per user. Cut access at $10/day. The architecture wraps 200+ models behind a single API key, combines three retrieval signals across three databases, and runs guardrails on every request.

System Topology

Infrastructure Decisions

• LLM Gateway: OpenRouter. Chose over direct OpenAI and Anthropic SDKs because one API key unlocks 200+ models. Switching from GPT-4o-mini to Claude 3.5 Sonnet is a routing table change, not a code change. The client is OpenAI-compatible, so dropping OpenRouter later requires changing one base URL.
• Vector Storage: PostgreSQL 16 with pgvector. Chose over Pinecone and dedicated vector databases because the document metadata, conversation history, cost logs, and embeddings all live in one database. One connection pool, one backup strategy, one migration tool. pgvector's IVFFlat index handles the current scale without a second service.
• Knowledge Graph: Neo4j 5 Community. Chose over property graphs in PostgreSQL because Cypher queries for multi-hop entity traversal are readable and fast. The original design treated Neo4j as a required service. First deploy to the 1GB VPS proved that wrong: the JVM alone needs 512MB, leaving nothing for the application, PostgreSQL, and Redis to share. Making graph search return empty results instead of crashing was a 15-line change that kept the architecture without the runtime cost.
• Cache Layer: Redis 7 with hiredis. Chose over application-level caching because rate limiting and session state need sub-millisecond reads shared across workers. The semantic cache itself lives in PostgreSQL (pgvector cosine similarity at 0.95 threshold), but Redis handles the rate-limit counters with INCR + EXPIRE.
• Framework: FastAPI with async SQLAlchemy. Chose over Django because every route handler is async, and the LLM calls (120-second timeout) would block synchronous frameworks. Pydantic v2 handles request validation and config management in the same type system.
• Agent Framework: Pure Python ReAct loop, 90 lines. Chose over LangGraph and LangChain because the agent needs exactly one thing: call an LLM, check for tool calls, execute them, repeat up to 5 times. A short executor does that without 40+ transitive dependencies.
• Injection Detection: 8 weighted regex patterns. Chose over LLM-as-judge because regex runs in microseconds. LLM-as-judge adds 500ms+ latency and costs money on every request. The injection threshold (0.8) is tunable without retraining.

Constraints That Shaped the Design

• Input: Documents (PDF, TXT, Markdown, URLs) uploaded via REST API, or natural language queries sent to the chat endpoint. Documents are deduplicated by SHA-256 content hash before embedding.
• Output: Grounded AI responses with source attribution, relevance scores, and per-request cost tracking. Streaming available via SSE.
• Scale Handled: Single-tenant deployment on a 1GB DigitalOcean VPS. With 512-character chunks and 1536-dimensional embeddings, the pgvector IVFFlat index stays in memory up to ~50,000 chunks. Beyond that, the index needs HNSW and likely a dedicated machine.
• Hard Constraints: $10/day LLM budget per user, enforced at the cost tracker before every call. OpenRouter rate limits handled with 3-attempt exponential backoff (1s min, 8s max). Neo4j dropped from production because 1GB RAM cannot run the app, PostgreSQL, Redis, and a JVM-based graph database simultaneously. Rate limiting uses fixed-window Redis counters (INCR + EXPIRE per api_key:path:minute) with fail-open on Redis errors.
• Operational Boundary: Deployed with Traefik reverse proxy. TLS via Let's Encrypt. No auto-scaling: if the VPS is saturated, responses slow down rather than spawn new instances.

Decision Log