MarketCompass — Architecture Overview
Status: Design for implementation Owner: CompassFoundry Labs Project: AI-powered equity research platform. This is not a greenfield project — an existing skeletal frontend + backend already works; this document describes the target architecture to implement into it.
1. Why This Architecture
MarketCompass's eventual Analyst Agent (Phase 4) needs a stable seam between four concerns: what should handle this request (Router), how execution is carried out (Orchestrator), what business capability is invoked (Skills), and what does the work (Tools + LLM Service). Cutting that seam once makes Phases 2–4 additive rather than a redesign.
2. Package Map
Every package below is real as of Phase 1; §12's diagram and §13's contract table are the authoritative description of how they fit together. chat.py is the API entry point behind which the Orchestrator sits; /api/chat/stream still runs the pre-Orchestrator ChatAgent/legacy route_request() path (phase1-implementation-plan.md Milestone 8), a known parallel path, not a gap.
backend/src/
├── api/ # FastAPI app factory, routes (health, chat, agents, providers)
├── agents/ # Orchestrator, AgentContext, ContextBuilder, legacy ChatAgent (/api/chat/stream only)
├── router/ # Router — formalized by routing-design.md
├── skills/ # Skill classes + Skill Factory
├── tools/ # Tool classes + Tool Registry
├── services/ # LLMService (llm/ package underneath — factory, provider_catalog, LangChain provider wrappers)
├── models/ # Pydantic contracts — domain-model-design.md
├── observability/ # log_event / record_metric / start_span facade
├── core/ # logging, tracing primitives — wrapped by observability/, not modified
├── streaming/ # SSE / token streaming, unaffected by any layer above
└── config/ # settings, validation, paths
config/
├── routing/ # skills_routing.yaml, thresholds.yaml
├── skills/ # registry.yaml
└── tools/ # registry.yaml
evaluations/ # placeholder — §7
Phase 2+ additions (not yet built — phases/phase2-implementation-plan.md): backend/src/rag/, backend/src/memory/, backend/src/skills/fallback_agent_skill.py, config/rag/, config/memory/, top-level data/.
3. LLM as a Platform Service, Not a Tool
The LLM is not modeled as LLMTool, a peer of NewsTool/SECFilingsTool. It's the reasoning engine every Skill depends on, not an optional external integration — so LLM-specific concerns (retries, failover, cost tracking, prompt versioning) live in a dedicated LLMService, not a generic Tool contract.
Skill
├── Tool Dependencies (0..n) -- external data: CompanyProfileTool, NewsTool, etc.
└── LLM Service Dependency (1) -- the platform's reasoning engine
| Skill | Tool Dependencies | LLM Service Dependency |
|---|---|---|
CompanySkill |
CompanyProfileTool |
LLMService |
FinancialSkill |
FinancialMetricsTool, CompanyProfileTool |
LLMService |
NewsSkill |
NewsTool |
LLMService |
GenericChatSkill |
(none) | LLMService |
FallbackAgentSkill (Phase 2+, §8) |
any, chosen at runtime by an internal LangChain agent | LLMService |
LLMService is injected into every Skill the same way — not optional, not looked up through the Tool Registry. Resolved once at Orchestrator/Skill-construction time from a single, application-scoped instance (llm-service-design.md).
Internally, LLMService uses LangChain's provider wrappers (langchain-anthropic, langchain-openai, etc.) instead of hand-rolled vendor SDK calls. LLMService's external contract (generate()/stream()) is unchanged — nothing above it is affected. See §8 for the full scope of where LangChain is (and isn't) used.
4. Centralized Configuration Hierarchy
Registries live in config/, not inside source packages — consistent with routing config and the project's "no code deployment for config changes" principle.
config/
├── routing/
│ ├── skills_routing.yaml
│ └── thresholds.yaml
├── skills/
│ └── registry.yaml
├── tools/
│ └── registry.yaml
├── rag/ # Phase 2+ — rag-design.md §2
│ ├── ingestion.yaml
│ ├── vectorstore.yaml
│ └── embeddings.yaml
└── memory/ # Phase 2+ — memory-design.md §2
├── short_term.yaml
└── long_term.yaml
backend/src/skills/, backend/src/tools/, backend/src/rag/, and backend/src/memory/ contain only code (base contracts, concrete implementations, registry/config loader logic) — never data. backend/src/config/settings.py holds explicit paths to each registry/config file (SKILLS_REGISTRY_PATH, TOOLS_REGISTRY_PATH, and — Phase 2+ — the config/rag//config/memory/ paths).
5. LLM Service Layer
Skills
↓
LLM Service (backend/src/services/llm_service.py)
↓
llm/ package (factory.py, provider_catalog.py, providers/* — LangChain wrappers inside)
↓
Provider SDKs (Anthropic, OpenAI, Ollama, OpenRouter)
LLMService is the only thing Skills are allowed to call for inference — no Skill imports llm/factory.py directly. Responsibilities owned here (not by Skills, not by llm/ alone): model/provider selection policy, retry handling, provider failover, token accounting, cost tracking, and prompt versioning. llm/ remains the low-level, provider-specific mechanics; LLMService is the policy layer on top.
6. Observability as a First-Class Layer
Named as an explicit layer, not just "logging exists" — the system's pitch is explainable, confidence-scored routing, which needs to be provable in telemetry, not just in a reasoning string returned once per request.
| Category | Examples |
|---|---|
| Structured logging | Every request logged with trace_id, route_name, confidence, skills_invoked, tools_invoked |
| Tracing | trace_context.py/tracing.py, spanning Router → Skill → Tool/LLM Service boundaries |
| Metrics | Route selection frequency, average route confidence, tool latency (p50/p95), tool failure rate, LLM token consumption, request cost, response latency |
| Routing confidence analytics | Distribution of confidence scores per route over time |
| Cost tracking | Token usage × provider pricing, surfaced per request and aggregated |
Observability sits beside every layer — every layer emits into it; nothing depends on it to function correctly.
7. Evaluation Framework Placeholder
A top-level evaluations/ package, structural only in Phase 1 — not implemented beyond a stub.
evaluations/
├── datasets/ # golden query/route/answer sets, empty in Phase 1
├── routing_eval.py # will score router accuracy against datasets/
└── response_eval.py # will score response quality/hallucination in later phases
Distinct from the unit/integration test suite (§14): evaluation answers "is the behavior still good," not "does the code work as written."
8. Router-First Flow, Context Builder, and LangChain Scope
This is the architecture's core control-flow decision. Full reasoning: langchain-rag-router-decision.md.
The Router runs first, on the raw query
The Router's YAML pattern-matching is cheap and shouldn't pay for embedding + vector search on every request, including trivial high-confidence ones. Current Tools (CompanyProfileTool, FinancialMetricsTool, NewsTool, SECFilingsTool) are live, structured lookups — a vector search doesn't improve on a direct API call for something like "AAPL's P/E ratio," and could introduce noise from stale embedded snapshots. High-confidence routing exists specifically because the query maps cleanly to a deterministic lookup. RAG earns its cost on the opposite case: queries that don't map to a clean Tool call and need synthesis across unstructured documents — which is why they're low-confidence in the first place.
So Router.route(query, AgentContext) -> RoutingDecision takes the raw query, not a RequestContext with retrieval already populated. The Context Builder is not an unconditional pre-Router step.
Flow
API → Orchestrator → Router (scores the RAW query)
│
┌─────────────────┴─────────────────┐
▼ ▼
HIGH CONFIDENCE LOW CONFIDENCE / NO ROUTE
│ │
▼ ▼
Skill Context Builder (RAG)
(fixed, hand-written execute()) (LangChain retrieval/embeddings
│ inside, called ONLY here;
┌────┴────┐ output mapped to typed
▼ ▼ RequestContext.retrieved_context)
Tools LLM Service │
▼
Fallback Agent Skill
(LangChain AGENT inside —
reasons over which Tools to
call, since sequence isn't fixed)
│
┌────┴────┐
▼ ▼
Tools LLM Service
Both branches return a typed SkillResult to the Orchestrator.
How SkillRequest.context gets populated on each branch
BaseSkill.execute(request: SkillRequest) (skills-design.md §2) is one contract for every Skill — there is no separate signature for high-confidence vs. fallback. What differs is how RequestContext gets built, not its shape:
- High confidence: the Orchestrator constructs a trivial
RequestContextitself, inline —user_query+session_id/trace_idfromAgentContext, emptyconversation_history/retrieved_context. This is data-wrapping, not a call to the Context Builder component; it costs nothing and requires no dependency injection. This is what "Context Builder absent as a dependency" (§8 below) actually means: the component that can do retrieval is never invoked, not thatRequestContextgoes unpopulated. - Low confidence / no route: the Orchestrator calls
ContextBuilder.build(query, agent_context) -> RequestContextinstead of building it inline. Phase 1 this returns the identical trivial shape (PassthroughContextBuilder); Phase 2+ it performs real retrieval and populatesretrieved_context.
This keeps SkillRequest{context: RequestContext, routing: RoutingDecision} (domain-model-design.md §3) and the BaseSkill.execute() signature unchanged and identical for every Skill, including FallbackAgentSkill — only the construction path is conditional, never the contract.
Context Builder: no-op in Phase 1, real RAG + memory from Phase 2
ContextBuilder.build(query, AgentContext) -> RequestContext is a no-op passthrough in Phase 1 (PassthroughContextBuilder, RequestContext wraps the raw query plus empty retrieved_context/conversation_history). The interface and its conditional (fallback-only) wiring were established in Phase 1 specifically so Phase 2 only changes internals, not callers, in two steps: RagContextBuilder (Milestone 1, rag-design.md §4 — real LangChain-based RAG retrieval only) is later replaced by RagMemoryContextBuilder (Milestone 4, rag-design.md §4, memory-design.md §5 — adds short-/long-term memory), invoked by the Orchestrator only on the low-confidence/no-route branch. Output is mapped to the existing typed RequestContext fields before leaving the component — LangChain never leaks past that boundary. High-confidence Skills never see the Context Builder — not injected, not optional, simply absent as a dependency, except for user_id: whichever code path constructs RequestContext on a given branch populates that field directly from AgentContext.user_id (Phase 2's Auth Foundation milestone) — the Orchestrator's own inline construction on the high-confidence branch, the Context Builder's build() on the fallback branch (the same way it already sets session_id/trace_id) — since it's a zero-cost identity fact, not a retrieval result.
Which queries actually reach this branch widens in Phase 2: routing-design.md §5b adds a reference/comparison-cue gate to the Router — a query that would otherwise route high-confidence but implicitly depends on prior context (e.g. "compare AAPL's P/E to the one I asked about earlier") gets low_confidence=true forced, specifically so it reaches the Context Builder's memory instead of silently losing that half of the question on a Skill that never sees it. See ADR-006-memory-strategy.md for the full reasoning and the accepted residual risk this doesn't catch (mitigated by a backstop instruction in every Skill's system prompt, skills-design.md §2).
LangChain scope
LangChain is scoped to three isolated pockets, each inside a component whose external contract is already a typed Pydantic model. It is never the orchestration framework — the YAML Router and Orchestrator control flow are unchanged, and Router/Orchestrator/Skill Factory never import or know about LangChain.
- LLM Service (§5) — provider wrappers, replacing hand-rolled SDK calls.
- Context Builder (above) — retriever/vectorstore/embedding components, fallback-only.
- Fallback Agent Skill (below) — agent reasoning over which Tools to call.
Fallback Agent Skill
A Skill invoked only when RoutingDecision.low_confidence == true or no route matched. Constructed by the Skill Factory from a config/skills/registry.yaml entry like any other Skill — declares Tool dependencies and an LLMService dependency the normal way, returns SkillResult. The only difference is internal to execute(): instead of a hand-written Tool call sequence, it uses a LangChain agent (create_react_agent + AgentExecutor, text-based ReAct prompting) to decide which Tools to call and in what order, because this is the one case where that sequence genuinely isn't known in advance. Indistinguishable to the Orchestrator from a normal Skill.
AgentExecutor needs a LangChain BaseChatModel to reason with — this is satisfied by a thin adapter (_LLMServiceChatModel, skills-design.md §3a) that wraps the Skill's injected LLMService.generate() rather than a raw provider client, so the agent's internal reasoning calls still go through LLMService's retry/failover/cost-tracking pipeline like every other Skill's LLM calls. Because ReAct reasons entirely in text (no native tool binding), the adapter only needs plain prompt-in/text-out — it never has to bridge LLMRequest/LLMResponse to LangChain's structured tool_calls, so those contracts stay exactly as domain-model-design.md §3 defines them. The adapter lives inside fallback_agent_skill.py, the same "map/adapt at the boundary" placement already used for the BaseTool → StructuredTool adapter.
Skill → Tools → LLM Service → SkillResult (ordinary Skill, e.g. FinancialSkill)
Skill → [LangChain agent decides which Tools, in what order] → LLM Service → SkillResult (Fallback Agent Skill)
What if a high-confidence Skill someday needs retrieved context?
Deferred, not built speculatively. Orchestrator-owned conditional retrieval (the design above) is chosen over making ContextBuilder an injectable dependency any Skill could request, because no Skill today has a genuine need for retrieved context. If one arises, the retrofit is small and local: promote that one Skill to receive ContextBuilder via injection so it can call build() itself before executing — note this is not the pattern FallbackAgentSkill uses today (it never receives ContextBuilder; the Orchestrator calls it and passes the resulting RequestContext via the ordinary SkillRequest, skills-design.md §3a) — without forcing the more general injection pattern onto Skills that don't need it.
A related, explicitly deferred question: could a high-confidence Skill's own internal logic ever need dynamic, runtime-decided planning (variable steps depending on an intermediate result), independent of the Router's confidence in routing to it? If so, that Skill could be internally agentic using the same containment pattern as the Fallback Agent Skill. Hypothetical, not a concrete near-term need — no action taken.
Non-goals
No LCEL chains, LangChain agents, or LangGraph in Router, Orchestrator, or Skill Factory. No Skill other than the Fallback Agent Skill (and, hypothetically, a future dynamically-planning Skill per above) uses LangChain's agent framework internally.
9. Rules Engine as a Platform Service, Not a Tool (Phase 3)
Not implemented in Phase 1 — the Rules Engine itself is a Phase 3 deliverable. This section only decides where it will live, using the same Tool-vs-Service test §3 already established for the LLM: is this an external, optional capability (Tool) or a fixed, always-present capability a Skill's core function depends on (Service)?
| Question | LLM Service (§3) | Rules Engine |
|---|---|---|
| External integration MarketCompass doesn't own? | No — internal reasoning capability | No — internal, declarative business logic owned by CompassFoundry Labs |
| Optional per-Skill, or structurally required? | Required by every Skill | Required by every Skill that produces financial analysis (FinancialSkill, CompanySkill) |
| Cross-cutting policy concerns? | Yes — retry, failover, cost tracking | Yes — rule versioning, consistent threshold evaluation semantics, explainability output shape |
| Interprets a Skill's own output, or fetches raw external data? | N/A (produces output) | Interprets/normalizes a Skill's already-fetched Tool data — not itself a data source |
Every answer matches the LLM's, not a Tool's — so this is RulesEngineService, not RulesEngineTool.
Skill
├── Tool Dependencies (0..n) -- external data
├── LLM Service Dependency (1) -- reasoning engine
└── Rules Engine Service Dependency (0..1) -- declarative business logic, financial-analysis Skills only
Skill → Tools → LLM Service → Rules Engine Service → SkillResult
RulesEngineService is injected the same way LLMService is — resolved once at Skill-construction time, not looked up through the Tool Registry. Skills that don't produce financial analysis (NewsSkill, GenericChatSkill) simply have no Rules Engine dependency.
Whether SkillResult.metadata: dict is sufficient for rule output or needs to become a typed field is an open Phase 3 design-time call, not resolved here. This section does not design the Rules Engine's internals (rule schema, evaluator implementation, config/rules/ YAML shape) — that's Phase 3 scope.
10. Security as a Cross-Cutting Concern
Security isn't a component with its own single responsibility — like Observability (§6), it's a concern that touches every layer. Controls attach to two existing chokepoints rather than a new security/ package:
- API layer boundary — authentication, rate limiting, input validation, resolved once before a request reaches the Orchestrator.
- LLM Service boundary — prompt injection handling, resolved once since every Skill's LLM call already funnels through
LLMService(§3).
| Phase | New security surface | Approach |
|---|---|---|
| 1 | Public API, secrets, cost-abuse | .env for secrets, slowapi for rate limiting, scoped CORS, pip-audit |
| 2 (RAG + Memory + Auth Foundation, Milestone 3) | First real user data — long-term memory (memory-design.md); watchlists/reports remain unscheduled, further-future |
FastAPI Users (self-hosted, MIT-licensed, FastAPI-native); JWT auth at the API boundary; user_id-scoped queries for data isolation (security-design.md §"Phase 2") |
| 3 (Rules Engine) | YAML config trust boundary | yaml.safe_load() only; reuses Pydantic schema validation |
| 4 (Analyst Agent) | Prompt injection via retrieved content | Delimited context blocks inside LLMService; citation grounding doubles as an injection-detection aid |
No paid identity provider (Okta, Auth0, WorkOS) or SSO/SAML at any phase — ruled out by the free-tooling constraint and by not being warranted for a single-tenant project with no enterprise customers. Total added cost across all four phases: $0.
11. Skill Factory: Construction, Not Behavior
A Skill Factory removes dependency-wiring boilerplate when constructing Skills from config/skills/registry.yaml. It handles construction only — resolving a YAML entry's declared Tools, LLMService config, and optional RulesEngineService dependency, then injecting them into a hand-written BaseSkill subclass. It never interprets YAML as behavior: a Skill's execute() — prompt construction, branching, response shaping (including the Fallback Agent Skill's internal LangChain agent, §8) — remains ordinary Python.
This follows the same declarative-vs-behavioral axis already applied elsewhere: Router thresholds are YAML, pattern-matching logic is code; Rules Engine thresholds are YAML, the evaluator is code.
Adding a Skill still requires writing a Python class — a new Skill always means new behavior. What the factory removes is the hand-written constructor call wiring that class to its dependencies, not the need to write the class. Three-step process: class, class-map entry, YAML entry — an explicit registry, not dynamic discovery.
12. Layered Architecture
flowchart TB
subgraph client["Client"]
UI["Frontend (React/Vite)"]
end
subgraph api["API Layer — backend/src/api"]
ROUTES["/api/chat, /api/agents, /api/health, /api/providers"]
end
subgraph orch["Agent Orchestrator — backend/src/agents"]
ORCH["Orchestrator"]
CTX["AgentContext"]
end
subgraph models["Models Layer — backend/src/models"]
MCTX["context.py\n(RequestContext)"]
MROUT["routing.py\n(RoutingDecision)"]
MSKILL["skills.py\n(SkillRequest, SkillResult)"]
MTOOL["tools.py\n(ToolResult)"]
MLLM["llm.py\n(LLMRequest, LLMResponse)"]
end
subgraph routing["Confidence Router — backend/src/router"]
ROUTER["Router (scores RAW query)"]
PM["Pattern Matcher"]
ENT["Entity Extractor"]
CFG["Config Loader"]
end
subgraph cb["Context Builder — backend/src/agents/context_builder.py"]
CBUILD["ContextBuilder\n(no-op Phase 1; RagContextBuilder Milestone 1;\nRagMemoryContextBuilder Milestone 4+;\ncalled ONLY on low-confidence/no-route branch)"]
RAGPKG["rag/ (Phase 2+)\nLangChain retriever/vectorstore/embeddings\nrag-design.md"]
MEMPKG["memory/ (Phase 2+)\nshort-term (session_id) + long-term (user_id, SQLite)\nmemory-design.md"]
CBUILD --> RAGPKG
CBUILD --> MEMPKG
end
subgraph skills["Skills Layer — backend/src/skills"]
SREG["Skill Registry (loader)"]
S1["FinancialSkill"]
S2["CompanySkill"]
S3["NewsSkill"]
S4["GenericChatSkill"]
S5["FallbackAgentSkill (Phase 2+)\nLangChain agent inside\nreplaces generic_chat on the fallback branch"]
end
subgraph tools["Tools Layer — backend/src/tools"]
TREG["Tool Registry (loader)"]
T1["CompanyProfileTool"]
T2["FinancialMetricsTool"]
T3["NewsTool"]
T4["SECFilingsTool"]
end
subgraph llmsvc["LLM Service — backend/src/services/llm_service.py"]
LLMS["LLMService\n(model selection, retry, failover, cost/token accounting)"]
LLMPKG["llm/ package\n(factory, provider_catalog,\nLangChain provider wrappers in providers/*)"]
end
subgraph cfgstore["Centralized Configuration — config/"]
C1["config/routing/*.yaml"]
C2["config/skills/registry.yaml"]
C3["config/tools/registry.yaml"]
end
subgraph obs["Observability — backend/src/observability"]
LOG["Structured Logging"]
TRACE["Tracing"]
METRICS["Metrics"]
end
subgraph eval["Evaluation (placeholder) — evaluations/"]
EV["routing_eval.py / response_eval.py"]
end
UI --> ROUTES --> ORCH
ORCH --> CTX
ORCH -->|raw query| ROUTER
ROUTER --> PM
ROUTER --> ENT
ROUTER --> CFG
CFG -.reads.-> C1
ROUTER -->|produces RoutingDecision| MROUT
MROUT -->|RoutingDecision| ORCH
ORCH -->|HIGH CONFIDENCE: constructs SkillRequest| MSKILL
ORCH -->|LOW CONFIDENCE / NO ROUTE| CBUILD
CBUILD -->|produces RequestContext| MCTX
MCTX -->|RequestContext, retrieved_context populated| MSKILL
MSKILL -->|SkillRequest| SREG
SREG -.reads.-> C2
SREG --> S1 & S2 & S3 & S4 & S5
S1 & S2 & S3 & S5 -->|invoke tools| TREG
TREG -.reads.-> C3
TREG --> T1 & T2 & T3 & T4
T1 & T2 & T3 & T4 -->|produces ToolResult| MTOOL
MTOOL -->|ToolResult| S1 & S2 & S3 & S5
S1 & S2 & S3 & S4 & S5 -->|constructs LLMRequest| MLLM
MLLM -->|LLMRequest| LLMS
LLMS --> LLMPKG
LLMS -->|produces LLMResponse| MLLM
MLLM -->|LLMResponse| S1 & S2 & S3 & S4 & S5
S1 & S2 & S3 & S4 & S5 -->|produces SkillResult| MSKILL
MSKILL -->|SkillResult| ORCH
ORCH -.emits.-> obs
ROUTER -.emits.-> obs
SREG -.emits.-> obs
TREG -.emits.-> obs
LLMS -.emits.-> obs
obs -.feeds.-> eval
13. Layer Contracts
| Layer | Responsibility | Must NOT do |
|---|---|---|
| API Layer | Auth, request validation, response shaping, streaming transport | Contain routing or business logic |
| Orchestrator | Manage AgentContext, call the Router first, conditionally invoke Context Builder on the fallback branch, invoke Skills, call MemoryService.record() twice (user turn, then assistant turn) after FallbackAgentSkill returns (Phase 2+ Milestone 4, memory-design.md §5), assemble the response |
Know about specific Tools or the LLM Service directly; retrieve memory/RAG context itself (that's the Context Builder's job); call Context Builder unconditionally; call MemoryService.record() on the high-confidence branch |
| Router | Score confidence per configured route against the raw query, return a routing decision | Execute anything; call Tools, the LLM Service, or the Context Builder; hold conversation state |
| Context Builder | Assemble the RequestContext the fallback path operates on (no-op in Phase 1; real LangChain retrieval in Phase 2+); invoked only on low-confidence/no-route |
Make routing decisions; call Tools or the LLM Service directly; run on the high-confidence path |
| Skills | Represent a business capability; orchestrate Tools and the LLM Service | Call another Skill directly; call llm/factory.py directly instead of LLMService; read registry YAML directly |
| Tools | Wrap one external, non-LLM integration with a typed input/output contract | Contain business/domain logic or routing logic; perform LLM inference |
| LLM Service | Model/provider selection policy, retries, failover, prompt execution, token/cost accounting (LangChain provider wrappers internal) | Contain business logic (that's a Skill's job) or fetch external non-LLM data (that's a Tool's job) |
Models (backend/src/models/) |
Define Pydantic models for all inter-component communication | Contain business logic or routing logic; perform I/O or external calls |
Configuration (config/) |
Express routing rules, skill registry, tool registry declaratively | Contain secrets (those stay in .env/settings.py) |
| Observability | Structured logs, traces, and metrics across every layer | Gate or alter request execution (side-effect-only) |
| Evaluation (placeholder) | Will assess routing accuracy and response quality against curated datasets | Run inline in the request path — offline/batch, not per-request |
14. Request Execution Flow
High-confidence path
sequenceDiagram
participant U as User
participant API as API Layer
participant O as Orchestrator
participant R as Router
participant S as Skill
participant T as Tool
participant L as LLM Service
participant M as Models Layer
participant Obs as Observability
U->>API: POST /api/chat {messages, session_id}
API->>O: dispatch(ChatRequest)
O->>O: build AgentContext (session, trace_id)
O->>R: route(raw query, AgentContext)
R->>R: extract entities, pattern match, score confidence
R->>M: construct RoutingDecision
M-->>R: RoutingDecision
R-->>O: RoutingDecision(skills, confidence, reasoning)
O-->>Obs: emit routing metrics (route, confidence)
Note over O: high confidence — Context Builder NOT called;<br/>O builds a trivial RequestContext inline
O->>M: construct SkillRequest(context, routing)
M-->>O: SkillRequest
O->>S: execute(SkillRequest)
S->>T: call(...)
T->>M: construct ToolResult
M-->>T: ToolResult
T-->>S: ToolResult
S-->>Obs: emit tool latency/success metrics
S->>M: construct LLMRequest(prompt, context)
M-->>S: LLMRequest
S->>L: generate(LLMRequest)
L->>L: select model/provider, apply retry/failover policy
L->>M: construct LLMResponse
M-->>L: LLMResponse
L-->>S: LLMResponse(content, model_used, tokens, cost)
L-->>Obs: emit token/cost/latency metrics
S->>M: construct SkillResult(content, citations, confidence)
M-->>S: SkillResult
S-->>O: SkillResult
O-->>API: AgentResponse
API-->>U: streamed / JSON response
Low-confidence / fallback path
sequenceDiagram
participant O as Orchestrator
participant R as Router
participant CB as Context Builder
participant Mem as Memory Service
participant FS as Fallback Agent Skill
participant T as Tool
participant L as LLM Service
participant M as Models Layer
O->>R: route(raw query, AgentContext)
R-->>O: RoutingDecision(low_confidence=true OR no route)
Note over O: fallback branch — Context Builder called
O->>CB: build(query, AgentContext)
CB->>CB: retrieve (rag/, Phase 2+)
CB->>Mem: load(AgentContext) [Phase 2+ Milestone 4]
Mem-->>CB: MemorySnapshot(conversation_history, memory_items)
CB->>M: construct RequestContext(retrieved_context=[...], conversation_history=[...], memory_items=[...])
M-->>CB: RequestContext
CB-->>O: RequestContext
O->>M: construct SkillRequest(context, routing)
M-->>O: SkillRequest
O->>FS: execute(SkillRequest)
FS->>FS: LangChain agent decides which Tools to call, in what order
FS->>T: call(...) [one or more, order not fixed]
T-->>FS: ToolResult(s)
FS->>L: generate(LLMRequest) [via _LLMServiceChatModel adapter]
L-->>FS: LLMResponse
FS->>M: construct SkillResult
M-->>FS: SkillResult
FS-->>O: SkillResult
Note over O: Milestone 4+ — record both turns after the Skill returns
O->>Mem: record(AgentContext, ConversationTurn(role="user", content=query))
O->>Mem: record(AgentContext, ConversationTurn(role="assistant", content=SkillResult.content))
15. Configuration-Driven Behavior
| File | Owns | Consumed by |
|---|---|---|
config/routing/skills_routing.yaml |
Route → required skills, keyword/pattern hints, entity requirements | Router |
config/routing/thresholds.yaml |
Per-route confidence thresholds, fallback margin | Router |
config/skills/registry.yaml |
Skill name → implementing class, required Tool names | Skill Registry |
config/tools/registry.yaml |
Tool name → implementing class, provider config keys | Tool Registry |
backend/src/config/settings.py |
LLM Service policy defaults (default provider/model, retry count, failover order), secrets via .env |
LLM Service |
config/rag/*.yaml (Phase 2+) |
Corpus source paths, chunking parameters, Chroma persist directory/collection name, embedding provider/model | rag/ package, via the Context Builder |
config/memory/*.yaml (Phase 2+) |
Short-term TTL/max-turns; long-term SQLite path and retention policy | memory/ package, via the Context Builder |
16. Testing Strategy
| Layer | Test Type | Key Assertions |
|---|---|---|
| Models | Unit | Every Pydantic model validates correctly; field types enforced; required fields reject missing values |
| Router | Unit | Fixed YAML fixtures produce expected routes/confidence, scored on the raw query with no RequestContext dependency |
| Context Builder | Unit | Phase 1 no-op returns RequestContext unchanged; interface is stable for Phase 2 substitution; integration test confirms it is invoked only on the low-confidence/no-route branch, never on high-confidence |
| Skills | Unit + contract | Skill honors declared Tool dependencies and its single LLMService dependency; both are mocked; Skill accepts SkillRequest and returns SkillResult; Fallback Agent Skill's internal LangChain agent is mocked/stubbed for unit tests |
| Tools | Unit + integration | Unit tests mock external clients; tagged integration suite hits sandboxed APIs; Tool returns ToolResult with correct structure |
| LLM Service | Unit + integration | Unit tests verify retry/failover/model-selection policy with a mocked llm/ package; integration tests verify at least one real provider round-trip through the LangChain wrapper; accepts LLMRequest and returns LLMResponse |
| Orchestrator | Integration | End-to-end high-confidence path: API → Router → Skill → Tool/LLM Service → response; end-to-end fallback path: API → Router → Context Builder → Fallback Agent Skill → response; all model round-trips validate |
| Config | Schema validation | All config/ subtrees validated against Pydantic schemas at startup |
| Observability | Smoke test | Every layer's emitted log/metric event conforms to a shared schema (trace_id present, etc.) |
| Evaluation (placeholder) | N/A in Phase 1 | Stub functions exist and are wired to a (currently empty) dataset location |
17. Forward Compatibility
| Future Phase | What this architecture already provides | What gets added, not rewritten |
|---|---|---|
| Phase 2 (RAG + Memory + Auth) | Architecture fully designed, not yet implemented — phases/phase2-implementation-plan.md, ADR-006, ADR-010, rag-design.md, memory-design.md. Context Builder interface already wired to the fallback branch; RequestContext already has retrieved_context (currently always empty) |
RagContextBuilder (Milestone 1) replaces the no-op and populates retrieved_context; RequestContext.user_id is added (Milestone 3) and RequestContext.memory_items is added (Milestone 4), each alongside AgentContext's matching field; RagMemoryContextBuilder (Milestone 4) replaces RagContextBuilder; FallbackAgentSkill replaces generic_chat on the fallback branch (Milestone 2); Auth Foundation adds user_id end to end (Milestone 3); Router gains a reference/comparison gate (routing-design.md §5b, Milestone 4) — Skills/Orchestrator control flow otherwise untouched |
| Phase 3 (Rules Engine) | Skills already call Tools + LLMService; RulesEngineService placement already decided (§9); SkillResult has metadata: dict escape hatch |
RulesEngineService added as a third standard Skill dependency; config/rules/*.yaml added to the centralized config hierarchy; rule results land in SkillResult.metadata or a new typed field |
| Phase 4 (Analyst Agent) | Orchestrator already separates "decide" (Router) from "assemble context" (Context Builder, conditional) from "do" (Skills); all inter-layer communication is typed | Multi-step planning inside the Orchestrator, reusing the same Context Builder/Router/Skill calls per step; models gain additive fields (e.g. WorkflowState) without breaking existing interfaces |
| Domain entities (Phase 2–3) | backend/src/models/ already exists as internal contracts layer |
New backend/src/domain/ layer for business entities (Company, FinancialStatement, etc.), added alongside models/, not replacing it |
| Evaluation maturity | evaluations/ package and directory convention already exist |
Real datasets, real scoring logic — no new top-level structure needed |
| Observability maturity | Metrics categories already named and emission points already wired | Dashboards/alerting on top of already-emitted data — no new instrumentation points needed |
18. Explicit Non-Goals
- No dynamic/plugin discovery for Skills or Tools — explicit registries only, under
config/. - No multi-step planning in the Orchestrator yet.
- No embedding-based semantic routing for the Router itself (
ADR-002-yaml-routing.md) — the Router always scores the raw query via YAML pattern-matching, never via retrieval. - Context Builder is a no-op in Phase 1, not real memory; from Phase 2 it runs only on the low-confidence/no-route branch, never unconditionally and never on the high-confidence path.
- Evaluation framework is a placeholder, not a working pipeline, in Phase 1.
- No multi-agent systems, autonomous planning loops, or graph orchestration frameworks (LangGraph, etc.) anywhere in Router, Orchestrator, or Skill Factory.
- LangChain is never the orchestration framework and is confined to three pockets (§8): LLM Service provider wrappers, Context Builder retrieval internals, and the Fallback Agent Skill's internal agent. No other Skill uses a LangChain chain or agent by default.
- Security is scoped deliberately: no paid identity provider, no enterprise SSO, no dedicated authorization engine, no compliance framework (SOC 2/HIPAA) — appropriate to a single-tenant, no-enterprise-customer project.
- The Skill Factory (§11) is scoped to construction/dependency-wiring only — YAML never expresses Skill behavior, and class resolution uses an explicit map, not dynamic import or filesystem scanning.