ADR-002: Confidence-Based YAML Routing (Rule/Pattern-Based, Not Embedding/LLM-Based)
Status
Accepted
Context
The Orchestrator needs to decide which Skill(s) handle an incoming query, with a confidence score attached so low-confidence cases can degrade safely rather than confidently doing the wrong thing. Three routing strategies were considered:
- Rule/pattern-based routing, driven by YAML (keywords, entity requirements, thresholds) — the approach already partially implemented in
backend/src/router/. - Embedding-based semantic routing — embed the query and each route's description, route to nearest neighbor.
- LLM-based routing — ask an LLM to classify the query into a route.
Decision
Use rule/pattern-based YAML routing as the sole routing mechanism for Phase 1.
Rationale
| Criterion | Rule/YAML | Embedding-based | LLM-based |
|---|---|---|---|
| Cost per request | $0, no network call | Embedding API cost + vector search | LLM call cost + latency |
| Latency | Sub-millisecond | Tens of ms (embed + search) | Hundreds of ms to seconds |
| Explainability | Full — reasoning field shows exactly which keyword/entity fired |
Partial — similarity score, not "why" | Poor unless the LLM is prompted to explain, and even then it's a rationalization, not a mechanism |
| Testability | Deterministic unit tests, no mocking of external services | Requires a fixed embedding model/version to be reproducible | Non-deterministic without temperature=0 and pinned model, still model-drift-prone |
| Interview/portfolio value | Demonstrates configuration-driven architecture, a distinct and asked-about pattern in agentic systems interviews | Demonstrates RAG-adjacent skills, but that's explicitly Phase 2's job, not Phase 1's | Demonstrates prompt engineering, not orchestration architecture |
| Budget fit | Fits the $40/month constraint with zero marginal routing cost | Adds a paid embedding call to every request | Adds a paid LLM call to every request, on top of the LLM call the Skill itself will make |
The deciding factors are explainability (this project's stated differentiator — "confidence-based routing," "explainable financial rules") and cost (routing must not itself consume LLM budget that should go to actual analysis).
Consequences
Positive:
- Zero-cost, zero-latency routing decisions.
- Fully unit-testable without network access or API keys.
- Routing behavior changes are pure YAML edits — no redeploy.
Negative / accepted tradeoffs:
- Cannot handle genuinely novel phrasings that share no keywords with configured routes. Mitigated by: (a) the always-available
generic_chatfallback route, which never fails to match, and (b) treating this as a known limitation to be revisited in a later phase with an additive embedding-based re-ranker — not a Phase 1 blocker. - Keyword lists require manual curation as new domains are added (e.g. new financial jargon). Accepted as a reasonable maintenance cost versus the alternatives' cost/latency/explainability tradeoffs.
Revisit Trigger
If false-negative routing (queries incorrectly falling through to generic_chat) becomes measurably frequent once real usage/logging exists, add an optional embedding-based confidence booster — not a replacement — that only fires when rule-based confidence is below threshold. This keeps the deterministic path as the primary, cheap mechanism.