Routing Design — Confidence-Based YAML Routing

The Router answers exactly one question, scoring the raw query directly (langchain-rag-router-decision.md, architecture-overview.md §8) so it never pays for Context Builder retrieval (LangChain-based from Phase 2+) on requests that don't need it — including every trivial high-confidence one. RequestContext still exists (domain-model-design.md §3); the Router is simply not one of its consumers — it computes entities internally and writes them only to RoutingDecision.matched_entities. The scoring model matches what's implemented in pattern_matcher.py (base_confidence + entity gate), plus the Phase 0 pre-scoring guard (BLOCKED_PATTERNS) and §4's route schema, aligned with the repository's current skills_routing.yaml shape (id, patterns, base_confidence). The low-confidence band is a single margin below high_confidence (§5/§6); thresholds.yaml has exactly two keys, high_confidence/fallback_margin — no mid_confidence/phase2_trigger/borderline_margin/llm_fallback_accept. §3 and §6 define the concrete RoutingDecision shape for the true no-route-matched case. §5b (Phase 2) adds a reference/comparison-cue gate: a query that matches a route's pattern and clears the entity gate can still contain a reference to something outside the query text itself (e.g. "compare AAPL's P/E to the one I asked about earlier") — without this gate, such a query would route high-confidence straight to a Skill that never sees conversation history, silently dropping the referential half of the question. The gate forces low_confidence=true on a match containing a comparison/anaphora cue, routing it to the Context Builder + memory (memory-design.md) + FallbackAgentSkill instead. This is Phase 2 scope, decided alongside ADR-006-memory-strategy.md, not a Phase 1 gap fix — Phase 1 had no memory for such a query to reach anyway.

1. Purpose

The Router answers exactly one question: given a query, which Skill(s) should handle it, and how confident is that decision? It does not execute anything. It is a pure decision function: (query: str, AgentContext, config) -> RoutingDecision.

2. Input Contract (Revision 5 — raw query, not RequestContext)

Router.route(query: str, agent_context: AgentContext) -> RoutingDecision

AgentContext (backend/src/agents/base/agent_context.py) already carries session_id and trace_id — everything the Router needs beyond the query itself for tracing/observability. It never carries retrieved_context, so there is nothing for the Router to wait on.

Why not RequestContext: RequestContext.retrieved_context is populated by the Context Builder, and per architecture-overview.md §8 the Context Builder now runs after the Router, only on the low-confidence/no-route branch — an unconditional pre-Router step would force every request, including high-confidence ones, through a component that may do real (LangChain) retrieval from Phase 2 onward. Scoring the raw query keeps the Router's cost and latency at zero regardless of Context Builder's Phase 2+ internals.

Revision 3 corrections, still in force: the Router still computes entities internally and writes them only to RoutingDecision.matched_entities (§7); it never re-derives or reads RequestContext.entities (removed in Revision 3 — see domain-model-design.md §3).

RequestContext (still populated later, by the Orchestrator, for SkillRequest — see context-builder-design.md §2 and architecture-overview.md §8's "How SkillRequest.context gets populated on each branch") is a Skill-facing contract now, not a Router-facing one.

3. Routing Decision Contract (Revision 6 — no-route-matched shape defined)

RoutingDecision:
  route_name: str | None
  skills: list[str]
  confidence: float
  matched_entities: dict
  low_confidence: bool
  reasoning: str

Three concrete outcomes (see §6 for the tier logic that produces each):

low_confidence=false in the no-route-matched row is deliberate: low_confidence means "a route was selected, but treat it cautiously" — it does not apply when nothing was selected. The Orchestrator's fallback branch (Milestone 8) is triggered by route_name is None OR low_confidence is True, not by low_confidence alone.

4. Configuration Schema (Unchanged Location and Shape)

Both files continue to live under config/routing/ — this was already correctly external in v1 and is the pattern Change 2 (centralized configuration) generalizes to Skills and Tools.

config/routing/skills_routing.yaml

Phase 1 routes in the repository today use this shape (Milestone 6 will add required_skills for multi-skill routes):

routes:
  - id: stock_quick_summary
    pipeline: stock_quick_summary          # or skill: chat — one destination per route
    patterns:
      - "quick summary"
      - "summary on ticker"
    entities:
      required: [ticker]
    base_confidence: 0.80                # assigned on pattern match; tune vs thresholds.yaml

  - id: general_chat
    skill: chat
    patterns:
      - "hello"
      - "help"
    base_confidence: 0.90

config/routing/thresholds.yaml (Revision 6 — resolved)

high_confidence: 0.70
fallback_margin: 0.30

Retired: mid_confidence, phase2_trigger, borderline_margin, llm_fallback_accept. These four keys existed in the repository's real thresholds.yaml but were never reconciled against a single spec — mid_confidence/phase2_trigger trace back to an earlier design where the Router's own mid/low bands would eventually hand off to an in-Router LLM classifier (phase2_trigger, llm_fallback_accept); that design contradicts the Router's zero-LLM-call invariant (§8, ADR-002-yaml-routing.md) and was never implemented — router.py's current "phase2" branch literally logs "Phase 2 classifier not implemented". borderline_margin was an undocumented, unused third value. Milestone 6 collapses all of this to the two-key model above; high_confidence - fallback_margin (0.70 − 0.30 = 0.40) is numerically identical to the old mid_confidence cutoff, so this is a renaming/consolidation, not a behavior change.

5. Confidence Scoring Model (Revision 4 — simplified, matches pattern_matcher.py)

Phase 1 uses a deterministic, two-step model — no weighted component arithmetic. Implemented in score_routes() (pattern_matcher.py); Milestone 6 calls it unchanged.

1. Pattern match (required). For each route, if any configured patterns entry is a case-insensitive substring of user_query, the route is a candidate. If no pattern matches, confidence = 0.0 and matched = false.
2. Assign base_confidence. On pattern match, confidence = route.base_confidence from YAML (default 0.5).
3. Entity gate. If the route declares entities.required and any required entity is missing (extracted by entities.py), cap confidence at 0.20 and set matched = false. The route is not eligible to win.
4. Winner selection. Among routes with matched = true, the highest confidence wins. Ties are resolved by sort order from score_routes().

Tuning: set each route's base_confidence relative to thresholds.yaml (high_confidence, fallback_margin) — e.g. 0.80 for a route that should clear the high-confidence tier when its pattern and entities both match, or a value between high_confidence - fallback_margin and high_confidence for a route that should only ever be selected with low_confidence=true.

Explicit non-goal for Phase 1: graded pattern-strength scoring or separate additive weights (the former Revision 3 §5 0.5/0.4/0.1 model). Substring matching is binary; decomposing it into weighted components added complexity without changing behavior.

5a. Phase 0 Pre-Scoring Guard (Repository behavior — carry forward)

Before pattern scoring, the router checks BLOCKED_PATTERNS (currently "ignore previous instructions", "jailbreak") as case-insensitive substrings of user_query. A match short-circuits to the fallback chat path with a guard trace entry. This is cheap safety behavior with zero LLM cost; Milestone 6 carries it forward as a pre-scoring step in the formalized router.

5b. Reference/Comparison Gate (Phase 2 — new)

Runs after the entity gate (§5 step 3), only on a route that would otherwise win high-confidence (matched, entities satisfied, confidence >= high_confidence). Checks user_query (case-insensitive substring match, same style as BLOCKED_PATTERNS) against a small, hand-maintained REFERENCE_CUES list — comparison words ("compare", "vs", "versus", "compared to") and anaphoric/temporal references ("the one", "that company", "the company i", "earlier", "before", "again", "previous", "last time"). A match overrides the outcome to low_confidence = true regardless of the numeric confidence computed — the route and matched_entities are still reported as-is (this is not the same as the entity gate's hard cap-to-0.20/matched=false; the route genuinely matched, it just isn't safe to execute deterministically because part of the query depends on context the Router can't see).

Why here and not elsewhere: the Router is the only component that sees the raw query text before any branching decision is made — pushing this check downstream (into a Skill, or into the Orchestrator) would mean the high-confidence path already committed to a Skill that can't handle the reference before anyone noticed. Doing it here keeps the fix a routing-confidence decision, zero LLM calls, consistent with ADR-002-yaml-routing.md's cost/latency/explainability rationale — it is pattern-matching, not classification.

Why a cue list, not general anaphora resolution: the risk case is narrow — a comparison/reference pattern paired with an otherwise-complete entity set — not "any query containing a pronoun." A hand-maintained list is tractable to reason about and extend, the same tradeoff already accepted for BLOCKED_PATTERNS.

Accepted limitation, documented rather than silent (same posture as §7's ticker false-positive risk): this list will not catch every phrasing that implicitly depends on prior context. The failure direction matters — a missed cue routes a mixed query to high-confidence anyway, which is a Skill answering confidently but incompletely, not silently corrupting data. skills-design.md §2's backstop prompt instruction (every Skill must flag, not drop, a part of the query it lacks data for) is the deliberate second layer of defense for exactly this residual case — see ADR-006-memory-strategy.md for why both layers exist together rather than relying on either alone.

6. Low-Confidence and Fallback Behavior (Revision 6 — concrete tier boundaries)

See architecture-overview.md §12/§14 for the rendered flow/sequence diagrams. Three tiers, evaluated against the winning route's confidence from score_routes() (§5):

1. confidence >= high_confidence → select the winning route. RoutingDecision.low_confidence = false.
2. high_confidence - fallback_margin <= confidence < high_confidence → select the same winning route (same route_name/skills — this is not a different route, just a lower-confidence acceptance of the same match). RoutingDecision.low_confidence = true.
3. confidence < high_confidence - fallback_margin (including the case where no route matched at all, confidence = 0.0) → no route matched. route_name = None, skills = [], low_confidence = false (per §3's table).

Phase 2 addition: §5b's reference/comparison gate can override tier 1 to tier 2 — a query that computes confidence >= high_confidence and clears the entity gate still gets low_confidence = true if it also matches a reference/comparison cue. Tier 3 is never reached this way; the gate only ever demotes an otherwise-winning match, it doesn't invalidate one.

This directly fixes the gap Milestone 0 found in the real router: today's route_request() mid-confidence branch (confidence >= mid_confidence) selects a route identically to the high-confidence branch, with no flag distinguishing the two. Tier 2 above is that same numeric band (mid_confidence was 0.40 = 0.70 - 0.30), now explicitly flagged.

What handles the low_confidence=true / no-route case differs by phase, and this is a Skill-selection change, not a Router change:

• Phase 1: the Orchestrator still calls ContextBuilder.build() on this branch (PassthroughContextBuilder, a no-op) before invoking generic_chat (GenericChatSkill) — the conditional wiring is real starting now, only the retrieval behind it is a no-op. FallbackAgentSkill doesn't exist yet, so GenericChatSkill is the Skill invoked either way.
• Phase 2+: ContextBuilder gains real LangChain retrieval, and the Skill invoked becomes FallbackAgentSkill instead of GenericChatSkill, per langchain-rag-router-decision.md and skills-design.md §3. For a query whose scoring outcome is unaffected by §5b's reference/comparison gate, the Router's own output shape (RoutingDecision.low_confidence=true or no matched route) is identical across phases — only what the Orchestrator does with that RoutingDecision changes. This does not mean the Router scores every query identically in both phases: §5b is itself a new Phase 2 scoring behavior that can turn a query which would have been low_confidence=false in Phase 1 into low_confidence=true in Phase 2 (same route, same confidence value, different flag) — see §5b and §9.

7. Entity Extraction (Unchanged scoring role; known false-positive risk flagged for Milestone 6)

Router-owned, deterministic (ticker regex + allowlist, company name lookup, simple date/period references). Not the Context Builder's responsibility — the Context Builder assembles conversational/retrieved context; the Router still owns query-local entity extraction, since that's inseparable from its own scoring logic.

Flagged, not fixed here: the real extract_entities() (entities.py) treats any bare 1–5 letter all-caps token outside a small stopword list as a ticker (e.g. "I need HELP" → ticker="HELP"). Because the entity gate (§5 step 3) directly caps confidence and clears matched based on this extraction, a false-positive ticker can wrongly satisfy an entities.required: [ticker] gate. This is a pre-existing behavior, not something Revision 6 introduces, and Milestone 0's "keep score_routes()/extract_entities() as-is" decision means it is not rewritten as part of this revision. It is called out here so Milestone 6's exit criteria explicitly decide whether to accept this risk for Phase 1 (documented, low-stakes since Tools re-validate the ticker downstream) or scope a small fix (e.g. requiring a known-ticker allowlist match rather than a bare capitalization heuristic) — either is acceptable, but it should be a deliberate call, not a silent gap.

8. Why Rule/Pattern-Based Instead of Embedding/LLM-Based Routing (Unchanged)

See ADR-002-yaml-routing.md — this decision is unaffected by the LLM Service change. The Router still makes zero LLM calls; it would be a direct contradiction of that ADR's cost/latency/explainability rationale to route through LLMService.

9. Extensibility for Later Phases (Revision 5)

Explicit non-goal, reaffirmed: the Router never gains a RequestContext or retrieved-context dependency in any phase — that would reintroduce the unconditional-retrieval cost this revision removes. If a route ever needs retrieval-informed scoring, that's a new architectural decision (see architecture-overview.md §8's deferred edge case), not an assumed future state of this design.