RAG Design — Retrieval for the Fallback Branch

Implements the what behind ADR-007-rag-strategy.md (conditional, fallback-only, LangChain internally) and ADR-010-rag-memory-tech-stack.md (Chroma, OpenAI embeddings). Delivered by the RAG Foundation milestone (phases/phase2-implementation-plan.md).

1. Position in the Architecture

Unchanged from context-builder-design.md §2 — RAG is one of two things the Context Builder does on the low-confidence/no-route branch, after the Router, never before it and never on the high-confidence path. This document covers the internals of that retrieval; it does not change where or when the Context Builder is called.

LOW CONFIDENCE / NO ROUTE
        │
        ▼
Context Builder
 ├── RAG retrieval (this document)      → RequestContext.retrieved_context
 └── Memory retrieval (memory-design.md) → RequestContext.memory_items / .conversation_history
        │
        ▼
Fallback Agent Skill

2. Package Layout

backend/src/rag/
├── ingest.py          # loads raw corpus, chunks it, writes to the vector store
├── embeddings.py       # config-driven embedding provider selection (OpenAI / Ollama)
├── vectorstore.py       # Chroma client setup, collection management
└── retriever.py           # LangChain retriever wrapping the Chroma collection

config/rag/
├── ingestion.yaml       # corpus source paths, chunking parameters
├── vectorstore.yaml      # Chroma persist directory, collection name
└── embeddings.yaml         # provider ("openai" | "ollama"), model name, dimension (if truncated)

data/                        # gitignored — raw corpus + persisted Chroma index

backend/src/rag/ contains only code, following the same config/code split already established for Skills/Tools (architecture-overview.md §4) — corpus sources, chunking parameters, and store location are declared in config/rag/, not hardcoded.

3. Ingestion Pipeline

raw corpus (data/, sources TBD — see §6)
   │
   ▼
ingest.py: load → chunk → embed → write
   │            │        │        │
   │            │        │        └── Chroma collection (vectorstore.py)
   │            │        └── embeddings.py (OpenAI text-embedding-3-small, or Ollama nomic-embed-text)
   │            └── chunking strategy (config/rag/ingestion.yaml — size/overlap TBD at implementation time)
   └── config/rag/ingestion.yaml declares source paths

Ingestion is an offline/batch process, run explicitly (a script, not a request-path operation) — the same "offline, not inline" pattern the Evaluation Framework already uses (architecture-overview.md §7). No request ever triggers ingestion; the Context Builder only ever reads the already-built Chroma collection.

4. Retrieval — Wired Into the Context Builder

Milestone 1 (this document's scope) — RAG only, no memory yet:

# backend/src/agents/context_builder.py — Milestone 1, replaces PassthroughContextBuilder
class RagContextBuilder(ContextBuilder):
    async def build(self, query: str, agent_context: AgentContext) -> RequestContext:
        retrieved = await self._retriever.retrieve(query)   # rag/retriever.py, LangChain retriever internally
        return RequestContext(
            user_query=query,
            conversation_history=[],   # populated starting Milestone 4 — memory-design.md
            retrieved_context=[RetrievedItem(source=d.source, content=d.content, score=d.score) for d in retrieved],
            session_id=agent_context.session_id,
            trace_id=agent_context.trace_id,
        )

Milestone 1 does not set memory_items or user_id. Neither field exists on RequestContext yet at this point (domain-model-design.md §3 describes the target shape for Phase 2 as a whole, not the Milestone 1 state), and agent_context.user_id doesn't exist either — AgentContext only gains user_id in Milestone 3 (Auth Foundation). RequestContext.memory_items lands in Milestone 4 alongside real memory retrieval. Constructing RagContextBuilder without referencing either avoids a milestone-1 implementer hitting an attribute that doesn't exist yet.

Milestone 3 (Auth Foundation) adds RequestContext.user_id: str | None = None to the model and AgentContext.user_id to the context object; RagContextBuilder.build() is extended with one line, user_id=agent_context.user_id, populating it on the fallback branch the same way it already populates session_id/trace_id (the high-confidence branch gets it from the Orchestrator's own inline RequestContext construction — see architecture-overview.md §8).

Milestone 4 (memory-design.md §5) extends this to RagMemoryContextBuilder — same class, renamed once it also calls MemoryService.load(), adding conversation_history/memory_items population (and the RequestContext.memory_items field itself, per domain-model-design.md). No interface change: ContextBuilder.build(query, agent_context) -> RequestContext is exactly what context-builder-design.md §4 already established in Phase 1, and each milestone is a one-line wiring swap at the Orchestrator's single call site (context-builder-design.md §7) — Milestone 1 replaces PassthroughContextBuilder with RagContextBuilder, Milestone 4 replaces RagContextBuilder with RagMemoryContextBuilder. No change to the Router, Orchestrator control flow, or SkillRequest/BaseSkill.execute() contracts, at any milestone.

Retrieved documents are mapped to the existing typed RetrievedItem (domain-model-design.md §3: source, content, score) before leaving rag/retriever.py — LangChain's retriever/document types never cross the Context Builder boundary, per ADR-009-langchain-scope.md.

5. LangChain Scope (Reaffirmed)

Confined entirely to backend/src/rag/: embeddings.py uses LangChain's embedding-provider wrappers, vectorstore.py/retriever.py use LangChain's Chroma vector store and retriever interfaces. The Orchestrator, Router, and Skill Factory still never import or know LangChain exists (ADR-009). FallbackAgentSkill (skills-design.md §3a) receives the already-typed RequestContext.retrieved_context via SkillRequest — it does not call into rag/ directly.

6. Explicitly Open, Deferred to Implementation Time

Per ADR-010 §"Open Question" — these are about the production corpus, not the fixture corpus §7 needs (see below, not deferred):

Exact production corpus sources and size. Candidates (company descriptions, earnings call transcripts, analyst notes, financial glossary/definitions) are the categories ADR-007's context names as RAG's target use case, but the concrete source list, format, and volume are an implementation-time data-sourcing task, not an architecture decision.
Chunking strategy (chunk size, overlap, splitter type) — depends on the actual corpus's document shapes once sourced; a placeholder in config/rag/ingestion.yaml, not designed here.
Embedding dimension truncation — only relevant if/when Pinecone is adopted (ADR-010); no action needed while Chroma is the store.

Not deferred — a minimal test/fixture corpus is a Milestone 1 prerequisite, not a data-sourcing task. Milestone 1's exit criteria (phases/phase2-implementation-plan.md) require "low-confidence/no-route requests get real retrieved_context," and §7's unit/contract tests require "a small, checked-in test corpus" — neither is satisfiable while corpus sourcing is treated as fully open. A handful (5-10) of hand-written or lightly-adapted documents in one of ADR-007's target categories (e.g. a few company description paragraphs), checked into the repo alongside rag/ fixtures, is enough to prove ingestion → embedding → retrieval works end-to-end and to give RagContextBuilder something real to retrieve. This is the first task inside Milestone 1 itself, not a blocker on it — distinct from, and much smaller than, sourcing the eventual production corpus above.

7. Testing Approach

Unit tests: ingest.py's chunking logic against fixture documents (no network/embedding calls); vectorstore.py/retriever.py against a fixture Chroma collection built from a small, checked-in test corpus.
Contract test: RagMemoryContextBuilder.build() returns a RequestContext matching the same shape PassthroughContextBuilder did (all required fields populated), so FallbackAgentSkill and the Orchestrator need no test changes beyond exercising the new class.
Integration test (tagged, not in the default suite): a real embedding call + Chroma round-trip, mirroring the existing RUN_LLM_INTEGRATION pattern (llm-service-design.md §7).

8. Anti-Patterns Explicitly Rejected

❌ Calling rag/retriever.py from anywhere other than the Context Builder — Skills, Router, and Orchestrator never touch it directly, same rule as context-builder-design.md §8.
❌ Running ingestion inline on the request path "since the corpus is small" — ingestion is always an explicit offline step, regardless of corpus size, so request latency never depends on it.
❌ Letting a LangChain Document or retriever type leak past rag/retriever.py — always mapped to RetrievedItem before reaching RequestContext.
❌ Hardcoding the embedding provider — always resolved via config/rag/embeddings.yaml, mirroring LLMService's provider selection.