Evaluation Framework — Architectural Placeholder

1. Why AI Systems Need an Evaluation Framework (Distinct from Testing)

Traditional software testing answers: "given this input, does the code produce the expected output?" — deterministic, code-behavior-focused. AI systems need an additional discipline because two of the riskiest components in this architecture are not deterministic in the way unit tests assume:

The Router's confidence scoring is deterministic given fixed YAML, but its correctness (did it route this real-world query to the right Skill?) is a judgment call that requires a labeled dataset of "query → correct route," not a single hardcoded assertion.
The LLM Service's output is fundamentally non-deterministic — the same prompt can produce different phrasing, different correctness, or a hallucinated fact across calls or model versions.

Unit tests confirm code behaves as written. Evaluation confirms the system still behaves well against curated, representative, ideally-growing datasets — a discipline much closer to ML model validation than to conventional software QA. This project's Phase 3 pitch ("explainable financial rules") and Phase 1 pitch ("confidence-based routing") are both empirical claims — an evaluation framework is what makes them falsifiable rather than aspirational.

2. Scope of This Placeholder (Phase 1)

This is explicitly not implemented in Phase 1 — it is a structural placeholder so its eventual shape doesn't require retrofitting later, and so the architecture visibly anticipates it (a meaningful interview/portfolio signal on its own).

evaluations/
├── datasets/
│   └── (empty in Phase 1 — future: golden_routing_queries.jsonl, golden_qa_pairs.jsonl)
├── routing_eval.py       # stub: will score Router accuracy against datasets/golden_routing_queries.jsonl
├── response_eval.py      # stub: will score response quality/hallucination against datasets/golden_qa_pairs.jsonl
└── README.md              # explains intended future usage (this document, condensed)

routing_eval.py and response_eval.py contain function signatures and docstrings only in Phase 1 — no scoring logic, no CI wiring, no dataset content.

# routing_eval.py (Phase 1 stub)
def evaluate_routing(dataset_path: str) -> RoutingEvalReport:
    """
    Future: load labeled (query, expected_route) pairs, run each through
    the live Router, and report accuracy, confusion matrix, and average
    confidence for correct vs. incorrect routes.
    Not implemented in Phase 1.
    """
    raise NotImplementedError

3. Future Capabilities (Not Built Now, Architecturally Anticipated)

Capability	What it evaluates	Rough Phase
Routing evaluation datasets	Router accuracy against labeled (query → expected route) pairs	Phase 2, once real usage data exists to seed the dataset
Golden test datasets	Fixed (query → expected answer or expected key facts) pairs for regression detection	Phase 2–3
Regression testing	Re-run golden datasets on every meaningful change (prompt, model, routing config) to catch quality regressions before deploy	Phase 3
Prompt evaluation	Compare output quality across prompt template versions (ties into `LLMService`'s prompt-versioning metadata)	Phase 3
Response quality evaluation	Score coherence/relevance/completeness of Skill outputs, likely via a rubric-based LLM-as-judge approach	Phase 3–4
Hallucination detection	Flag claims in a response not traceable to retrieved context/citations — depends on Phase 2's RAG citations existing first	Phase 4
Benchmark comparisons	Compare MarketCompass's analysis quality against a fixed baseline (e.g. a simple prompt-only approach) to demonstrate the value of the Skills/Tools/Rules architecture	Phase 4, strong interview material

4. How This Supports Future Phases

Phase 2's RAG work needs a way to know retrieval is actually improving answers — response_eval.py's eventual scope is exactly that.
Phase 3's Rules Engine claims "explainable financial analysis" — evaluation is what turns "explainable" from a design intention into a checkable property (e.g. does the explanation cite the right rule/threshold).
Phase 4's Analyst Agent will chain multiple Skill calls — evaluation at that point needs to assess the whole workflow's output, not just one Skill in isolation; having the evaluations/ package and dataset convention already established means Phase 4 extends it rather than inventing it under deadline pressure.

5. Explicit Non-Goal for Phase 1