Observability Design — First-Class Cross-Cutting Layer
Implementation status: Milestone 9a delivered backend/src/observability/ (log_event, record_metric, start_span) and migrated existing request-path emitters (ChatAPI, Router, ConfigLoader, ChatAgent, LLMFactory, LLMService, provider LLM calls). Milestone 9 completed instrumentation for Tools, Skills, Orchestrator, and the Context Builder, and closed the session_id gap on the Router/Orchestrator/Skill per-layer log lines flagged during Milestone 8 — every live layer now emits trace_id (Tools/Skills/Router/Orchestrator/Context Builder) and session_id (Router/Orchestrator/Skills; Tools correlate via trace_id only). No Prometheus/Grafana in Phase 1 — metrics are structured log lines on agentic.metrics.
1. Why This Is Its Own Layer
A system whose value proposition is confidence-scored, explainable routing over cost-constrained LLM calls must prove routing quality and cost behavior over time — not only return a one-off reasoning string. Observability is a named side-effect-only layer beside every other layer.
2. Position in the Architecture
API ──┐
Orchestrator ──┤
Router ──┤
Context Builder ──┤ (fallback branch only, Phase 2+)
Skills ──┼──▶ Observability (logs / traces / metrics)
Tools ──┤
LLM Service ──┘
Invariant: observability must be side-effect-only — emission failure never fails a request. Router emits before Context Builder; high-confidence traces have no Context Builder span.
3. Responsibilities
| Category | What's captured | Emitted by |
|---|---|---|
| Structured logging | trace_id, session_id, route_name, confidence, skills_invoked, tools_invoked, low_confidence, errors |
Orchestrator, Router (+ others via facade) |
| Tracing | Span per layer boundary | All layers |
| Metrics — routing | Route selection, confidence, low-confidence / fallback rates | Router |
| Metrics — tools | Latency, success/failure | Tools |
| Metrics — LLM | Tokens, cost, latency, retries, failover | LLM Service |
| Metrics — system | E2E latency, volume | Orchestrator/API |
| Prompt telemetry | prompt_version (placeholder in Phase 1) |
LLM Service |
4. Interface
Observability (backend/src/observability/):
def log_event(event_name, *, component, trace_id=None, fields=None, level=INFO) -> None
def record_metric(name, value, tags=None) -> None
def start_span(name, trace_id, *, fields=None) -> Span
INFO log format: ComponentName: event_name key=value ... (include trace_id when available).
Thin facade over core/tracing.py, core/trace_context.py, core/logging_config.py. One import for every layer — no ad hoc formats.
5. Example Metrics (conceptual — backend later)
| Metric | Why it matters here |
|---|---|
| Route selection frequency | Whether YAML coverage matches real usage |
| Average route confidence | Evidence for/against rule-based routing |
| Tool failure rate | Flaky external providers |
| LLM token / request cost | Stay inside the budget constraint |
| Low-confidence / fallback rate | Signal that keyword lists need work |
6. Relationship to Evaluation
Observability = what happened in production, continuously. Evaluation = is behavior still correct against a benchmark, on demand. Observability never scores quality; it may later feed Evaluation datasets.
7. Phase 1 Depth
- 9a (done): facade + migrate existing emitters + shared format / Cursor rule.
- 9 (done): Tools/Skills/Orchestrator instrumentation confirmed (already emitting since Milestones 4/5/8); added Context Builder emission (
build_start/build_complete,context_builder.latency_ms) andsession_idon every Router/Orchestrator/Skill event line, closing the §3 field-coverage gap. - No new metrics backend technology in Phase 1.
8. Anti-Patterns
- ❌ Each layer inventing its own log format — use the facade.
- ❌ Observability blocking or altering request execution.
- ❌ Conflating Observability with Evaluation.
- ❌ Logging response/prompt bodies or secrets at INFO.