LLM Service Design — Platform Inference Layer (Revision 4)

Revision note: Revision 2 corrected generate() to generate(request: LLMRequest) -> LLMResponse. Revision 3 synced LLMResponse's field names (text→content, prompt_tokens→input_tokens, completion_tokens→output_tokens) to domain-model-design.md §3. Neither changed responsibilities, retry/failover policy, or Phase 1 implementation depth. Revision 4 scopes where LangChain enters this layer, per langchain-rag-router-decision.md: backend/src/llm/providers/* adopt LangChain's provider wrappers (langchain-anthropic, langchain-openai, etc.) in place of hand-rolled vendor SDK calls. This is scoped strictly inside llm/providers/* — LLMService.generate()'s external contract, BaseProvider.complete()/.stream(), and llm/factory.py are all unchanged; nothing above LLMService is affected or even aware LangChain exists. See §2a.

Implementation status (Milestone 3 + 3a, 2026-07-24): Delivered at backend/src/services/llm_service.py with LLMRequest/LLMResponse in backend/src/models/. Phase 1 token counts are estimated (chars÷4) because providers return Message without usage metadata. Cost uses backend/src/config/llm_pricing.py. Settings: llm_max_retries, llm_failover_order. providers/* use LangChain provider wrappers internally (ChatAnthropic, ChatOpenAI, ChatOllama; OpenRouter via OpenAI-compatible ChatOpenAI) — Milestone 3a; BaseProvider / llm/factory.py / LLMService contracts unchanged.

1. Why This Layer Exists

Every Skill needs to reason over data using an LLM. In the original design that dependency was modeled as LLMTool, a peer of NewsTool/SECFilingsTool. Architecture review correctly identified that as a category error: the LLM is not an interchangeable external data source, it's the platform's core reasoning capability, and it carries concerns (retries, provider failover, cost/token accounting, prompt versioning) that don't belong in the generic Tool contract. LLMService is the fix — a dedicated, always-present service layer between Skills and the existing llm/ package.

2. Position in the Architecture

Skills (CompanySkill, FinancialSkill, NewsSkill, GenericChatSkill)
   ↓  (single injected dependency, not registry-resolved)
LLM Service — backend/src/services/llm_service.py
   ↓  (existing package; factory.py/provider_catalog.py unchanged, providers/* internals migrating to LangChain — §2a)
llm/factory.py → llm/provider_catalog.py → llm/providers/{anthropic,openai,ollama,openrouter}_provider.py
   ↓
Provider SDKs / APIs (via LangChain provider wrappers, §2a)

LLMService does not replace backend/src/llm/ — that package continues to own provider-specific request/response mechanics exactly as it does today. LLMService is the policy layer on top of it: it decides which provider/model to use, how to handle failure, and what to record, then delegates the actual call to llm/factory.py.

2a. LangChain Scope (Revision 4)

Per langchain-rag-router-decision.md, this is one of exactly three places LangChain is used anywhere in the system:

llm/providers/anthropic_provider.py    → wraps langchain-anthropic
llm/providers/openai_provider.py       → wraps langchain-openai
llm/providers/ollama_provider.py       → wraps LangChain's Ollama integration
llm/providers/openrouter_provider.py   → wraps LangChain's OpenAI-compatible chat model, pointed at OpenRouter

Each provider class still implements BaseProvider.complete()/.stream() with the same messages: list[Message] signature (phase1-implementation-plan.md Milestone 0's verbatim source-review finding) — LangChain is an implementation detail inside each provider class, not a change to the provider contract llm/factory.py resolves against. LLMService.generate(), llm/factory.py, and llm/provider_catalog.py require zero changes for this migration; only the four files under llm/providers/ change internally.

Why here and not somewhere else: every other layer (Router, Orchestrator, Skill Factory, Skills, LLMService itself) already treats llm/providers/* as a black box behind BaseProvider. Swapping hand-rolled SDK calls for LangChain's equivalents inside that black box is invisible to every consumer — the textbook case for where this architecture allows a third-party framework in at all (architecture-overview.md §8, "LangChain scope").

3. Responsibilities

4. Interface

LLMService:
  async def generate(request: LLMRequest) -> LLMResponse

LLMRequest:
  prompt: str
  context: RequestContext
  model_hint: str | None = None

LLMResponse:
  content: str
  model_used: str          # e.g. "ollama:llama3.1:8b" (provider:model)
  input_tokens: int
  output_tokens: int
  estimated_cost_usd: float
  latency_ms: float
  retries_used: int
  metadata: dict           # e.g. {"prompt_version": "v1"}

Skills only ever call generate(). They never see llm/factory.py, provider names beyond an optional hint, or retry/failover mechanics — those are entirely internal to LLMService.

model_hint accepts provider:model or a bare model name (keeps the configured default provider). Failover entries in settings accept provider alone or provider:model.

5. Lifecycle

• Construction: one LLMService instance is constructed at application startup (in the same place server.py currently wires up dependencies), configured from settings.py (default provider/model, retry count, failover order, price table path).
• Injection: the Skill Registry loader passes the single LLMService instance to every Skill's constructor when instantiating skills from config/skills/registry.yaml — this is why Skills don't declare it as a named dependency the way they declare Tools; it's structurally guaranteed.
• Per-request: generate() is called zero or more times per Skill execution (most Skills call it once, after gathering Tool data); each call independently goes through the selection/retry/failover/accounting pipeline.
• Shutdown: no special teardown beyond whatever the underlying provider SDKs require (already handled by llm/factory.py today).

6. Future Extensibility

7. Testing

• Unit tests mock llm/factory.py entirely and assert LLMService's policy logic: does it retry the configured number of times, does it fail over to the secondary provider, does it compute token/cost accounting correctly from a mocked provider response.
• Integration tests (tagged, run separately) make one real call per configured provider to confirm the adapter still matches the expected LLMResponse shape — deliberately minimal, to respect the project's budget constraints.
• Skills, in turn, only ever need a fake LLMService (a simple object returning canned LLMResponses) in their own unit tests — they never need to know retries or failover exist.