ADR-011: A Standalone, Opt-In Export Pipeline for the Public Website — Not a Fourth LangChain Pocket

Status

Accepted

Context

The company website (a separate repo, website) tells a Company → Platform → Product story (CompassFoundry Labs → SignalFoundry → MarketCompass) and wants a "how we build" showcase: a RAG-backed Q&A widget plus static code-viewer pages, grounded in this repo's real architecture docs and code. This repo (SignalFoundry, formerly agentic-ai-framework) stays private forever, including the proprietary Rules Engine (Phase 3, not yet built — rules-engine-design.md, ADR-008). Full design discussion: docs/plans/public-showcase-plan.md.

Two risks shaped this decision: (1) accidentally exposing Rules Engine content through whatever feeds the website's corpus, and (2) accidentally growing a fourth LangChain touchpoint outside the three pockets ADR-009 deliberately closed off.

Decision

A standalone tool, scripts/public_export/, with its own pyproject.toml and no dependency on backend/src or backend/scripts. It reads an opt-in allow-list (config/public_export/allow_list.yaml) — a path not listed is never exported, including files added later — and produces two outputs from one run: a chunked/embedded RAG corpus and syntax-highlighted static HTML, both written into the website repo's src/content/showcase/.

1. Opt-in, not deny-list. A deny-list must be remembered and updated every time something sensitive is added; an allow-list is safe by default — a new Rules Engine file added to this repo tomorrow is automatically excluded without anyone having to remember to add it to an exclusion rule.
2. Outside backend/src entirely — not a fourth LangChain pocket. ADR-009 confines LangChain to exactly three pockets inside the product's request path. This tool isn't part of that path at all: it's an offline, manually-run, build-time process with a different audience (public website visitors, not MarketCompass users) and a different corpus (curated docs/code, not equity-research data). It doesn't import LangChain — scripts/public_export/embed.py calls OpenAI's embeddings REST endpoint directly via httpx, which was already a project dependency.
3. Outside backend/scripts too. That directory's existing scripts (try_router.py, etc.) import the full backend package and its dependency set (fastapi, mcp, boto3, langchain-*...) to exercise the running product. Coupling the export tool there would pull in a dependency footprint that has nothing to do with exporting text files.
4. Writes files only — never commits or pushes in the target repo. Content crossing from a private repo into a second repo's git history should get a human-reviewed diff, not silent automation.

Alternatives Considered

A. Deny-list instead of allow-list Rejected: the safety property this pipeline exists to provide — Rules Engine content never leaks — depends on someone remembering to exclude it. An allow-list makes that the default instead of a maintained exception.

B. Build the export logic inside backend/src/rag/ once Phase 2 lands, reusing the product's own RAG ingestion code Rejected: Phase 2's backend/src/rag/ (per rag-design.md) serves MarketCompass's own conversational product against a Chroma index in a long-lived backend process. This pipeline serves a public marketing site, runs offline, and ships its output as a static JSON file to a Vercel-deployed Next.js app with no persistent local disk. Different audience, different runtime constraints, different freshness requirements — forcing shared code would be premature coupling between two systems that only superficially resemble each other (both happen to involve "embeddings").

C. Compute embeddings via langchain-openai's embedding wrapper, matching the LLM Service's provider-wrapper pattern Rejected: that pattern (ADR-009 pocket 1) exists because the LLM Service needs failover across multiple providers with a consistent interface. This tool has one job (call one embeddings endpoint, offline, infrequently) — a direct httpx REST call is fewer moving parts and doesn't reach for a dependency this tool doesn't otherwise need.

Consequences

Positive:

• Rules Engine content structurally cannot appear in the public corpus without someone explicitly adding a path to allow_list.yaml — the failure mode of "someone forgot to exclude it" is closed off by construction.
• ADR-009's three-pockets invariant stays exactly three; this tool is legibly outside the product's architecture, not a fuzzy fourth case.
• The export tool's dependency set (httpx, pyyaml, markdown-it-py, pygments, bleach) stays small and has no version-coupling to backend/pyproject.toml.

Negative / accepted tradeoffs:

• Two small, independent Python environments now exist in this repo (backend/, scripts/public_export/) instead of one — accepted because sharing one would either bloat the export tool's dependencies or constrain the backend's.
• The allow-list's safety property is only as good as the review discipline behind each export run (someone actually checking new entries against ADR-008/rules-engine-design.md before running it) — a process control, not something the code alone enforces. Documented explicitly in scripts/public_export/README.md and config/public_export/allow_list.yaml's header comment.

Revisit Trigger

If the public corpus grows large enough that manual allow-list curation becomes a bottleneck (unlikely at the scale this repo's docs currently represent), or if Phase 3's Rules Engine work makes it structurally harder to keep proprietary code physically separate from allow-listable code within backend/src, revisit whether an allow-list-only model still holds.