Public Export Pipeline Design

Implements the what behind ADR-011-public-export-pipeline.md (standalone, opt-in, LangChain-free). Full milestone plan and cross-repo context: docs/plans/public-showcase-plan.md.

1. Position in the Architecture

This pipeline is not part of the MarketCompass request path and has no relationship to the Router/Orchestrator/Context Builder/Skills/Tools layers described in architecture-overview.md. It's a separate, offline, manually-run tool that reads this repo as a data source and writes into a second, separate repo (website). It does not depend on Phase 2's backend/src/rag/ (RAG Foundation, not yet implemented) and Phase 2 does not depend on it either — see ADR-011's Alternative B for why the two stay decoupled despite both involving embeddings.

SignalFoundry (private, source of truth)
  docs/, backend/src/  ──►  allow_list.yaml (opt-in)  ──►  export.py
                                                              │
                                     ┌────────────────────────┴───────────────────────┐
                                     ▼                                                ▼
                          RAG corpus (chunks + embeddings)              static code/doc viewer HTML
                                     │                                                │
                                     └────────────────► website/src/content/showcase/ ◄┘
                                                          (committed to website's own git)

2. Package Layout

scripts/public_export/
├── export.py          # entrypoint: allow-list → read → chunk → embed → render → manifest → write
├── allow_list.py       # loads/validates config/public_export/allow_list.yaml
├── chunker.py            # header-based markdown chunking + per-file code chunking
├── render.py               # markdown → sanitized HTML (docs), Pygments syntax highlight (code)
├── embed.py                  # httpx REST call to OpenAI embeddings — no SDK, no LangChain
├── manifest.py                 # deterministic id/permalink derivation + manifest entry shapes
└── pyproject.toml                # standalone deps — httpx, pyyaml, markdown-it-py, pygments, bleach

config/public_export/
├── allow_list.yaml     # the opt-in list — see its header comment for the Rules Engine exclusion
├── chunking.yaml         # header-split thresholds
└── embeddings.yaml         # provider/model/api-key-env

website/src/content/showcase/   # generated output, committed in the website repo
├── manifest.json
├── embeddings.json
├── pygments.css
├── docs/<doc-id>.html
└── code/<mirrors source_path>.html

Same config/code split as every other layer in this repo (architecture-overview.md §4) — what's exported and how it's chunked/embedded is declared in config/public_export/, not hardcoded in export.py.

3. Chunking Strategy

Header-based, not fixed-size sliding-window — natural section boundaries stay coherent, unlike arbitrary windows.

1. Split each doc on ## (H2) boundaries first. If a resulting section still exceeds max_chunk_tokens (config/public_export/chunking.yaml), split further on ### (H3) within it.
2. Every chunk carries a heading_path (e.g. ["ADR-009", "Decision"]), prepended as light context to the text actually embedded/sent to the LLM (# ADR-009 > Decision\n\n<chunk text>), so a chunk retrieved on its own still carries enough orientation.
3. No overlap for the common case. Header boundaries are natural semantic breaks. Overlap (overlap_tokens) only applies to the fallback path — a single H2/H3 section still too large is force-split on paragraph boundaries with modest overlap, since that split point isn't semantically chosen.
4. Code defaults to one chunk per file (the allow-listed modules are small — router/skills/tools files, ~100–300 lines). Only files exceeding code_max_chunk_tokens get split, on a simple def/class/async def regex boundary — not a full AST parser, matching this pipeline's "avoid infra the corpus size doesn't need" posture.

Token counts are a rough character-based estimate (len(text) // 4), sufficient for chunk-size thresholds — this pipeline isn't optimizing token budgets precisely enough to need real tokenization.

4. Embeddings — Two Places, Deliberately

No Chroma / hosted vector DB anywhere in this pipeline or its output. ADR-010's Chroma choice fits a long-lived backend process with local disk (Phase 2's product RAG). This pipeline's output is a flat embeddings.json array consumed by a Vercel serverless function with neither — brute-force cosine similarity over a few hundred chunks is trivial at that scale and keeps the website's retrieval dependency surface at zero new packages.

5. Manifest / Permalink Contract

manifest.json ties every chunk to a stable, human-legible URL. id/permalink are derived only from source_path (manifest.py's slugify_doc_path / code_id), never from array position or export-run order:

{
  "generated_at": "2026-07-28T00:00:00Z",
  "source_commit": "<this repo's git sha at export time>",
  "docs": [
    {
      "id": "adr-009-langchain-scope",
      "source_path": "docs/adr/ADR-009-langchain-scope.md",
      "title": "ADR-009: Scope LangChain to Three Isolated Pockets",
      "doc_type": "adr",
      "permalink": "/showcase/docs/adr-009-langchain-scope",
      "chunks": [{ "chunk_id": "adr-009-langchain-scope#0", "heading_path": ["ADR-009", "Decision"] }]
    }
  ],
  "code": [
    {
      "id": "backend-src-router-config-loader-py",
      "source_path": "backend/src/router/config_loader.py",
      "permalink": "/showcase/code/backend/src/router/config_loader.py",
      "language": "python",
      "chunks": [{ "chunk_id": "backend-src-router-config-loader-py#0", "heading_path": [] }]
    }
  ]
}

Code permalinks mirror the source tree exactly, so the "this is exactly what's allow-listed, nothing more" property is visible to a reader. A file dropped from allow_list.yaml produces a visible 404 for its old permalink on the next export, not a silently reassigned id.

6. Rules Engine Exclusion — Defense in Depth

1. Structural (primary): Rules Engine content is never in allow_list.yaml — absent from the corpus entirely, not merely filtered at query time. See the file's header comment for the exact excluded paths.
2. Process: scripts/public_export/README.md and docs/plans/public-showcase-plan.md both instruct an explicit review of allow_list.yaml diffs against ADR-008/rules-engine-design.md before every export run.
3. Downstream: the website's /api/ask system prompt carries its own explicit refusal for Rules Engine questions — a second, independent layer, documented in the website repo, not duplicated here.

7. Non-Goals

• Not a general-purpose static site generator — output is consumed by the website repo's own Next.js build, not served directly by this pipeline.
• Not a replacement for or dependency of Phase 2's backend/src/rag/ — see ADR-011 Alternative B.
• No automatic/scheduled runs — export.py is invoked manually, matching this repo's existing "ingestion is offline/batch, not request-path" pattern (rag-design.md §3).