ADR-010: RAG and Long-Term Memory — Storage and Embedding Technology Choices
Status
Accepted
Context
ADR-007-rag-strategy.md decided where RAG runs (conditional, fallback-only) and what framework implements it internally (LangChain), but explicitly deferred the vector store, embedding model, and long-term memory store choices as "open Phase 2 implementation decisions." ADR-006-memory-strategy.md decided memory's shape (two tiers, fallback-only, user_id-scoped long-term) but not its storage technology either. This ADR closes both gaps.
Deployment context that constrains the choice: single instance, requests queued, no horizontal/auto-scaling — this project does not need, and explicitly does not want, a hosted or self-managed database server. Final cloud/hosting provider is undecided and confirmed irrelevant to this decision.
Decision
| Concern | Choice | Config-driven? |
|---|---|---|
| Vector store | Chroma (embedded, local, file-persisted) | Store path via config/rag/*.yaml |
| Embeddings | OpenAI text-embedding-3-small (primary) |
Yes — same provider-selection pattern the LLM Service already uses; Ollama nomic-embed-text stays available as a free offline/CI path |
| Long-term memory store | SQLite, embedded file, accessed via SQLAlchemy models | Path via config/memory/*.yaml |
New top-level data/ directory (sibling of backend/, config/, docs/) holds the raw ingestion corpus, the persisted Chroma index, and the SQLite file — gitignored.
Rationale: Vector Store — Chroma
Embedded and file-persisted matches the single-instance deployment exactly — no server process to run, no network hop, no separate infra to provision or pay for. Pinecone (hosted, scales independently of the app instance) is a plausible later move if hosted scale is ever needed, but nothing about the current deployment model requires it, and adopting it now would add infra this project doesn't otherwise have. Not committed to, no timeline — noted here so the embedding-dimension question below has a reason to stay open rather than being resolved arbitrarily.
Alternatives considered: Pinecone — deferred, not rejected, per above. A hand-rolled flat-file/numpy similarity search — rejected; Chroma already solves this at the same operational cost (still just a local file) with a maintained LangChain integration, so hand-rolling would be undifferentiated work.
Rationale: Embeddings — OpenAI text-embedding-3-small, Config-Driven
Kept config-driven for the same reason LLM providers already are (llm-service-design.md §2, Settings.default_provider) — not a new pattern, an extension of one already in place. This keeps Ollama nomic-embed-text available as a free, fully offline path (useful for CI and for anyone running this project without an OpenAI key) without a rewrite if the default ever changes.
Dev-time cost of OpenAI embeddings is negligible at this project's scale — a rough estimate is ~$0.05 for a full ingestion pass over an estimated few-hundred-document corpus. Cost was not the deciding factor; consistency with the project's existing config-driven-provider pattern was.
Alternatives considered: Ollama nomic-embed-text as the primary — rejected as primary (kept as the fallback path) because embedding quality on OpenAI's model was preferred for the initial corpus, and the dev-time cost difference is negligible. A larger OpenAI embedding model (text-embedding-3-large) — not adopted; no evidence yet that retrieval quality on this project's corpus needs it, and it costs more per token for an unproven benefit.
Rationale: Long-Term Memory Store — SQLite via SQLAlchemy
| Option | Verdict | Why |
|---|---|---|
| SQLite + SQLAlchemy (adopted) | Adopted | Embedded, file-based, zero infra to run or pay for; SQLAlchemy models keep a migration path to Postgres/MySQL open later without needing it now; matches the single-instance deployment model exactly |
| MySQL | Ruled out | Requires a managed or self-hosted server — infra this project doesn't otherwise need at current scale |
| DuckDB | Ruled out | OLAP/columnar engine — wrong shape for memory's OLTP-style point lookups by session_id/user_id, despite also being embedded and free |
| AWS RDS / Azure Database free tiers | Ruled out | Checked and rejected — both are ~12-month new-account trials tied to that specific vendor, not free-forever or vendor-neutral |
| GCP Cloud SQL | Ruled out | No free tier at all |
No cloud-provider managed-DB free tier is both free-forever and vendor-neutral, which reinforces SQLite over any managed-DB option — on top of the explicit preference not to host or commit to a DB server at all, independent of which cloud is eventually picked.
Open Question — Explicitly Deferred
Whether to truncate OpenAI embedding dimensions up front (relevant only if Pinecone is adopted later, since Pinecone's index dimensionality is fixed at creation time and harder to change than Chroma's). Deferred to the RAG Foundation milestone (phases/phase2-implementation-plan.md) — not decidable now, since it depends on a vector-store migration that isn't committed to or scheduled.
Exact corpus sources and size for RAG ingestion — also deferred to the RAG Foundation milestone; not an architecture decision, an implementation-time data question.
Consequences
Positive:
- Zero new infrastructure cost or operational surface for either RAG or memory — both stay file-embedded, consistent with the project's single-instance, no-managed-DB constraints.
- Embedding provider stays swappable the same way LLM providers already are, without a design change if OpenAI pricing/availability ever changes.
Negative / accepted tradeoffs:
- Chroma and SQLite are both single-process, file-based stores — neither scales horizontally. Acceptable because the deployment model (§ Context) doesn't plan to scale horizontally either; revisiting this ADR is the correct response if that assumption ever changes, not a Phase 2 concern.
SQLAlchemymigration path to Postgres/MySQL is available, not built — moving off SQLite later is a real migration, not a config flip.
Revisit Trigger
If the deployment model changes to require horizontal scaling or a hosted, multi-instance-shared vector/memory store, revisit both choices together (they were evaluated under the same single-instance assumption) — not independently, and not preemptively.