Build RAG over your own docs
You have a pile of your own docs — an internal wiki, product manuals, a support knowledge base — and you want to ask questions and get answers grounded in them, with citations, instead of the model guessing. RAG (retrieval-augmented generation) fetches the passages that actually matter and feeds them to the LLM, so answers stay factual and traceable to a source.
RAG is just search plus prompt-stuffing — the clever part is retrieval, not the LLM call. The loop is deliberately dumb: chop your docs into chunks, embed each chunk into a vector store, embed the incoming question, pull the nearest chunks, and paste them into the prompt with a hard instruction to answer only from that context and cite it. Before you build any of this, sanity-check whether you even need it: if your whole corpus fits in a context window (a few hundred pages), stuff it all in and lean on prompt caching — it's simpler and more accurate. Reach for RAG when the corpus is too big for context or changes too often to re-send every time.
Step by step
- 1
Decide if you actually need RAG
Count your tokens. If the whole corpus is under ~150k tokens and fairly static, skip the vector DB entirely — put it all in the prompt and use prompt caching. Only build the retrieval pipeline below when the corpus is genuinely too big or updates constantly.
- 2
Ingest and chunk the docs
Load your sources (Markdown, PDFs, HTML) and split them into ~500-800 token chunks with a bit of overlap, splitting on structure (headings, paragraphs) rather than blind character counts. Keep metadata with each chunk: source URL, title, section, and updated-at.
- 3
Embed and store in pgvector
Run each chunk through an embedding model (text-embedding-3-small is cheap and good) and store the vector plus content and metadata in Postgres with the pgvector extension on Supabase. Add an HNSW index on the embedding column so search stays fast as you grow.
- 4
Build retrieval: hybrid search + rerank
Embed the question and pull the top ~20 chunks by cosine distance, but also run a keyword/BM25 search and merge the results — pure vector search misses exact names, IDs, and error codes. Then pass those candidates through a reranker (Cohere Rerank) and keep only the best 5-8.
- 5
Generate a grounded, cited answer
Concatenate the top chunks with numbered source labels, and instruct the model to answer using only that context, cite sources like [1], and say 'I don't know' when the answer isn't present. This one instruction is what kills most hallucination.
- 6
Build an eval set and measure
Write 20-30 real questions with their expected source documents. Score whether the right chunks show up in retrieval and whether answers are correct. You cannot tell if RAG works by vibes — this eval set is how you know which knob to turn next.
- 7
Handle updates and re-indexing
When a doc changes, re-chunk and re-embed just that document and upsert by a stable source id so you don't accumulate stale duplicates. If you ever switch embedding models, you must re-embed the entire corpus — queries and stored vectors have to come from the same model.
Starter snippet
import { embed, generateText } from 'ai';
import { openai } from '@ai-sdk/openai';
import { anthropic } from '@ai-sdk/anthropic';
import { sql } from './db'; // postgres.js client, pgvector enabled
export async function answer(question: string) {
// 1. Embed the question with the SAME model used at ingestion time
const { embedding } = await embed({
model: openai.embedding('text-embedding-3-small'),
value: question,
});
// 2. Retrieve nearest chunks (<=> is pgvector cosine distance)
const chunks = await sql`
select content, source
from doc_chunks
order by embedding <=> ${JSON.stringify(embedding)}::vector
limit 8
`;
// 3. Generate a grounded, cited answer from ONLY those chunks
const context = chunks
.map((c, i) => `[${i + 1}] ${c.source}\n${c.content}`)
.join('\n\n');
const { text } = await generateText({
model: anthropic('claude-sonnet-4-5'),
system:
'Answer using ONLY the context. Cite sources like [1]. ' +
'If the answer is not in the context, say you do not know.',
prompt: `Context:\n${context}\n\nQuestion: ${question}`,
});
return text;
}✕ Watch out for
- Chunking wrong: chunks too big drown the real answer in noise; too small and fragments lose their meaning. Start around 500-800 tokens with overlap and split on structure, not raw character counts.
- Relying on pure vector search. Embeddings are fuzzy and miss exact strings — product names, IDs, error codes, acronyms. Add keyword/BM25 (hybrid search) or you'll whiff on the literal queries users actually type.
- Mismatched embedding models. If your stored vectors and your query come from different models — or you swap models later without re-indexing — retrieval silently rots and you won't see an error, just worse answers.
- Skipping the reranker and dumping 20 marginal chunks into the prompt. It buries the answer, burns tokens, and invites the model to latch onto an irrelevant passage. Retrieve broad, rerank, keep the top handful.
- No evaluation set. Without 20-30 real questions and their expected sources, you're tuning blind and can't tell a real improvement from a lucky demo.
✓ Pro tips
- When you hand this to an AI coding agent, name the exact stack up front — 'pgvector on Supabase, Vercel AI SDK, text-embedding-3-small, no LangChain.' Left vague, agents pull in a heavy framework that abstracts away the retrieval layer you most need to see and debug. Then tell it to print the retrieved chunks and their similarity scores for a test question BEFORE it writes any generation code, so you can confirm retrieval actually surfaces the right passages first.
- Build the dumbest end-to-end version first — top-k cosine, no hybrid, no rerank — and get a real answer flowing. Only add hybrid search and reranking where your eval set shows actual misses; most people over-engineer retrieval before they've measured it.
- Store rich metadata (source URL, title, section, updated-at) on every chunk. It's what lets you produce real citations, filter by recency or product area, and re-index cleanly when a single doc changes.