Retrieval-Augmented Generation: Architecture

Lesson 13 showed Retrieval-Augmented Generation reducing dialogue hallucination rates by 8.6x, and named it as the most effective mitigation this course covers. This lesson builds the architecture behind that number: the original research formulation, the production pipeline that turned it into an industry standard, and the vector search and reranking machinery that makes retrieving the right passage, out of millions, fast enough to matter.

RAG does not make a model smarter. It changes the question the model has to answer, from “recall this fact from memory” to “summarize what this specific retrieved document says.” The second question is much easier to get right, and much easier to audit when it goes wrong.

This lesson covers Lewis et al.’s original RAG formulation and its two variants, production chunking strategy, the two dominant approximate nearest-neighbor algorithms behind modern vector databases, and the two-stage retrieval pattern that pairs a fast, imprecise first pass with a slow, precise second pass.

Section 01The original problem RAG was built to solve

Lewis et al. (2020), in “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” identified four structural weaknesses in purely parametric language models, meaning models whose only source of knowledge is what got baked into their weights during pretraining, exactly the mechanism covered in Lesson 6.

1. Static knowledge. Updating what a parametric model knows requires expensive fine-tuning or full retraining. There is no way to simply add a new fact.
2. No provenance. A parametric model’s output carries no citation back to a source. There is nothing to audit or verify against.
3. Hallucination on long-tail entities. Facts about rare or narrow topics, underrepresented in the pretraining corpus, are exactly where the imitative-falsehood mechanism from Lesson 13 is most likely to produce a confident, wrong answer.
4. Parametric capacity bottlenecks. Compressing an enormous training corpus into a fixed-size parameter vector, θ, necessarily loses precision. Some facts simply do not survive that compression intact.

Lewis et al.’s fix combines two separate systems: a parametric memory, a pretrained sequence-to-sequence transformer (BART), and a non-parametric memory, a dense vector index built over Wikipedia passages, searched using a Dense Passage Retriever (DPR). Knowledge gets updated by adding or replacing documents in the vector index directly, with no change at all to the core model’s weights.

The query flow

A query x is passed through a dense query encoder, producing a vector representation. A Maximum Inner Product Search runs across the pre-indexed document embeddings to find the top-K candidate passages z. Those retrieved passages get fed alongside the original query x into the generator, which conditions its token generation on both.

Lewis et al. defined two distinct ways to combine retrieval with generation.

RAG-Sequence: one retrieved passage, the whole output

In RAG-Sequence, the model retrieves the top-K passages once, then uses a single fixed passage to condition generation of the entire output sequence y. The full sequence probability marginalizes over which of the K retrieved passages was actually used:

P_η(z|x) is the retrieval probability, computed from the DPR retriever’s dot-product similarity between the query and each candidate passage. P_θ(yᵢ | x, z, y_1:i−1) is the generation probability of token yᵢ given the query, the single chosen passage z, and every output token generated before it, exactly the autoregressive chain-rule structure from Lesson 6. The entire generated answer is grounded in whichever one passage was selected.

RAG-Token: a different passage for every token

RAG-Token marginalizes over the retrieved passages separately at every single output token, rather than committing to one passage for the whole answer:

This lets the generator draw on different retrieved documents for different parts of its own output. A single answer that needs to combine a fact from one document with a fact from another can do so, one token at a time, rather than being locked into whichever single passage got selected first.

Lewis et al. showed that fine-tuning both the retrieval encoder η and the generator θ jointly, end to end, produced state-of-the-art results on open-domain question answering benchmarks like Natural Questions and TriviaQA, and generated more specific, factual, and diverse text than a purely parametric baseline with no retrieval at all.

Section 02Chunking: how a document gets split before it goes into the index

Production RAG systems expand the original two-module design into a full ingestion pipeline. Before any text gets embedded and indexed, it has to be split into discrete chunks, and how that split happens has a direct, measurable effect on retrieval quality.

In practice, fixed-size chunking with a 10% to 20% overlap is the standard production baseline: a 512-token chunk paired with a 64-token overlap guarantees that any entity mention sitting near a boundary gets fully captured inside at least one embedded chunk, rather than being cut in half.

Section 03Dense vector indexing: making search fast enough to matter

Once a document is chunked, an embedding model, the same mechanism from Lesson 7, maps each chunk into a vector in ℝ^d, where d typically runs from 768 to 1,536. Brute-force cosine distance search across every stored vector, comparing a query against millions of candidates one at a time, costs O(N · d), which is unacceptable latency for a real-time system. Modern vector databases instead use Approximate Nearest Neighbor (ANN) indexing, trading a small amount of retrieval accuracy for a drastic reduction in search time, down to sub-linear, roughly O(log N), bounds.

HNSW: a multi-layer graph

Hierarchical Navigable Small World (HNSW), defined by Malkov and Yashunin (2018), builds a multi-layer graph over the indexed vectors. The top layer contains a sparse graph with long-range links, functioning like express lanes for fast, coarse navigation across the whole vector space. Lower layers get progressively denser, with the bottom layer containing the complete, fully-connected graph across every indexed vector.

Search runs top-down: it starts at the sparse top layer, performs greedy graph traversal to find a local minimum, drops down one layer to the corresponding node, and repeats until it reaches the bottom layer, where the final nearest neighbors get selected. Three parameters govern the build-time and query-time tradeoffs, documented in production vector databases such as pgvector and Weaviate:

• M, typically in the range [16, 64]: the maximum number of bidirectional connections per node. Higher M improves recall but costs more memory and slower index construction.
• efConstruction: the size of the candidate list evaluated while the index is being built, controlling the build-time precision tradeoff.
• efSearch: the size of the candidate list evaluated at query time. Higher efSearch improves recall at the cost of slower queries.

IVF: partitioning the space into clusters

Inverted File Index (IVF) takes a different approach: it partitions the vector space into Voronoi cells using k-means clustering. Every indexed vector gets assigned to its nearest cluster centroid during index construction. At query time, the index first identifies the nprobe nearest centroids to the query vector, then only scans the vectors actually belonging to those specific cells, skipping every other partition in the index entirely. IVF, especially when paired with Product Quantization for vector compression, requires significantly less RAM than HNSW, but generally produces lower recall than HNSW under high query-per-second load.

Section 04Bi-encoders vs. cross-encoders: fast and imprecise, or slow and exact

The embedding-based retrieval covered in Section 3 relies entirely on bi-encoder models: a query q and a passage p get mapped independently into vector representations, v_q = E(q) and v_p = E(p), and similarity is scored as a simple dot product:

Because a passage’s embedding can be precomputed once and stored in the index, comparing it against a new query at search time costs only O(1), the fast lookup that makes the ANN search in Section 3 possible at all. The cost of that speed is that a bi-encoder never lets the query and the passage actually interact during encoding. Each gets projected into vector space in complete isolation from the other, with no token-level attention crossing between them. This works fine for simple topical matching, but it degrades badly on queries involving negation, precise conditions, or specific logical structure, exactly the kind of query where “similar words” and “actually answers the question” diverge.

Nogueira and Cho (2019), in “Passage Re-ranking with BERT,” established the fix: a cross-encoder. Instead of encoding the query and passage separately, a cross-encoder concatenates them into a single input, [CLS] ∘ q ∘ [SEP] ∘ p, and passes that combined sequence through every transformer self-attention layer together. Every token in the query gets to directly attend to every token in the passage, and vice versa, which is exactly the full self-attention mechanism from Lesson 4, just applied jointly across query and passage instead of within a single sequence.

Cross-encoders are meaningfully more accurate than bi-encoders, because they can actually reason about the specific relationship between a query and a passage rather than comparing two independently-computed vectors. But that accuracy has a cost: every single query-passage pair requires a full transformer forward pass. Running a cross-encoder across an entire corpus of millions of documents, for every query, is computationally intractable.

The two-stage production pattern

Production systems resolve this tradeoff by combining both approaches in sequence, a pattern documented across systems like Cohere Rerank and OpenSearch’s reranking pipelines.

1. Stage 1, bi-encoder plus HNSW index. High-recall, low-precision candidate generation. Retrieves the top-K passages, commonly K = 100, out of a corpus of millions, in well under 10 milliseconds.
2. Stage 2, cross-encoder reranking. High-precision filtering. The much smaller set of K candidates from Stage 1 gets reranked using full query-passage cross-attention, producing a final top-k selection, commonly k = 5, that actually gets inserted into the model’s prompt context.

This pattern exists because a cross-encoder’s cost only becomes tractable once the candidate pool has already been narrowed down to a small handful of documents. Running the fast, imprecise bi-encoder first to cut millions of candidates down to a hundred, then running the slow, precise cross-encoder only on that final hundred, gets both the speed of vector search and the accuracy of full cross-attention, without paying the cost of running a cross-encoder against the entire corpus.

Conclusion

Every piece of this architecture exists to solve one of the same four structural weaknesses Lewis et al. identified in a purely parametric model. Splitting knowledge out into a searchable, updatable vector index solves the static-knowledge and provenance problems directly: new facts get added by inserting new chunks, and every generated claim can be traced back to whichever chunk it came from. RAG-Sequence and RAG-Token give two different ways to actually condition generation on what gets retrieved. Chunking strategy determines how cleanly a fact survives the process of being cut out of its original document. HNSW and IVF make searching millions of vectors fast enough to run inside a real-time request. And the two-stage bi-encoder-then-cross-encoder pattern is the direct engineering answer to a tradeoff this lesson derived explicitly: a fast retrieval method that is not precise enough, paired with a precise method that is not fast enough, combined so that each one only has to do the part of the job it is actually good at.

None of this guarantees a correct final answer. A perfectly retrieved, perfectly relevant chunk can still get ignored by the model that receives it, or a chunk boundary can split a fact in half before it is ever indexed. The next lesson covers exactly where this architecture breaks down in practice, how to measure that breakage with real metrics, and what the OWASP Top 10 for LLM Applications identifies as the specific security risks that come with storing knowledge in a vector index instead of model weights.