How retrieval-augmented generation (RAG) works

Retrieval-augmented generation (RAG) is a pattern for grounding a large language model in data it was never trained on. Instead of fine-tuning the model, you retrieve relevant content at query time and pass it to the model as context. The model answers from that supplied text, so its output reflects current, private information without a new training run.

What RAG stands for and why it exists

RAG stands for retrieval-augmented generation. The idea comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, presented at NeurIPS that year, which combined a sequence-to-sequence model with a dense vector index of Wikipedia. The point was simple and still holds: a model's weights are a frozen snapshot, but the world keeps moving. Retrieval hands the model a fresh, citable source at the moment it answers, which cuts hallucination and lets you update knowledge by changing the index rather than retraining.

How a RAG pipeline works

A production RAG system runs in four stages.

1. Ingest. Split source documents into chunks and store them, usually as embeddings in a vector database, alongside metadata like source and section.
2. Retrieve. Embed the user's question, run a similarity search for the closest chunks, and optionally re-rank the shortlist with a stronger model.
3. Augment. Assemble the retrieved chunks into the prompt alongside the question and any system instructions.
4. Generate. The model answers using the supplied context, ideally citing which chunks it used so the answer stays traceable.

The first stage happens offline as you index a corpus. The last three happen on every request, which is why retrieval latency and quality shape the whole experience.

Embeddings and vector search

An embedding is a numeric vector that captures the meaning of a piece of text. Similar ideas land near each other in that vector space, so "how do I reset my password" sits close to a help article about account recovery even when they share no keywords. That is the difference from classic keyword search: lexical matching looks for the same words, while vector search compares meaning.

Retrieval measures closeness with a score like cosine similarity, then uses approximate nearest neighbor (ANN) algorithms such as HNSW to search millions of vectors in milliseconds. Vector databases that ship this in production include Pinecone, Weaviate, Qdrant, Milvus, Chroma, and pgvector for teams that want to stay inside Postgres. Many strong systems run hybrid search, blending a keyword score like BM25 with dense vectors so rare terms and exact matches are not lost.

Chunking and retrieval quality

You rarely embed a whole document. You split it into chunks, and where you draw those boundaries decides what retrieval can find. Chunks that are too small scatter a single fact across several pieces; chunks that are too large bury the relevant sentence inside noise the model has to wade through.

A common starting point is chunks of roughly 200 to 1,000 tokens with about 10 to 30 percent overlap, so a fact that straddles a boundary still appears whole in at least one chunk. Structure-aware splitting, which respects headings, lists, and paragraphs, beats a blind fixed-size cut on real documents. Adding a cross-encoder re-ranker after the first search reliably lifts the precision of what actually reaches the prompt.

RAG versus fine-tuning

Here's the thing people get wrong: these are complementary tools, not rivals, and teams often use both. Fine-tuning trains new behavior into the weights, so it's the right choice for tone, format, and task-specific skills that stay stable. RAG injects knowledge at inference, so it fits facts that change and answers you need to trace to a source.

Reach for RAG when the information is dynamic, private, or large, or when provenance matters, since updating it is just a re-index. Fine-tuning earns its keep when you want the model to respond a certain way no matter what the specific facts are. A support assistant that has to sound on-brand and cite a constantly changing help center usually wants both.

Failure modes and evaluation

Retrieval quality is the whole game. Bad chunking, weak embeddings, or no re-ranking produce confident answers built on the wrong context. The recurring failure modes are retrieval misses where the right passage never surfaces, hallucination even when the context is present, a stale index that faithfully serves outdated content, and context-window limits that truncate the evidence before the model sees it.

Because these fail quietly, production RAG needs an evaluation harness, not just a vector store and a prompt. Tools like RAGAS score a pipeline on dimensions such as faithfulness, answer relevance, and context precision and recall, which lets you tell a retrieval problem apart from a generation problem instead of guessing.

Where RAG is heading

Two directions dominate current work. GraphRAG builds a knowledge graph of entities and relationships to guide retrieval, which helps with multi-hop questions that a flat similarity search struggles to answer. Agentic RAG combines RAG with ai-agents so the model decides when to retrieve, rewrites its own queries, retrieves in several passes, and checks whether it has enough evidence before it commits to an answer. Both push RAG from a single lookup toward something closer to how a careful researcher works. For a wider view of the surrounding techniques, see generative-ai.