Retrieval-Augmented Generation (RAG) Explained

A language model only knows what it absorbed during training, which means it knows nothing about your internal wiki, last week's policy update, or the contents of a customer's contract. Retrieval-augmented generation, usually shortened to RAG, closes that gap by finding relevant text at question time and handing it to the model along with the question. This guide walks through each stage of a RAG pipeline, the design choices at each step, and how these systems go wrong.

The basic idea

RAG splits the work into two jobs:

1. Retrieval: given a user's question, find the passages most likely to contain the answer.
2. Generation: give the model those passages plus the question, and ask it to answer using that material.

Picture an open-book exam. The model is a capable student who writes well, and retrieval is the process of flipping to the right pages before writing. If the right pages are open, the answer is usually good. If the wrong pages are open, even a strong student will struggle.

That analogy points to the central fact about RAG: most quality problems are retrieval problems. Teams often spend their energy tuning the prompt when the real issue is that the right passage never reached the model.

Stage one: preparing documents

Before anything can be retrieved, source documents need to be collected, cleaned, and split up.

Ingestion and cleanup

Real documents are messy. PDFs have headers, footers, and page numbers stitched into the middle of sentences. Web pages carry navigation menus and cookie banners. Spreadsheets and slides lose structure when flattened to text.

Cleanup work is unglamorous but pays off directly. Useful steps include:

• Removing repeated boilerplate such as headers, footers, and legal notices
• Preserving headings, since they carry meaning about what a passage covers
• Converting tables into a readable text form rather than a jumble of cells
• Recording metadata such as document title, section, date, owner, and access permissions

Chunking

Whole documents are usually too long to retrieve and pass to the model, so they are split into chunks. Chunk design is one of the most influential choices in a RAG system.

• Chunks that are too small lose context. A chunk reading "This limit does not apply to enterprise accounts" is useless if the limit itself is described in the previous chunk.
• Chunks that are too large dilute relevance. A long chunk that mentions the answer in one sentence among many unrelated paragraphs may not rank highly, and it wastes space in the model's input.

Common approaches include:

• Fixed-size chunks with some overlap between neighbors, so sentences near a boundary appear in both. Simple and predictable.
• Structure-aware chunks that split on headings, sections, or paragraphs. These tend to keep ideas together.
• Hierarchical chunks, where small chunks are used for matching but a larger surrounding section is passed to the model once a match is found.

A practical trick is to prepend context to each chunk, such as the document title and section heading, so a chunk still makes sense when read in isolation.

Stage two: indexing with embeddings

To find relevant chunks, the system needs a way to compare a question with thousands or millions of passages quickly. The most common method uses embeddings.

An embedding model converts a piece of text into a list of numbers that represents its meaning. Texts with similar meaning end up with similar lists of numbers, even when they use different words. "How do I cancel my subscription?" and "Steps to end a recurring plan" should land close together.

During indexing, every chunk is embedded and stored in a vector index, a data structure built to find the nearest vectors to a query quickly. When a question arrives, it is embedded with the same model, and the index returns the chunks whose vectors are closest.

Keyword search still matters

Embedding search is good at meaning but can miss exact matches. Product codes, error messages, names, and acronyms are often better handled by traditional keyword search, which scores documents by shared terms.

Many production systems use hybrid search: run both keyword and embedding retrieval, then merge the results. This covers the case where a user types an exact error string as well as the case where they describe a problem in their own words.

Stage three: retrieval

At query time, the system takes the user's input and retrieves a set of candidate chunks. Several refinements make this step more reliable.

Query rewriting

Users do not always ask clear, standalone questions. In a conversation, a follow-up like "What about for teams?" means nothing without the earlier messages. A common step is to have a model rewrite the latest message into a self-contained search query before retrieval.

Other variations include generating several alternative phrasings of a question and retrieving for each, or breaking a compound question into parts.

Metadata filtering

Metadata recorded during ingestion lets you narrow the search before ranking. Examples:

• Only search documents the current user has permission to see
• Only search the product line or region the question is about
• Prefer the most recent version of a policy

Permission filtering is not optional in most business settings. If retrieval ignores access controls, the model can quote a document the user was never supposed to see.

How many chunks to retrieve

Retrieving more candidates raises the chance that the right passage is included, but passing too many to the model adds cost, slows responses, and gives the model more irrelevant text to sort through. A common pattern is to retrieve a generous set of candidates and then narrow it down with reranking.

Stage four: reranking

Initial retrieval is optimized for speed across a large collection, so its ranking is approximate. Reranking takes the top candidates and scores each one more carefully against the question.

Rerankers typically read the question and a candidate passage together and output a relevance score. Because they look at both texts jointly, they catch distinctions that vector similarity misses, such as a passage that mentions all the right terms but answers a different question.

The usual flow looks like this:

1. Retrieve a broad set of candidates using hybrid search.
2. Rerank those candidates with a more precise model.
3. Pass only the top few to the language model.

Reranking adds latency, so teams balance the number of candidates against response-time requirements.

Stage five: grounded generation

With the best passages selected, the system builds a prompt for the language model. A typical prompt includes:

• Instructions describing the task and the rules for using sources
• The retrieved passages, clearly delimited and labeled with identifiers
• The user's question

Instructions that help keep answers grounded in the retrieved material include:

• Answer only using the provided passages.
• If the passages do not contain the answer, say so rather than guessing.
• Cite the passage identifiers that support each claim.
• If passages conflict, point out the conflict and prefer the most recent or authoritative source.

Citations are valuable beyond user trust. They let you check, automatically or by hand, whether an answer is actually supported by what was retrieved.

A useful mental check: if a knowledgeable person read only the passages you sent, could they answer the question? If not, the model cannot reliably do it either.

Where RAG systems fail

Failures cluster into a handful of patterns. Diagnosing which one you are facing is the fastest path to a fix.

Failure	What it looks like	Typical fixes
Missing content	The answer is not in the indexed documents at all	Expand sources; detect and report "no answer" cases
Retrieval miss	The answer exists but was not retrieved	Better chunking, hybrid search, query rewriting, more candidates
Ranking miss	The right chunk was retrieved but cut before reaching the model	Reranking; adjust how many passages are passed along
Lost context	The chunk is found but lacks the surrounding detail needed	Larger or hierarchical chunks; add headings to chunks
Ignored context	The right passage is in the prompt, but the model answers from memory	Stronger grounding instructions; fewer distracting passages
Stale or conflicting data	Old and new versions of a document disagree	Version metadata, recency filtering, index refresh jobs
Unsupported synthesis	The model combines passages into a claim none of them make	Require citations per claim; verify claims against sources

Two failures deserve special attention. First, a system that never says "I don't know" is almost always hiding retrieval misses behind confident guesses. Second, stale indexes creep in quietly. If documents change and the index is not rebuilt, the system keeps answering from outdated text with no visible warning.

Evaluating a RAG system

Because a RAG pipeline has several stages, evaluate the stages separately as well as end to end. Otherwise you will know answers are bad without knowing why.

Build a test set

Start with a set of realistic questions paired with the correct answer and, ideally, the document or passage that contains it. Include:

• Common, straightforward questions
• Questions that need information from more than one passage
• Questions with exact terms such as codes or names
• Questions whose answer is not in the documents at all
• Follow-up questions that depend on conversation history

Even a modest, carefully chosen set is far more useful than informal spot checks.

Measure retrieval

For each test question, check whether the known relevant passage appears in the retrieved results, and how high it ranks. If retrieval scores are poor, stop there and fix retrieval before touching the prompt.

Measure generation

For the final answers, check several properties:

• Correctness: does the answer match the reference?
• Faithfulness: is every claim supported by the retrieved passages?
• Completeness: does it cover all parts of the question?
• Appropriate refusal: does it decline when the documents do not contain the answer?

Human review is the most reliable way to judge these, but it is slow. Many teams use a language model as a grader with a clear rubric, then spot-check its judgments against human reviewers to confirm it agrees with them.

Keep evaluating after launch

Log questions, retrieved passages, and answers (with appropriate privacy controls). Look for queries with no good matches, answers that users flag, and topics that come up often but are poorly covered. Feed those findings back into the test set so each fix is protected against regressions.

Key takeaways

• RAG answers questions by retrieving relevant passages at query time and asking the model to respond using them.
• Retrieval quality usually determines answer quality, so invest in cleanup, chunking, hybrid search, and reranking before tuning prompts.
• Grounding instructions and per-claim citations make answers checkable and reduce unsupported claims.
• Apply permission filters during retrieval so users never receive content they are not allowed to see.
• Evaluate retrieval and generation separately with a realistic test set, including questions that have no answer in your documents.