RAG — retrieval-augmented generation — is a technique where a system retrieves relevant documents at question time and puts them into the model’s prompt, so the answer is grounded in your content rather than in whatever the model absorbed during training.
The problem it solves
A language model on its own answers from its parameters. That knowledge is frozen at training time, knows nothing about your internal documents, and degrades gracelessly: when the model doesn’t know something, it often produces a fluent, confident, wrong answer. Fine-tuning is expensive, slow to update, and bad at recalling specific facts. RAG takes a different route — leave the model alone and change what it reads before answering.
How the pipeline works
A typical RAG system has an indexing side and a query side.
Indexing. Documents are split into chunks — passages small enough to fit several into a prompt, large enough to carry meaning. Each chunk is run through an embedding model to produce a vector embedding, and the vectors are stored in a vector database alongside the chunk text and its metadata (source, date, access permissions).
Query. The user’s question is embedded with the same model, and the store runs a similarity search — usually via an approximate nearest neighbor (ANN) index — to find the chunks closest to the question in vector space. The top results are assembled into the prompt along with the question, and the model writes an answer from them, ideally citing which chunk supported which claim.
Production systems add refinements: hybrid search that combines vector similarity with keyword matching, a reranking step that reorders candidates with a more accurate model, and permission filters so retrieval respects who’s asking.
What it doesn’t fix
RAG bounds hallucination; it doesn’t eliminate it. The model can still misread a retrieved passage, blend two sources, or answer from its parameters when retrieval comes back thin. And the ceiling is retrieval quality: if the right passage isn’t found — because the chunking split it awkwardly, or the question’s wording is far from the document’s — the model can’t be grounded in it. Most disappointing RAG systems are retrieval problems wearing a generation costume.
That makes measurement unusually important for a technique this fashionable. Teams that run RAG seriously maintain evaluation sets — questions with known-good answers — and track retrieval hit rates, eval pass rates, token costs, and latency over time.
Measuring RAG systems
Those measurements are ordinary analytics. LLM observability tools like Langfuse, LangSmith, and Helicone record traces of every retrieval and generation step into databases such as ClickHouse or Postgres, and a BI tool like Metabase can query those tables directly — dashboards for cost per query, latency percentiles, and eval pass rate sit alongside the rest of your reporting.
Related terms
Put it to work
- LLM analytics — Overview
- LLM observability — Integrations
- Eval pass rate — Metric
- Token usage — Metric
