What Is RAG in AI? Retrieval Augmented Generation Explained Without the Hype

The most common search that puts this post in front of people is "rag meaning". 330 impressions in the last 16 months, average position 81, zero clicks. Those people wanted a dictionary. Google offered them a machine-translated essay about AI, and nobody took the deal.

They were right. The old version called RAG the future of AI and explained it with a sustainable fashion blog that never existed. So I threw it out. If you came for the cloth, wrong aisle, sorry. If you want to know what people mean by RAG in AI and whether you need it, keep reading.

RAG stands for retrieval augmented generation. Before the AI answers, your system searches a pile of documents, pulls out the few passages that look relevant, and pastes them into the prompt next to the question. The model answers from what it was just handed, instead of from what it remembers from training.

It's an open-book exam. The term comes from a 2020 paper led by Patrick Lewis, and the basic idea hasn't moved much since. What has moved is how often you need it, and most explainers skip that part.

How Retrieval Augmented Generation Works, Using One Refund Question

Picture a made-up online shop with 400 help articles. A customer asks the chat widget: "Can I return a sale item after 30 days?" Four things happen, and the first one happens long before anyone asks.

1. Indexing. The articles get cut into chunks of a few paragraphs. Each chunk goes through an embedding model, which turns text into a long list of numbers that stands for its meaning. Those numbers go into a vector database.
2. Retrieval. The question gets the same treatment. The database returns the chunks whose numbers sit closest, so "return a sale item" can find a paragraph called "Refunds on reduced goods" even though the words don't match.
3. Augmentation. Those chunks get pasted into the prompt, along with an instruction like "answer only from the text below."
4. Generation. The model writes the answer, ideally with a link to the article it used.

The model itself never changes. There's no training step. If the refund policy changes on Tuesday, you update one article, re-index it, and the next customer gets the new answer. That's why RAG caught on: editing a document is cheap and retraining a model is not.

Most serious setups also run plain keyword search next to the vectors, because embeddings are weak at exact strings like part numbers and error codes. You'll see this called hybrid search.

RAG vs. a Plain LLM, and Where ChatGPT Fits

ChatGPT with web search off answers from memory. Turn search on and it's doing RAG, with the internet as the document pile. I've written about how to see what ChatGPT actually searches for, and Google's AI Overviews work the same way: retrieve from the search index, then write.

A plain LLM only has what went in during training. That stops at a cutoff date, and it never included your contracts or your price list. RAG gives it a way to look things up at the moment of the question.

Fine-tuning is the other term that gets thrown around here. It changes how a model behaves, like its tone or output format. It's a poor way to teach facts that change every month. So when a vendor tells you their AI is "trained on your data", ask what that means. Usually it's RAG, and that's fine, but it's one of the questions worth asking before you pay for an AI tool.

RAG Does Not Stop Hallucinations

In 2024, Stanford researchers tested legal research tools built on RAG, the kind some vendors said would eliminate hallucinations. Lexis+ AI and two Thomson Reuters tools each hallucinated between 17% and 33% of the time. That's better than a general chatbot like GPT-4, and a long way from eliminated. Those tools had curated case law to search through, which is a cleaner pile than most companies will ever have.

There are two separate ways this breaks. Retrieval can miss, so the right passage never reaches the model and it answers from whatever it did get. Or retrieval works and the model misreads the passage, or blends two of them into something neither says.

The first one you can measure. In Anthropic's retrieval tests, a standard setup failed to pull the right chunk into the top 20 results 5.7% of the time. Adding context to each chunk, keyword search and a reranking step brought that down to 1.9%. Better, and still not zero.

Then there's the failure nobody can blame on the model. If the PDF in your pile is from 2021, RAG will quote it faithfully and with total confidence. Someone has to own the documents.

When You Don't Need RAG at All

I counted this blog. 190 published posts, about 325,000 words, roughly 460,000 tokens. Anthropic's own advice is that below 200,000 tokens you "can just include the entire knowledge base in the prompt". I'm over that line, but the whole blog would fit into a 1 million token context window. And when my agent needs related posts for internal links, it reads a plain JSON list of titles and slugs, without embeddings or a vector database anywhere near it.

So check the size of your pile before you build anything. A 40-page handbook doesn't need a retrieval pipeline.

Long prompts have their own bill though. Qdrant, who sell a vector database, so read it with that in mind, worked out the input cost at October 2026 prices: about $0.01 per query for RAG at 5,000 tokens, $0.40 for a 200,000 token prompt, $0.02 if that prompt hits the cache. They also collect the research on models getting worse as prompts grow. Chroma tested 18 models and found that "performance grows increasingly unreliable as input length grows." Chroma sells a vector database too, so both of them like this conclusion.

The other thing eating classic RAG is agentic search. Claude Code is my daily driver, and it doesn't work from an embedding index of my code. Boris Cherny, who created it, said early versions tried RAG with a local vector database and the team didn't keep it. Instead the model explores like a developer would: it reads a file, looks at the imports, then reads those files. After each search it reads the results and decides what to look for next.

Retrieval didn't go away there. It turned into a tool the agent decides to call, often through MCP.

Where RAG Still Earns Its Place

Claude's own Projects feature switches RAG on automatically when your project knowledge approaches the context window limit. If the company selling million token context windows still needs retrieval past a certain size, so will you.

RAG makes sense when at least one of these is true:

• The pile is too big. Years of support tickets won't fit in a prompt.
• It changes constantly. Prices and stock go stale faster than you'd want to rebuild a giant prompt.
• Different people may see different things. Retrieval can filter by permissions before the model sees anything. A giant prompt can't.
• You have to show your source. A support answer or a legal one needs a link to the passage it came from.

If you're in Germany and the data can't leave your servers, the retrieval part is the easy bit. Embedding models run fine locally, and the harder job is the state of your files.

If You're Going to Build One

One tool I built, AI Internal Linker, does the retrieval half of this. It exports a WordPress site's content, runs it through OpenAI embeddings, stores the vectors in ChromaDB and clusters them to work out which posts belong together. It's a linking tool, not a customer-facing chatbot, so take the rest of this section as reasoning rather than a measurement.

Write the test first. Take 30 real questions from your inbox and write down the correct answer to each. Without that list you can't tell whether a change helped, and you'll tune the thing by vibes.

Then use the most boring tool that passes the test. OpenAI's hosted file search does semantic and keyword search with no infrastructure on your side. If you already run Postgres, pgvector keeps your vectors next to the rest of your data. Dedicated databases like Pinecone or Qdrant come after you've outgrown those. A framework like LangChain is optional, and I'd leave it out until you can name what it saves you.

Show the sources in the answer, and tell the model to say it doesn't know when the passages don't cover the question. A bot that admits a gap costs you one support ticket. A bot that invents a refund policy costs more.

What I Don't Know Yet

RAG is a lookup step in front of a model. That idea isn't going anywhere, because models will always be missing your latest data. The classic version, with its chunks and fixed pipeline, is less certain. Context windows keep growing and agents keep getting better at searching. I can't tell you where the line will sit in two years.

If you're trying to work out whether your documents need RAG or just a cleanup, that's the kind of question my AI consulting is for. I can't promise you a bot that's never wrong. What I can offer is a straight answer on whether you need one at all, and the honest flowchart might save you the call. Otherwise, let's talk.