RAG, short for retrieval-augmented generation, is a way of building a chatbot so it looks things up in your own documents before it answers. The system searches an index of your content, pulls the passages that match the question, and hands them to the AI model to write a reply grounded in that material.
Where does the term RAG come from?
The name comes from a 2020 research paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and eleven co-authors. The paper observed that large language models store facts inside their own parameters, but have a limited ability to access and precisely use that knowledge.
The authors paired a language model with a second, searchable memory (in their case, an index of Wikipedia). They reported that the combined system produced more specific, diverse and factual language than a model working from its built-in knowledge alone. The paper also named two open problems that RAG was designed to help with: showing where an answer came from, and updating what the model knows.
Those two problems are exactly what a business cares about. You want answers that trace back to your policy, and you want them to change when the policy changes.
How does RAG work, step by step?
Vendors describe the pipeline in slightly different words, but the stages are the same. Here is the sequence, based on the AWS explanation of RAG:
- Index your content. Your documents are split into smaller passages (often called chunks). Each passage is converted into a numerical representation and stored in a searchable database.
- Retrieve. When someone asks a question, the question is converted the same way and matched against the stored passages. The closest matches come back.
- Augment the prompt. The retrieved passages are added to the question as context.
- Generate. The language model writes an answer using that context.
- Update. When documents change, the index is refreshed so the next answer uses the new version.
Anthropic sums it up in one line in its article on contextual retrieval: RAG retrieves relevant information from a knowledge base and appends it to the user’s prompt.
The model is not retrained in any of these steps. AWS calls RAG a more cost-effective way to introduce new data than retraining a foundation model. If you want to see how this fits into the bigger picture of bots that answer versus bots that take action, read AI chatbot vs AI agent.
Why does RAG reduce made-up answers?
A language model on its own answers from what it absorbed in training. AWS lists the known problems: presenting false information when it does not have the answer, giving out-of-date or generic information, and drawing on non-authoritative sources.
RAG changes the task. Instead of asking the model to recall, you ask it to read. AWS describes this as directing the model to retrieve from trusted, predetermined sources, which gives the organization more control over what comes out.
Two side benefits follow:
- Citations. Because the system knows which passages it used, the answer can point to them. People can check the source.
- Access control. AWS notes that developers can restrict retrieval of sensitive information to different authorization levels. Microsoft’s RAG overview gives the plain example: finance data should only be reachable by the finance team, even when an executive asks the chatbot.
What documents work well for RAG?
Microsoft states it directly: RAG quality depends on how you prepare content for retrieval. Content that tends to work well has these traits:
| Works well | Needs preparation first |
|---|---|
| Current FAQs, policies and procedures with one approved version | Several conflicting versions of the same policy |
| Text documents with clear headings and short sections | Scanned PDFs and images, which need text extraction |
| Product and service descriptions written in the words customers use | Internal jargon that does not match how people ask |
| Content with a named owner and a review date | Content nobody is responsible for updating |
The right column is not a dead end. Microsoft’s documentation lists tools for each case, such as automatic chunking for large documents, optical character recognition for images and PDFs, and synonym maps for terminology mismatches. It just means the work has to happen before launch.
One detail worth knowing: splitting documents into chunks can strip out context. Anthropic gives the example of a passage that says revenue grew 3% without naming the company or the period. Writing sections that stand on their own (naming the product, plan or location in each one) helps retrieval. For a full preparation checklist, see how to prepare your business data for AI.
Do you always need RAG?
Not always. Anthropic’s 2024 article notes that if a knowledge base is smaller than 200,000 tokens, you can include the entire knowledge base in the prompt and skip retrieval. A short FAQ and a handful of policies may fit. A product catalog, a help center or years of procedures usually will not, and that is where RAG earns its place.
How do you keep a RAG chatbot current?
The chatbot is as current as its index. AWS recommends updating the documents and their stored representations asynchronously, either through automated real-time processes or periodic batch processing. Microsoft describes incremental indexing as the way its search service keeps content fresh.
For a business owner, that translates into three habits:
- Give every source document an owner.
- Remove or archive retired versions so they cannot be retrieved.
- Re-index on a schedule, and immediately after any change to prices, hours, policies or terms.
What are the limits of RAG?
RAG improves answers. It does not guarantee them. The main limits:
- Retrieval can miss. OpenAI’s guide to optimizing accuracy points out that the system can supply the wrong context, in which case the model cannot possibly answer correctly.
- More context is not always better. The same guide warns that too much irrelevant context drowns out the real information and causes hallucinations.
- Questions rarely match document wording. Microsoft’s example is a user asking about a PTO policy for remote workers while the documents say “time off” and “telecommute.” Good systems combine keyword and meaning-based search to bridge that gap.
- Space is limited. Models accept a limited amount of input, so the system must return concise, highly relevant passages, not whole manuals.
- Bad content in, bad answers out. If the indexed policy is wrong or outdated, the chatbot will repeat it accurately.
OpenAI’s advice is to start with an evaluation set of questions and ground truth answers, then test changes against it. That is a practical step for any business: write down 30 to 50 real customer questions with the correct answers, and run them before launch and after every content update. Keeping a person available for the questions the bot cannot answer is part of the same discipline, as covered in what to automate in AI customer service.
The short version
RAG makes a chatbot read your approved documents before it answers, which lowers the risk of made-up replies and lets answers cite their sources. It does not retrain the model, and it does not fix weak content. Keep the source documents accurate, owned and re-indexed, and test with real questions.



