Your AI assistant answers confidently, but it has never seen your price list, HR policy or that contract from March. What is RAG (retrieval-augmented generation)? It's a lookup step that lets the model find relevant passages in your own documents before answering. Instead of guessing based on training data with a cut-off date, it grounds its response in material you control.
This page is for owners and managers deciding whether they actually need that step. Or if pasting the relevant document into a prompt does the job just fine. We'll cover when RAG earns its keep, what goes wrong in practice, and exactly what to check before spending money on it.
What RAG actually means
A large language model answers from what it learned during training. That knowledge is static and has a hard cut-off date. It knows a lot about the world up to that point, but nothing about your business, your policies or your customers.
AWS makes the comparison blunt: you can think of the LLM as an over-enthusiastic new employee who refuses to stay informed with current events but will always answer every question with absolute confidence.
RAG fixes that by adding a lookup step. Before writing its answer, a system finds the most relevant passages in your own material and hands them to the model alongside the question. The model then uses that fresh context plus its training data to create the response. No retraining is needed; you are just pointing it at better information for this particular answer.
That "open-book exam" picture carries through the rest of this page. The model is still answering from memory, but now it has a textbook in front of it that belongs to your firm. If the textbook is current and well-organised, the answers get specific and checkable. If it's out of date or messy, the answers are confidently wrong in ways that are harder to spot than a pure hallucination.
How RAG works, step by step
Take the example: "What is my annual leave entitlement if I started in September?" typed into an internal assistant. You don't need to build this yourself to follow the parts; you just need to be able to ask sensible questions on a vendor call.
Chunks and embeddings
Your documents are split into passages, called chunks. No one wants to hand the model your entire 40-page HR handbook when it only needs two paragraphs about pro-rated leave. Each chunk is then converted into a list of numbers, an embedding, that captures its meaning so similar text lands near each other in a search space. Those embeddings are stored in a vector database, which is really just a very fast lookup table for "what's close to this question".
Matching the question
When someone asks the leave question, that prompt goes through the same conversion into numbers. The system then queries your vector database for chunks whose embeddings sit near it. This is where document quality bites: if your handbook has three conflicting sections on start dates, retrieval might pull the wrong one and the model will answer from that flawed passage.
Generating with context
The matched chunks plus the original question are both given to the language model. It uses that new knowledge along with its training data to write the response. A well-built system can also show you which document each part of the answer came from, so a manager can check the source in two seconds instead of trusting the bot on faith.
Keeping it current
The moment your policy changes, that old chunk is still sitting in the vector database. If nobody updates the documents and regenerates the embeddings, the assistant keeps answering from last year's rules. This is not a technical problem; it's an ownership problem. Someone has to own "when we change the doc, the system knows."
Prompt, RAG or fine-tuning: which do you need?
This question decides your budget and your timeline, so let's be straight about it. Most firms should try the prompt approach first. Add RAG only when the document set is large or changes often, and rarely need fine-tuning at all. Here is why.
Pasting into a prompt
If you have five documents that change maybe twice a year, paste the relevant ones into the context window of your AI tool and ask the question directly. You get specific answers grounded in your files without building any infrastructure. The limit is obvious: if the material doesn't fit in one go, or if it changes every week, you'll be copy-pasting forever.
RAG for scale and freshness
RAG steps in when you need to pull from a large or frequently updated set of documents. Or when answers must come from your own private files rather than general knowledge. It reduces the number of tokens you send compared with pasting everything, which saves time and cost at scale. For a firm with hundreds of customer contracts, project notes and policy pages that shift quarterly, this is usually where it pays off.
Fine-tuning changes behaviour, not knowledge
Fine-tuning changes the model itself, by training it further on your own material. RAG changes what the model can look up when it answers. IBM notes that the two are often contrasted but can be used together.
For most business owners, though, fine-tuning is the last tool you reach for. Not the first.
| Question | Prompt | RAG | Fine-tuning |
|---|---|---|---|
| What it does | Reads docs you paste in each time | Looks up relevant passages before answering | Changes how the model behaves and speaks |
| Effort to set up | None, just copy-paste | Medium: document prep, vector store, refresh process | High: data curation, training runs |
| Stays fresh? | Only if you repaste current docs | Yes, when documents are updated | No until next retrain |
| Best when | Small, stable doc set, few users | Large or changing set, many users, need citations | You want a specific voice or format consistently |
Where RAG earns its keep in practice
A few situations where the lookup step clearly helps:
- HR and onboarding. A new starter asks how to claim mileage. The assistant pulls the exact section from your policy PDF and cites it, saving them an email back-and-forth with HR.
- Customer support. You build a chatbot over existing help articles. Here, a handful of clear pages outperform a messy wiki dump; quality beats quantity for retrieval accuracy. For complex or emotional issues, though, the system still needs a human handoff path built in.
- Contract and project search. Someone asks if you agreed to 60-day payment terms with a specific client. A search across signed PDFs and email threads is faster than scrolling through three folders.
If you're building something more agentic, where the AI decides which tool to call and when, RAG is often one of those tools alongside others. Our explainer on what an AI agent is covers how lookup fits into a wider workflow without replacing human judgement for high-stakes decisions.
What can go wrong with RAG
This section is where experience separates the marketing demos from the systems people actually use in anger. RAG reduces the risk of hallucinations, but it cannot make a model error-proof. Watch out for these failure modes first:
- Bad retrieval, confident answer. If the system pulls an irrelevant chunk, you get an answer that cites real text but misses the point entirely. The user sees the citation and assumes the match is valid; they are wrong.
- Chunk size and document prep matter more than model choice. Big chunks become vague, small ones lose context. Microsoft Learn's guidance on RAG ties retrieval quality to how content is prepared.
- Stale documents. We've already covered this, but it deserves repeating because it's a silent failure. If nobody updates the policy, the assistant can keep giving outdated answers for weeks. IBM notes that knowledge bases need continual updating.
- Permissions and data exposure. Microsoft Learn gives the example of finance data that should stay visible only to the finance team, even when an executive asks the chatbot. If your RAG system doesn't respect existing access controls, you've built a very convenient data leak. Similarly, if the vector database is breached and unencrypted, attackers can reverse the embedding process and recover the original text.
- The alternative risk you're solving for. A small business owner on Reddit describes staff pasting customer data, pricing and strategy notes into public AI tools "to get help". RAG on your own infrastructure or a private instance reduces that exposure, but only if configured correctly.
If your documents live across multiple systems and nobody owns the "one source of truth" job, retrieval quality will suffer regardless of which tool you pick. A tidy, trusted data foundation underneath makes every layer above it better, including RAG.
What to check before you buy or build one
You don't need to become an engineer to run a sensible supplier call. Keep these questions in your pocket:
- Does each answer cite its source? A system that can show you the exact document and section is checkable; one that just generates text is not.
- Who can see what? Does it respect existing permissions, or does every user get access to everything indexed?
- How are documents refreshed, and who owns that process? Ask for the workflow in plain words: "When we update the PDF, does something re-index automatically, or does a human run a job?"
- Does it combine keyword search with vector search? Hybrid approaches catch both exact-term matches and semantic matches, giving better recall than either alone.
- Where is the data stored, and who can reach it? Confirm the hosting location, encryption at rest and in transit, and whether embeddings are reversible. You'll want to run this past your own GDPR or compliance lead; we're not lawyers, but the question matters.
- How will you test it before rollout? Insist on a pilot with 20 real questions your staff actually ask, measured against known-correct answers. If the vendor says "trust us," that's your exit signal.
We haven't seen pricing figures in the materials we reviewed for this page. Costs vary wildly by volume and provider; expect a range, not a single number, until you've scoped your document set.
Does this apply to you?
Probably not yet
With none or one of these, pasting the documents into your current AI tool is the right place to start.
What to do next
Pick one document set you already use every week: your HR policy, your customer FAQ, or your latest round of client contracts. Write down 20 questions staff actually ask about that material. Try the prompt approach first: paste the documents into your existing AI tool and note where it fails, misses context, or gives a generic answer instead of yours. That list of failures is your brief for whether RAG would help.
If you'd like help scoping a pilot on your own documents before committing to build anything, that's what our AI service covers. Bring the 20 questions and we'll tell you straight whether you need more than paste-and-ask.
Key terms
- embedding
- A list of numbers that represents the meaning of a piece of text, so similar passages can be matched by how close they sit.
- vector database
- A store that holds embeddings and quickly returns the passages closest in meaning to a question.
- chunk
- A passage cut from a longer document. Chunk size affects how well the right passage is found.
- fine-tuning
- Further training of a model on your own material. It changes how the model behaves rather than what it can look up.
- hallucination
- A confident answer that states something untrue as fact. Retrieval reduces these but does not remove them.
- context window
- How much text a model can read in one go. Pasting documents into a prompt is limited by it.
Frequently asked questions
Is RAG the same as fine-tuning?
Does RAG stop AI making things up?
Is it safe to put company documents into a RAG system?
Can a small team set up RAG without a developer?
How often do I need to update the documents?
Do I need RAG if I already use ChatGPT or Copilot?
Sources
- What is Retrieval-Augmented Generation?, Amazon Web Services (accessed 5 Nov 2026)
- What is retrieval augmented generation (RAG)?, IBM (accessed 5 Nov 2026)
- What is Retrieval-Augmented Generation (RAG)?, Google Cloud (accessed 5 Nov 2026)
- Retrieval-augmented generation (RAG) in Azure AI Search, Microsoft Learn (accessed 5 Nov 2026)