What RAG is, in plain English
Think of RAG as an open-book exam for an AI. A language model on its own answers from what it learned in training, which knows nothing about your contracts, price lists or procedures. With retrieval-augmented generation, the system first looks up the relevant passages in your own documents, hands them to the model, and asks it to answer using only what it was given.
The flow has four steps:
- Ingest. Your documents are collected, cleaned and split into searchable pieces.
- Retrieve. When someone asks a question, the system finds the most relevant pieces.
- Generate. The model writes an answer based on those pieces.
- Cite. The answer links back to the source documents so people can check it.
The citation step matters. It is what turns a plausible-sounding answer into one a person can verify.
RAG vs custom GPT vs fine-tuning
| Option | What it is | Good for | Limits |
|---|---|---|---|
| Custom GPT or project with uploaded files | A hosted assistant with a small set of documents | Small teams, small document sets, quick tests | Limited control over permissions, retrieval and data location |
| RAG system | Search over your documents combined with a language model | Large or changing document sets, citations, access control | Needs engineering and good data preparation |
| Fine-tuning | Adjusting a model's behavior with examples | Style, format and narrow task behavior | Poor at keeping facts current; does not replace search |
Most business knowledge assistants are RAG systems. Fine-tuning is occasionally added for tone or format, but it is not how you teach a model your current policies.
What a production RAG system includes
| Layer | Role | Common choices |
|---|---|---|
| Connectors | Pull documents from Drive, SharePoint, Confluence, email, CRM | Official APIs and sync jobs |
| Parsing and chunking | Extract text and tables, split into meaningful pieces | Layout-aware parsers, OCR for scans |
| Index | Store text and embeddings for search | Vector database or Postgres with a vector extension |
| Retrieval | Find relevant passages | Hybrid keyword and vector search, with a reranker |
| Model | Write the answer from retrieved passages | A hosted or private language model |
| Permissions | Restrict results to what each user may see | Syncing access control from the source system |
| Interface | Where people ask | Slack, Teams, a web app or inside your intranet |
| Evaluation and logging | Measure quality and trace answers | Test sets, feedback buttons, audit logs |
Permissions: the step most pilots skip
An assistant that answers from every document will eventually show someone something they should not see, such as salary data or a confidential contract. Fixing that after launch is painful.
- Carry the source system's permissions into the index, so retrieval filters by what the person asking can access.
- Test with accounts at different access levels before launch.
- Exclude highly sensitive repositories at first, and add them deliberately.
- Log who asked what, and which documents were used.
Data preparation: what breaks accuracy
Most disappointing assistants are data problems, not model problems.
- Out-of-date and duplicate versions. The assistant cannot tell the 2021 policy from the 2025 one unless you tell it.
- Scanned PDFs with poor text recognition.
- Tables and diagrams that lose meaning when flattened into text.
- Missing metadata such as owner, date, department and document status.
- Conflicting sources. Decide which document wins and mark the rest as archived.
- Chunks that are too small or too large, which lose context or bury the answer.
How to test accuracy before you trust it
- Collect 50 to 100 real questions staff have asked, including awkward ones.
- Write the correct answer and the source document for each.
- Score the assistant on whether it answered correctly, cited the right source and declined when the answer was not in the documents.
- Repeat after every significant change, and keep the results.
A good assistant says "I could not find that" when it should. Reward that behavior in testing.
Security risks to plan for
- Prompt injection through documents. Text inside a file can attempt to instruct the model. Treat retrieved content as untrusted and limit what the assistant can do beyond answering.
- Data leakage. Use business-tier or API access that does not train on your data, and confirm where data is processed and stored.
- Over-trust. Add a visible reminder to check cited sources for important decisions.
Timeline and cost drivers
| Phase | What happens | Typical duration |
|---|---|---|
| Discovery | Pick use case, sources and success measures | 1 – 2 weeks |
| Ingestion and retrieval baseline | Connect sources, build index, tune search | 2 – 3 weeks |
| Interface and permissions | Build the chat experience, sync access control | 2 – 3 weeks |
| Evaluation and pilot | Test, fix, roll out to a small group | 2 – 4 weeks |
Cost rises with the number of sources, data cleanliness, permission complexity and required accuracy. Indicative ranges are in our AI automation cost guide.
FAQ
What is RAG in simple terms?
Retrieval-augmented generation lets an AI answer questions using your own documents. It searches your files for relevant passages, gives them to the language model, and returns an answer with citations. This reduces made-up answers and keeps your knowledge under your control.
Is RAG better than uploading files to ChatGPT?
For small, non-sensitive document sets, uploading files can be enough. RAG is better when you have many or frequently changing documents, need permission controls, want citations, or must keep data in a specific environment. It also gives you more control over quality.
Can RAG make mistakes?
Yes. It can retrieve the wrong passage, misread a table or answer from outdated material. Good design reduces this through testing, citations and the ability to say "not found," but people should verify answers that matter.
Is my company data safe in a RAG system?
It can be, if designed properly: access controls mirrored from source systems, no training on your data, encryption, audit logs and clear vendor terms. Sensitive or regulated data needs extra review, so involve your security or legal adviser.
How long does it take to build?
A pilot over one or two document sources typically takes about 6 to 12 weeks, including testing. More sources, complex permissions and strict accuracy needs lengthen it.
Sources and further guidance
Platform features and requirements change. Check the linked primary documentation for your implementation.
TURN INSIGHT INTO ACTION
Let’s talk about your next step.
Share your workflow, your current systems and what you want to improve. We’ll help shape a practical way forward.
Discuss your project ↗