How to make an AI assistant answer from your own documents (RAG explained)

How retrieval-augmented generation works, what to prepare, where projects go wrong and how to measure answer quality before launch.
4 min
09 Nov 2026
  • AI assistants
  • Guides
How to make an AI assistant answer from your own documents (RAG explained)

A general-purpose chatbot knows a lot about the world and nothing about your company. It does not know your prices, your refund policy or how your product is configured. If you ask it anyway, it may confidently invent an answer. Retrieval-augmented generation, or RAG, is the standard way to fix this: the assistant answers from your own documents, and can show which document each answer came from.

This article explains how RAG works, what you need to prepare, and where projects usually go wrong.

How RAG works, in four steps

  1. Collect. Your content is gathered from where it lives: help-center articles, PDFs, Google Docs, Notion or Confluence pages, product databases, past support tickets.
  2. Index. Each document is split into short passages. Each passage is turned into an embedding, a list of numbers that captures its meaning, and stored in a search index (a vector database). Keyword search is often added on top, because exact terms like product codes matter.
  3. Retrieve. When a user asks a question, the system finds the passages most related to it, usually five to twenty of them.
  4. Generate. The language model receives the question together with those passages and an instruction like “answer only from these sources; if the answer is not there, say so”. It writes the answer and can link to the sources.

The model itself is not retrained. That is the main practical advantage: when a document changes, you update the index, and the next answer already uses the new version.

Why not just fine-tune a model?

Fine-tuning changes how a model writes: tone, format, a narrow task. It is a poor way to teach facts that change. Facts learned in training cannot be traced back to a source, are hard to remove, and go out of date. For “answer questions about our business”, RAG is cheaper, easier to keep current and easier to check. Fine-tuning can be added later if you need a very specific style.

What you need to prepare

  • A list of sources and who owns each one. If nobody owns a document, it will go stale, and the assistant will repeat stale answers.
  • Access rules. An internal assistant must not show HR documents to everyone. Permissions from the source systems should carry over to the search index.
  • 50–100 real questions with correct answers, taken from support tickets, chats or emails. This becomes the test set used to measure quality before and after every change.
  • A decision on what happens when the assistant does not know: say so and offer a contact, create a ticket, or hand over to a person in the chat.

Where RAG projects go wrong

Messy content

Three versions of the same policy, scanned PDFs without text, key information hidden in images or tables. The assistant cannot find what the search cannot read. Cleaning up the 20% of documents that answer 80% of questions usually improves quality more than any technical change.

Bad splitting

If a document is cut in the wrong places, a passage can lose its context: a table row without its header, a step without the product it belongs to. Splitting by headings and keeping the document title with every passage solves most of this.

Retrieval finds the wrong passages

Most “wrong answers” are actually retrieval failures: the right passage was never given to the model. That is why we test retrieval separately. For each test question we check whether the correct passage is among the results, before looking at the generated answer.

No way to see what happens after launch

Log the questions, the retrieved passages and the answers (without personal data where it is not needed). Review them weekly at first. Unanswered questions show which content is missing.

How to measure quality

Before launch, run the test set and score each answer on three things:

  • Correct: does it match the reference answer?
  • Grounded: is every claim supported by the cited sources?
  • Honest: when the answer is not in the documents, does the assistant say so instead of guessing?

A language model can help with scoring at scale, but a person should check a sample by hand. Rerun the same test after every change to content, prompts or models, so you know whether a change helped.

Security and data

Questions you should get clear answers to from any vendor:

  • Which model provider processes the data, and in which region?
  • Is your data used to train their models? With business API plans from major providers it typically is not, but check the current terms.
  • Where is the search index stored, and who can access it?
  • How is personal data in documents and chat logs handled?

A realistic first version

A good first version covers one audience (customers or staff), one channel (website chat, Slack, Teams or a help-center widget), and the sources that answer most questions. It shows sources for each answer and hands over to a person when it is unsure. That is usually achievable in a few weeks, and gives real usage data for deciding what to add next.

Not sure whether you need an assistant, an automation or an agent? Read AI agent vs chatbot vs workflow automation. To discuss your documents and use case, see our AI assistants service or book a free 30-minute call.

    Book a free 30-min AI call