What is RAG (Retrieval-Augmented Generation)? An explainer for SMEs

Key Takeaways: RAG (Retrieval-Augmented Generation) is a technique where a language model first retrieves relevant documents from your own sources (retrieval) and uses them to formulate an answer (generation). Instead of relying on what the model once learned, it bases its answer on your current, controlled information. That solves two big problems: outdated knowledge and made-up answers (hallucinations). For SMEs, RAG is the practical way to build an AI chatbot or search assistant that genuinely knows something about your products, processes and customers, without having to train an expensive model of your own.

How does RAG work?

RAG consists of two steps. First, the retrieval step: your documents are split into chunks and converted into numbers (embeddings) that capture their meaning. These go into a vector database. When someone asks a question, the system searches for the chunks that are most similar to the question.

Then the generation step. The retrieved chunks are handed to the language model together with the original question, with the instruction: answer the question based on this context. The model is therefore given the facts and does not have to draw on its memory.

The big advantage is traceability. Because you know which documents were used, you can show the source for every answer. An employee or customer can verify where the answer came from, which greatly increases trust in the system.

Why RAG instead of fine-tuning a model?

When they think of "AI with our own knowledge", many business owners imagine training their own model. In practice, RAG is almost always the better choice. Fine-tuning means reprogramming the model with your data, which takes compute, expertise and time, and has to be redone with every change.

With RAG you leave the model alone and put your knowledge into a searchable library. When a price, procedure or product specification changes, you simply replace the relevant document. From that moment on the system uses the new version, without retraining.

On top of that, RAG is more transparent. A fine-tuned model is a black box: you don't know exactly why it answers something. With RAG you see the sources used and you can trace wrong answers back to an unclear or outdated document. For an SME that wants to stay in control, that is decisive. If you want to dig deeper into the trade-off, read why build an internal knowledge base chatbot.

What can you do with it in an SME?

The most common application is an internal knowledge base chatbot. Employees ask questions about procedures, products or customer agreements and get an instant answer with a source reference, instead of searching through folders or bothering colleagues. That demonstrably saves time, especially when onboarding new people.

A second application is customer service. An AI chatbot for customer service that runs on your manuals and FAQs answers routine questions 24/7 and only escalates what genuinely needs human attention. Quotes, contracts and emails can also be made searchable and summarizable with RAG.

RAG is moreover the foundation under many AI agents in SMEs: an agent that performs actions needs reliable knowledge to draw on. Always start with a defined, valuable problem rather than "everything at once". A well-functioning knowledge base for a single team delivers results faster than an ambitious system that no one trusts.

What do you need for a RAG system?

Technically you need three building blocks: a collection of sources, a vector database to make them searchable, and a language model that formulates the answers. For the model you can use a commercial API (such as OpenAI's or Anthropic's) or an open-source LLM that runs entirely within your own environment.

More important than the technology is the quality of your sources. Outdated, contradictory or poorly structured documents lead to bad answers, no matter how good the model is. So count on a clean-up of your documentation as part of the project. That is often the work that pays off the most.

Finally, privacy is a point of attention. If you work with customer or employee data, think in advance about where your data is processed. Read how to use ChatGPT in business safely and GDPR-compliantly before putting sensitive documents into an external system. A closed-off or local setup prevents surprises.

What does it cost?

The costs break down into build and usage. A first, defined RAG system (a knowledge base chatbot for one team, with a controlled set of documents) can typically be built for an SME for 8,000 to 20,000 euros, depending on the complexity and the state of your documentation. A proof of concept to prove the value first can often start from around 5,000 euros.

On top of that there are usage costs. With a commercial model API you pay per query; for an SME that usually comes down to a few tens to a few hundred euros per month, depending on usage. With your own hosted setup, server costs replace that bill.

Good to know: building a RAG system is in many cases development work that qualifies for WBSO subsidy. That can offset a significant part of the labor costs and lower the threshold to get started.

Stratalytic and RAG

• We build RAG systems on your own documents, with a source reference for every answer, so you stay in control.
• We start with a defined, valuable problem and a proof of concept, not a multi-year project.
• We advise independently on commercial versus open-source models, with privacy and the GDPR as the starting point.
• We help you clean up and structure your documentation, because that is often where the real gain lies.
• We investigate together whether your RAG project qualifies for WBSO, so your costs go down.

Schedule a 30-min intro call

Frequently asked questions

What exactly does RAG mean?

RAG stands for Retrieval-Augmented Generation. It is a technique that combines a language model (LLM) with a search step: first, relevant documents are retrieved from your own sources, then the model formulates an answer based on those documents. This keeps the answer grounded in your current, controlled information.

What is the difference between RAG and fine-tuning a model?

With fine-tuning you adjust the model itself with new training data, which is expensive and has to be redone with every change. With RAG you leave the model untouched and add knowledge through a searchable library of sources. RAG is cheaper, faster to update, and you can trace which document an answer came from.

Do I need to put sensitive company data in the cloud for RAG?

No, you don't. You can run RAG in a closed-off environment or with open-source models that stay entirely within your own infrastructure. For SMEs that work with customer data or confidential documents, this is often the sensible choice, also with the GDPR in mind.

How many documents do you need for a usable RAG system?

There is no minimum. RAG already works with a few dozen documents, such as manuals, quotes or FAQ pages. The quality depends more on how good and current your sources are than on their number. Start small with a defined set, measure the results, and then expand in a targeted way.