RAG vs fine-tuning: which one your LLM needs
Updated 2026-11-03
Both make a language model useful on your own material, in different ways. Retrieval-augmented generation (RAG) gives the model the right documents at the moment it answers. Fine-tuning changes the model’s weights so it behaves differently every time. Most company systems start with RAG and add fine-tuning only for specific behaviour.
What each one does
RAG. Your documents are split into passages and indexed, usually with embeddings in a vector database and often with keyword search alongside. When someone asks a question, the system finds the most relevant passages and puts them into the prompt. The model answers from them and can cite the source.
Fine-tuning. You train the model further on examples of inputs and the outputs you want. Parameter-efficient methods such as LoRA train a small set of extra weights, so this fits on modest GPUs for models up to a few tens of billions of parameters. The result is a model that follows your format, tone or task more reliably, without long instructions in every prompt.
Side by side
| RAG | Fine-tuning | |
|---|---|---|
| Adds knowledge | Yes, and it stays current when documents change | Poorly: facts learned in training go stale and are hard to update |
| Shows sources | Yes | No |
| Changes behaviour, format, style | A little, through the prompt | Yes, reliably |
| Data you need | Your documents | Hundreds to thousands of good input/output examples |
| Cost to update | Re-index changed documents | Retrain and re-evaluate |
| Access control | Filter passages by the user’s permissions | Everything trained in is available to every user |
| Typical uses | Internal search, policy and contract Q&A, support answers | Structured extraction, classification, a house writing style, a narrow domain language |
When to use RAG
- Answers must come from documents that change: policies, product data, contracts, tickets.
- People need to see where an answer came from.
- Different users may see different documents. Retrieval can respect permissions; fine-tuned weights cannot.
When to fine-tune
- The model must produce a strict format every time, such as JSON for a downstream system.
- The task is narrow and repetitive, and a smaller fine-tuned model can replace a larger general one, cutting hardware needs.
- The domain uses vocabulary or conventions the base model handles badly even with good context.
When to combine them
A common pattern for private deployments: fine-tune a mid-size open-weight model to follow your format and to use retrieved context well, then feed it passages with RAG. You get reliable output, current knowledge and sources.
Start with evaluation, not with training
Before either, build a test set of real questions or inputs with accepted answers. Run the base model with good prompts and RAG first; fine-tune only if the test set shows a gap that retrieval and prompting cannot close. The same test set then tells you whether a fine-tuned version is actually better.
For choosing the base model see open-weight models for companies; for the hardware it needs, the LLM VRAM calculator.