RAG vs Fine-Tuning vs Prompt Engineering

They fix different problems: prompting changes what you ask, RAG changes what the model knows, fine-tuning changes how it behaves. Research shows RAG consistently beats fine-tuning for knowledge — start cheap and escalate on evidence.

Reviewed

The three approaches get compared constantly and the comparisons are usually framed as a competition. They are not competing. They fix different problems, and most of the confusion comes from teams reaching for the expensive one to solve something the cheap one already handles.

The one-line version:

  • Prompt engineering changes what you ask.
  • RAG changes what the model knows at the moment you ask.
  • Fine-tuning changes how the model behaves by default.

If you take nothing else: you almost never need fine-tuning for knowledge, and that is the mistake teams make most often.

Start with the actual symptom

Before choosing a method, name what is going wrong. The three failures look similar from a distance and have completely different fixes.

The model doesn't know something. It gives a confidently wrong answer about your product, your policies, your customers, or anything that happened after its training cutoff. This is a knowledge problem. It is a RAG problem.

The model knows, but won't do it consistently. It gets the facts right and the format wrong, drifts out of your tone, ignores an edge case one time in ten, or produces four paragraphs when you asked for a bullet. This is a behaviour problem, and prompt engineering fixes the overwhelming majority of these. Fine-tuning is for what's left.

The model has the information and still reasons badly. It has everything it needs in context and draws the wrong conclusion. Neither RAG nor fine-tuning helps much. That is a prompting and decomposition problem — see chain-of-thought and advanced reasoning prompting.

Skipping this step is how teams end up fine-tuning to fix a formatting inconsistency, at roughly a thousand times the cost of writing a better prompt.

Prompt engineering: always first, and not because it's a compromise

Prompt engineering operates entirely at inference. The model is unchanged; you are changing the input. Iteration is measured in minutes, cost is roughly zero, and you can revert instantly.

That last property matters more than it sounds. A prompt change is a config change. A fine-tune is an artifact you have to version, evaluate, host, and re-do every time the base model updates.

Practically, this means writing genuinely structured prompts rather than a sentence and a hope — see the anatomy of a great prompt, system prompts vs user prompts, and few-shot vs zero-shot. Few-shot examples in particular do a surprising amount of what people expect fine-tuning to do: three good examples of your desired output format will fix most format drift, immediately, with no training run.

Escalate only when you have a prompt you have actually iterated on and measured, and it is still failing.

RAG: when the problem is knowledge

Retrieval-augmented generation was introduced by Patrick Lewis and colleagues at NeurIPS 2020 — models that combine pre-trained parametric and non-parametric memory for language generation. In practice: retrieve relevant documents at query time, put them in the context, and let the model answer from them.

Use it when the answer depends on information the model cannot have — your internal documentation, a customer's account history, this quarter's numbers, anything that changes.

RAG's advantages over training the knowledge in are practical rather than theoretical. Updating a document updates the answer immediately, with no retraining. You can cite the source, which matters enormously for trust and for verification. You can enforce access control at retrieval time, so a user only ever gets context they are allowed to see — something a fine-tuned model fundamentally cannot do, because the knowledge is baked into weights everyone shares.

The cost is latency, retrieval infrastructure, and a new failure mode: retrieve the wrong documents and the model will answer fluently from them. Most disappointing RAG systems are failing at retrieval, not generation.

RAG vs tools vs long context covers the narrower architectural question of how to get information into context, which is downstream of this one.

Fine-tuning: when the problem is behaviour

Fine-tuning continues training a base model on your examples, adjusting the weights. It is genuinely good at a specific set of things: consistent output in a proprietary format, a domain style or vocabulary the base model handles awkwardly, a narrow classification task where a smaller fine-tuned model beats a larger prompted one on cost and latency, and compressing a very long system prompt into the weights.

What it is not good at is teaching the model new facts — and this is where the evidence is unusually clear.

In Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (December 2023), Oded Ovadia, Menachem Brief, Moshik Mishaeli and Oren Elisha compared the two directly on knowledge-intensive tasks. RAG consistently outperformed unsupervised fine-tuning — both for information the models had encountered during pre-training and for entirely new knowledge. They also found that models struggle to learn new factual information through unsupervised fine-tuning at all, though exposure to many variations of the same fact during training helps.

Which is worth restating, because it contradicts the most common intuition about fine-tuning: training your documents into a model is a worse way to make it know your documents than just retrieving them. You pay more, you wait longer, you cannot cite anything, you cannot enforce permissions, and you get worse accuracy.

The other costs are ongoing. A fine-tune is pinned to a base model; when a better one ships, you re-do the work. You need a genuinely good dataset — typically hundreds to thousands of high-quality examples, and dataset quality dominates everything else. And you have to keep evaluating it, because fine-tuning can degrade general capability while improving the narrow task.

How to actually decide

Work down the list and stop at the first one that fixes it:

  1. Write a better prompt. Structure, explicit constraints, a defined output format. Measure it.
  2. Add few-shot examples. Especially for format and tone. Still measuring.
  3. Add retrieval if the failures are the model not knowing things.
  4. Add tools if it needs to look things up or act, rather than read.
  5. Fine-tune if, after all of the above, you still have a consistent behavioural gap — or the economics of a smaller model make it worth it.

Most production systems end up combining them: a fine-tuned or well-prompted model, with retrieval for current knowledge, inside a carefully engineered context. These are layers, not alternatives.

The step everyone skips is the measurement between them. Without an eval set you cannot tell whether the prompt change helped, which means you cannot tell whether you need the next step at all — so you escalate on vibes and end up fine-tuning a problem that a three-line format instruction would have solved. How to write evals for your AI agent and how to evaluate and test your prompts are the prerequisite, not the follow-up.

A note on cost

Rough orders of magnitude, because the exact numbers change monthly and anyone quoting precise figures is quoting stale ones. Prompt engineering costs your time and nothing else. RAG costs embedding and storage, plus more input tokens per call — prompt caching recovers a lot of that when your retrieved context is stable. Fine-tuning costs dataset preparation (usually the largest and most underestimated line), the training run, and often a higher per-token rate for serving a custom model.

The asymmetry is the point: the cheapest option is also the fastest to try and the easiest to undo. There is no reason to skip it.

Sources

← Back to Articles How we research & review →