Every company that adopts large language models eventually asks the same question: "how do we make it know *our* stuff?" There are two main answers — retrieval-augmented generation (RAG) and fine-tuning — and they are not interchangeable.

What each one actually does

  • RAG leaves the model unchanged. At question time, you search your own documents, pass the most relevant passages into the prompt, and the model answers from them.
  • Fine-tuning changes the model itself by training it further on examples, so certain behaviours or styles become built in.

A useful shorthand: RAG changes what the model knows right now; fine-tuning changes how the model behaves by default.

When RAG is the right tool

  • Your knowledge changes often — prices, policies, product docs, tickets.
  • You need answers with sources users can check.
  • Different users may see different documents, so access control matters.
  • You want results this week, not after a training project.

When fine-tuning earns its cost

  • You need a consistent format or tone the model keeps drifting from.
  • The task is narrow and repetitive, like classifying tickets or extracting fields.
  • You want a smaller, cheaper model to match a bigger one on one specific job.
  • Latency matters and long prompts full of context are too slow.

What a production RAG pipeline looks like

  1. Ingest — split documents into chunks that keep meaning intact (sections, not arbitrary character counts).
  2. Embed — turn each chunk into a vector and store it in a vector database such as Pinecone, Qdrant, Weaviate or Postgres with pgvector.
  3. Retrieve — combine vector search with keyword search; hybrid retrieval catches exact names and codes that pure semantic search misses.
  4. Rerank — reorder the top results so the best evidence comes first.
  5. Generate — answer only from the retrieved context, and cite it.
  6. Evaluate — keep a test set of real questions and measure accuracy on every change.
Most "the AI made it up" problems are retrieval problems. Fix what goes into the prompt before you blame the model.

The best answer is often both

Many mature systems use RAG for knowledge and a light fine-tune for behaviour — for example, a model tuned to always answer in your support team's structure, grounded by RAG in today's help-centre articles.

Where to start

Start with RAG, a small evaluation set of 50–100 real questions, and honest measurement. Only reach for fine-tuning when the evaluations show a behaviour problem that better prompts and better retrieval can't fix. That order saves most teams months of work.