RAG Explained: What It Is and When You Actually Need It

Your roadmap has an AI feature on it. A support assistant that answers from your help docs. A search box that understands what a customer means, not just the words they typed. A “summarize this case” button. Somewhere in the planning, someone said the word “RAG,” and now you are trying to work out whether you need it, what it costs, and what breaks.

This post answers that. Not how to build it (that is a separate, hands-on post), and not which vector database wins a benchmark. The question here is the one whoever owns this feature has to answer first: do you need RAG at all, and if so, what are you signing up for?

RAG in one analogy

A plain language model is a smart student taking a closed-book exam. It is good at reasoning and it memorized a lot, but it can only answer from what is already in its head, and its memory is frozen at the day its training stopped.

RAG (Retrieval-Augmented Generation) hands that same student the textbook and their own notes, and tells them to look up the relevant page before they answer. Same brain, better answer, because it is grounded in the actual source material instead of memory. That is the whole idea. Everything else is plumbing.

Mechanically it is two steps: retrieve (find the relevant text from your documents) and generate (write the answer using that text). The quality of a RAG feature lives almost entirely in the first step. If retrieval hands the model the wrong page, no model on earth saves the answer.

You need three things about it clear before you commit. The rest is implementation detail.

1. The model answers from text you give it at question time, not only from what it learned in training.
2. You have to find the right text first, and that finding is where quality is won or lost.
3. It lets you show citations and update knowledge instantly, without retraining anything.

Embedding dimensions, cosine versus dot-product, HNSW versus IVFFlat, chunk sizes, rerankers: all real, all implementation-time detail, none of it something you need to settle before deciding whether to build.

The three problems it solves, in your product’s terms

The standard framing (from AWS, IBM, and the original 2020 research) is that RAG fixes three things without retraining the model.

Made-up answers (hallucinations). A raw model will confidently invent a clause number, a dosage, or a coverage limit that sounds right and is wrong. Ground it in your retrieved documents and it quotes the real source instead of guessing, and your interface can link to the exact page. For an InsurTech assistant, that is the difference between “your limit is around $50,000” and quoting the actual policy PDF.

Stale knowledge. The model’s memory stops at its training date. Your product data changes daily. A field-service SaaS whose price book and standard procedures change every week cannot wait for a model retrain. Retrieval always reflects today’s version, because it reads today’s documents.

Private, per-customer data. The model was never trained on tenant A’s contracts, tickets, or inspection reports, and never will be. Retrieval is the only way it ever sees them. This is the core case for industry-specific software: “ask your data” over a customer’s own documents. It is also, as we will get to, where the real risk lives.

Do you actually need RAG?

Here is the part most articles skip, because most articles are written by people who sell RAG. A lot of teams reach for it too early.

Start with the cheapest thing that could work and only escalate when it provably fails. In practice:

You probably do NOT need RAG yet if:

• Your knowledge fits in a prompt. A small, stable set of documents can go straight into the system prompt or a long context window. No retrieval, no vector store, no pipeline.
• It is general knowledge the model already has.
• The corpus is small and maintained by a few trusted people (an internal wiki, your SOPs, a playbook). A structured file plus a long context beats a vector pipeline here.
• You have not yet proven that a prompt-only approach fails your accuracy target. Build the pipeline when the simple version misses, not before.

You DO need RAG when the answer depends on a body of text that is:

• Large, too big to paste into every prompt (thousands of documents, or more than a few hundred thousand tokens), or
• Changing faster than you would ever retrain or redeploy, or
• Private or per-tenant, your customers’ own documents the model has never seen, or
• Auditable, where you must show which source an answer came from, for compliance or just for trust.

If you check one or more of those, RAG is a real candidate. If you check none of them, you are about to build infrastructure you do not need.

One specific version of building too early: you probably do not need a dedicated vector database yet. Below serious scale, semantic search runs fine inside tools you already have. A million vectors is only a couple of gigabytes of memory. If you already run Postgres, the pgvector extension does retrieval inside your existing database with no new platform to operate. Reach for a specialized vector database when you actually outgrow that, not on day one. (I put real numbers on where that line is in a companion benchmark post.)

RAG versus the alternatives

RAG is not the only tool, and it is often paired with the others rather than chosen instead of them.

Two of these get confused constantly.

Fine-tuning does not add knowledge. It changes how the model talks, not what it knows. One engineer ran the same set of documents through RAG, fine-tuning, and a giant context window, and found fine-tuning was the worst way to add new facts: it hallucinated more than the base model. Use fine-tuning for voice and format, use RAG for facts.

The million-token context window did not kill RAG. When models grew to enormous context windows, a lot of people declared retrieval obsolete: just paste everything in. In that same experiment, the long-context version cost roughly 24 times more than the RAG pipeline and started dropping facts buried in the middle of the document well before it hit its advertised limit. (That is one engineer’s test on one dataset, so treat the exact multiple as illustrative, not a law.) The “lost in the middle” effect, where models pay less attention to the middle of a long prompt, is a well-documented phenomenon, not a fluke.

The mature 2026 view is that these work together: retrieval finds the relevant material from a big corpus, and a long-context model reasons over just that material. Long context is where the reasoning happens, not a replacement for finding the right evidence in the first place. Long context genuinely wins for a one-time read of a single large thing, one contract, one filing, one codebase. Not for a permanent feature over a growing, changing, per-customer corpus.

What it looks like when it works

Concrete, because “AI assistant” is too vague to plan around. These are the shapes that show up across industry-specific software, with outcome numbers that come from vendor and practitioner reports (so read them as “reported,” not as guarantees).

• LegalTech, clause and contract Q&A. “Show every contract with an auto-renewal clause expiring in 90 days,” with the matching clauses highlighted and cited. Reported reductions of 80% or more in manual contract-review time. RAG fits because the corpus is private, per-client, and citations are non-negotiable.
• InsurTech, policy and claims lookup. On a new claim, retrieve the claimant’s policy, match the incident against coverage and exclusions, surface similar past claims, and hand the adjuster a recommendation with a citation trail they can defend. RAG fits because of private per-tenant documents plus defensibility.
• Vertical health, guideline lookup. Answers grounded in clinical guidelines and literature, because a made-up answer here is dangerous and citations are mandatory.
• Field service, manual and SOP assistant. A technician asks a question, the assistant retrieves the right section of a large, frequently updated manual before answering.
• Support deflection, the feature most teams ship first. Well-configured support assistants are reported to resolve somewhere in the range of half to two-thirds of tier-one tickets on their own, with the best-tuned systems going higher.

There is one line worth internalizing from the support data: the gap between a 40% and a 70% resolution rate is almost never the AI model. It is the quality of the knowledge base, how deeply the thing is integrated, and how disciplined you were about scope. Which is the theme of the next section.

What goes wrong in production

RAG demos beautifully and disappoints in production for reasons that are almost always about retrieval, not the model. When the answer is wrong, the system usually fetched the wrong text, or none, and the model dutifully answered from garbage.

The recurring failure modes, all of which are engineering problems your team solves, not model problems you buy your way out of:

• Bad chunking. An answer is split across two pieces of a document and the system only retrieves one.
• Exact-term misses. Pure semantic search fails to find an error code, a product ID, or a specific clause number, because meaning-based matching is not keyword matching. (The fix is hybrid search, semantic plus keyword.)
• Too much irrelevant context, which dilutes the good material and produces confident, vague answers.
• No evaluation. Quality quietly degrades for months because nobody is measuring retrieval.

The teams that ship working RAG treat it like any other software system: they define metrics before writing code, build evaluation before features, and instrument from day one. That is not glamorous and it is exactly what separates a demo from a product.

Cost has a habit of sneaking up. A typical small application might run a vector store and embeddings for a few hundred dollars a month. The surprises come later. One team documented their infrastructure growing 3.5 times while usage grew only 2.6 times, driven by three things: re-indexing (every time you change the embedding model or chunking strategy, you re-embed the entire corpus, which for a large corpus is a real bill), the number of chunks fetched per query drifting upward over time, and vector-database pricing tiers that jump non-linearly as you grow. Budget for re-embedding, and price your infrastructure at twice your current corpus before you commit to a vendor.

Latency is real too: a straightforward RAG query adds roughly 100 to 500 milliseconds, and once you add reranking and query rewriting you are into the hundreds of milliseconds to low seconds. That is a product decision, not just an infrastructure one.

The part you cannot skip: isolation and embeddings

If you serve multiple customers, this section is the one that turns a nice AI feature into an incident. Building multi-tenant RAG is, underneath, a data-security problem wearing an AI costume.

Cross-tenant leakage is the headline risk. In a shared vector store, tenant A’s query can retrieve tenant B’s documents unless isolation is enforced properly. One security write-up testing a naive setup found cross-tenant leakage succeeded on every single query it tried, with no technical sophistication required. The fix is not subtle, but it has to be done at the right layer.

Do not filter tenants at the prompt layer. Telling the model “only answer from this customer’s documents” in a system prompt is not isolation. A model is a non-deterministic text generator and is susceptible to prompt injection. It will, at some measurable rate, surface material it should not have. Isolation has to be deterministic and enforced at the database and query layer, before any content reaches the model. In Postgres terms this is per-tenant filtering backed by the same row-level and access controls you would use for any sensitive data, which is exactly the multi-tenant architecture and tenant data isolation work you should already be doing.

And treat embeddings as sensitive data, because they are. A common and wrong assumption is that storing embedding vectors instead of the original text is safe because vectors are “just numbers.” Researchers have reconstructed a large fraction of short text inputs directly from their embedding vectors. If someone can read your vector store, assume they can recover a meaningful amount of what those vectors encode, which means your vector store is in scope for the same encryption and access controls as the raw documents. This is squarely a Security and Compliance question, not an afterthought.

If you decide to build, start boring

Say you have run the decision and RAG is genuinely the right call. The advice that saves the most money and pain is: start with the least exotic version that could work.

• If you already run Postgres, use pgvector and keep retrieval inside your existing database. No new platform to operate until you actually outgrow it.
• Use hybrid search (semantic plus keyword) and a reranker. That combination fixes the majority of retrieval failures before you touch anything fancy.
• Build evaluation from day one, so you can tell when quality moves.
• Add the advanced patterns (agentic retrieval, knowledge graphs) later, and only when a measured failure demands them, not because they are in the headlines.

The frameworks to do this in Java are mature now, which I covered in Spring AI vs LangChain4j. The step-by-step build on Postgres, and the benchmark showing exactly how far pgvector scales before you would need a dedicated vector database, are companion posts to this one.

The short version

RAG is handing your AI the right page from your own documents before it answers. It is the right tool when your knowledge base is large, changing, private to each customer, or has to be auditable. It is the wrong tool, or at least a premature one, when your knowledge fits in a prompt or you have not yet proven the simple approach fails. When you do build it, most of the risk is not in the model. It is in retrieval quality, in the cost curve after month three, and above all in keeping one customer’s data from surfacing in another’s answer.

If you are weighing an AI feature for your product and want a straight answer on whether RAG is worth it, what it will cost, and whether your tenant isolation is airtight before you ship it, that is exactly the kind of thing I help with under AI Enablement. Book a free 30-minute call and we will figure out the smallest thing that actually solves your problem.