I don't write code. Here's how I finally understood RAG.

I've worked in IT for years, but I don't write code. When LLMs went mainstream, the people around me started throwing around terms — RAG, embeddings, retrieval — and I realized I couldn't follow any of it. So I taught myself. The hard part wasn't the concepts; it was that the information is scattered across papers, vendor blogs, and tutorials that assume you're building the thing, not trying to understand it.

This is what I wish someone had handed me on day one. Say you ask an internal assistant: "What's our refund policy for enterprise customers?" A few seconds later you get an answer, with a link to the actual policy document.

It looks like the model knew. It didn't. The model has never seen your company's policy — it was trained long before that document existed, and it has no access to your files.

What follows is what actually happens between the moment you type a question and the moment an answer comes back: eight steps, plus the part that happens before you ask, and the loops nobody draws in the diagram.

What you're about to walk through is a RAG pipeline. RAG itself is simpler than it sounds: retrieve the relevant text first, then let the model generate an answer from it. The eight steps below are how that idea usually gets implemented.

One analogy runs through all of them. Picture an executive who assigns a task, and a secretary who has to go pull the files and draft the response.

1. Receive the question

The system takes in the question the user typed.

In the office: the executive assigns a task — "find out about X for me and give me a written response."

2. Embed and understand

First the question gets cleaned up to make it easier to search: vague phrasing is rewritten, synonyms are added, a complex question is broken into several smaller ones. Then it's converted into a string of numbers — a vector — so the computer can compare its meaning against the database.

In the office: the secretary makes sure they understand the assignment before going anywhere. They clarify what's vague and rephrase it into the standard terms the filing system uses.

3. Retrieve

Take that string of numbers and search the database for the most relevant passages. In a real system this is a wide-to-narrow funnel: search two channels at once (meaning-based vector search plus literal keyword search), filter out anything that shouldn't appear — content you don't have permission to see, content that's outdated — then use a more careful model to re-rank what's left. That last part is called reranking.

In the office: the secretary goes to the archive room and searches several channels at once, excludes classified files they aren't cleared to see and superseded versions, then reads each remaining document closely and keeps only the pages that actually answer the question.

4. Augment — assemble the prompt

The retrieved material, your original question, and the system's behavior instructions get assembled into one complete piece of text, ready to hand to the model. The word Augmented in Retrieval-Augmented Generation refers to exactly this: using the retrieved material to augment your original question.

In the office: the secretary puts the retrieved documents, the executive's original question, and the instruction "write from the material, don't improvise" into one working folder.

5. Generate

The language model reads that folder and writes the answer out word by word.

In the office: the department's designated writer takes the folder and drafts the response from the material inside.

6. Guardrails

The system checks the answer for problems — leaked information, wrong formatting, obvious fabrication. This check is standardly called guardrails. More sophisticated systems also have the model check its own answer against the retrieved material and revise if it finds a discrepancy; that self-check step is called reflection.

In the office: the drafter checks their own draft against the original documents, then the department head reviews and signs off before anything goes upward.

7. Respond with citations

The answer comes back to you along with a citation: which document, which passage, this statement is based on.

In the office: the response notes "based on Article X of Document Y," so the executive can pull the original and verify at any time.

8. Log

The system records the exchange, usually keeping two separate copies. A log — a persistent record for troubleshooting, auditing, and statistics. And conversation history — kept as context for your next question. The two serve different purposes and often live in different places.

In the office: the office logs it for the record — who assigned it, when it was handled, which files were pulled, what the response was — and at the same time files the exchange into that executive's correspondence folder, so it's easy to reference next time.

But there's a precondition those eight steps skipped — and it happened long before you typed anything.

Offline indexing: where the material actually comes from

The eight steps above all describe what happens "after the executive assigns the task" — but one precondition got skipped: who put the vectors into that vector database that Step 3 retrieval searches, and when? The answer is a separate preparation pipeline that runs continuously outside the eight steps, called offline indexing ("offline" meaning: it doesn't happen live while answering a question, but is done ahead of time):

Document → Parsing (reading formats like PDF and Word into plain text) → Chunking (cutting long documents into segments, say a few hundred characters each, because the model can't read too much at once and precise retrieval gets harder with long chunks) → Embedding (converting each small chunk into a numeric vector) → Storing it in the vector database.

This step is like the archive room routinely filing, cataloging, and organizing all documents by topic ahead of time, so they can be pulled on demand later — without doing this first, every time the executive assigns a task, you'd have to re-search the entire company's documents from scratch on the spot, which would be unbearably slow. It's also not a "do it once and done" job: new and revised documents, or adjustments to retrieval strategy, all trigger index updates or rebuilds — Loop D below covers what drives this.

Are these eight steps a straight line? Four loops

Looking at a single smooth Q&A exchange, the eight steps really are a straight line from start to finish (the output of one step is the raw material for the next, and the order can't be scrambled). But a real system hangs a few "loop-backs" off this main trunk, which kick in and send things back under certain conditions:

Loop A · Search Again: Before generating an answer, the model discovers "the retrieved material isn't enough to answer this question," so it rephrases and goes back to Step 2 and Step 3 to retrieve another round — possibly repeating several times until the material is sufficient. The key point: how many rounds it searches isn't hard-coded in advance — the model decides on the spot — this is exactly the "agentic" way of working that Chapter Four covers later.

Loop B · Guardrail Rejection: When the Step 6 check fails, the system falls back to Step 5 to regenerate, or back to Step 3 to re-retrieve, according to pre-defined rules. Unlike Loop A, here "under what conditions to fall back, to where, and how many times at most" are all rules engineers hard-coded in advance — the model has no say in the decision.

Loop C · Multi-Turn Conversation: The model itself has no memory — once a round of Q&A ends, it "remembers" nothing. The reason you can follow up with "what about the second one" is that Step 8 saves the question and answer into conversation history every round (note: this is the "conversation history" copy of the two records Step 8 keeps, not the log); when you send a new question, the working folder Step 4 assembles includes this history alongside the new question and newly retrieved material, and the model rereads all of it on the spot to figure out what "the second one" refers to. This loop in one sentence: the previous round's output becomes part of the next round's input.

Loop D · Logs Feeding Back: The logs Step 8 accumulates are periodically handed to an evaluation system for analysis (which kinds of questions get answered poorly, and whether it was retrieval or generation that got it wrong); engineers use the findings to adjust chunking methods and retrieval strategy, then rebuild the offline index. This loop cycles on a scale of days or weeks, and the decisions in this loop are made by people, not the system.

The three generations of RAG

RAG has kept evolving since it first appeared, and the industry conventionally divides it into three generations. (The original survey calls the third generation Modular RAG; this piece uses Agentic RAG, the term that has since become more common and describes the shift more precisely.)

The first generation is called Naive RAG: the eight steps run straight through, and each step does only the simplest possible version. The second generation is called Advanced RAG: the skeleton stays the same, but every step gets more refined — Step 2 adds rewriting and decomposition, Step 3 becomes the wide-to-narrow funnel, Step 6 adds reflection and correction, plus rule-triggered retries like Loop B. The "real system" described in this chapter belongs to this generation. The third generation is called Agentic RAG, which Chapter Four covers in depth.

The dividing line between the three generations isn't whether loops exist, but who decides the loop. In the first two generations, the looping rules are written into the code in advance by engineers, and the system just executes them; Agentic RAG instead hands "whether to search again, and how" to the model to judge on the fly — Loop A is exactly this case. As for Loops B, C, and D, in the first two generations their decision-making power likewise stays with rules and people: B is triggered by preset rules (the rules can ask the model to act as a "scorer" internally — for instance, having the model rate the quality of retrieved results — but which path to take after scoring is still decided by the rules), C attaches history in a fixed way, and D is driven by engineers — so their presence doesn't change the generational classification. Conversely, the moment a loop's decision-making is genuinely handed over to the model — for example, having the model judge for itself whether the answer is well-grounded and whether to re-retrieve or rewrite (the industry does have approaches like this, such as Self-RAG) — by the same standard, that loop becomes agentic too. In short: the three generations use the same components; what changes is who's in command.

That's the full path from question to answer. The next chapter is
about a question that comes up immediately after this one: if the
model doesn't know your documents, why not just train it on them?

This is the first chapter of a free online guide. The rest is here:
https://llmknowledge.pages.dev

Sources

• Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey (2023) — the origin of the Naive / Advanced / Modular three-generation split. arXiv:2312.10997
• Singh et al., Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (2025) — the third generation, where the model decides how to search rather than following hard-coded rules. arXiv:2501.09136
• Asai et al., Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, ICLR 2024 (oral) — a concrete case of a loop whose decision belongs to the model. arXiv:2310.11511

AI Disclosure: I wrote this because I needed it myself. I used an AI assistant to tighten the English — it's not my first language — but the content, the analogy, and the mistakes are mine.