Retrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation (RAG) is the most impactful architectural pattern for grounding LLM responses in real, current, organisation-specific data. Understanding it in depth is essential for building reliable AI applications.
The core problem RAG solves
LLMs have two fundamental limitations for enterprise applications: their knowledge has a cutoff date, and they do not know your organisation specific information (policies, products, clients, internal data). RAG addresses both by retrieving relevant documents at query time and including them in the prompt context, allowing the model to answer from current, specific information rather than general training data.
The RAG pipeline
1. Ingestion. Source documents (PDFs, web pages, databases) are chunked into segments, embedded (converted to numerical vectors that represent semantic meaning), and stored in a vector database.
2. Retrieval. When a user query arrives, it is also embedded. The vector database finds the stored chunks most semantically similar to the query — this is the retrieval step.
3. Augmentation. The retrieved chunks are inserted into the prompt as context: "Answer the following question using only the provided documents..."
4. Generation. The LLM generates a response grounded in the retrieved documents, dramatically reducing hallucination on factual questions about the provided content.
Chunking strategy matters
How you split documents for embedding significantly affects retrieval quality. Fixed-size chunks are simple but may split important context across boundaries. Semantic chunking (splitting at natural section boundaries) produces better retrieval but requires more sophisticated implementation. Chunk size is a key hyperparameter — too small loses context, too large dilutes relevance.
RAG is not magic — it is document retrieval plus context injection. Its quality depends entirely on retrieval quality. The most common RAG failure mode is not the generation step — it is returning the wrong documents.