RAG Architecture Guide: Retrieval-Augmented Generation for Business

Language models are impressive generalists that know nothing about your organization. Ask a model about your refund policy, your HR handbook or your product's configuration options, and it will either decline or invent a plausible answer. Retrieval-augmented generation (RAG) solves this by giving the model the right information at the moment it answers. This guide explains the architecture and the decisions that make it work, as part of our AI consulting series.

What is retrieval-augmented generation?

RAG is a pattern that combines a search system with a large language model (LLM). When a user asks a question, the system retrieves relevant passages from your content and includes them in the prompt, and the model writes an answer based on those passages, ideally with citations.

RAG is the most common architecture behind internal knowledge assistants, customer support assistants and document Q&A tools because it:

• Uses your current documents without retraining a model
• Lets users verify answers through citations
• Updates instantly when documents change
• Can respect existing document permissions

How does a RAG system work?

A RAG system has two pipelines: one that prepares content ahead of time and one that answers questions in real time.

Ingestion pipeline (offline):

1. Connect to sources such as SharePoint, Google Drive, a wiki, a website or a database.
2. Extract text from files, including PDFs, Word documents and web pages, and preserve structure such as headings and tables.
3. Chunk documents into passages of a manageable size.
4. Enrich chunks with metadata such as title, source, date, department and access permissions.
5. Embed each chunk into a numerical vector that captures its meaning.
6. Index chunks in a search service that supports both keyword and vector search.

Query pipeline (real time):

1. Understand the question, optionally rewriting it for better search.
2. Retrieve candidate chunks using hybrid search, filtered by the user's permissions.
3. Rerank candidates to put the most relevant passages first.
4. Generate an answer with the LLM using only the top passages and clear instructions.
5. Cite sources and log the interaction for evaluation.

What are the core components of a RAG architecture?

Cloud platforms offer managed versions of most of this stack. See the Vertex AI overview and the Azure AI services overview for platform-specific options.

How should you chunk documents?

Chunk along the document's natural structure, such as sections and headings, rather than at arbitrary character counts. Good chunking keeps related information together so retrieval returns complete answers.

• Keep chunks large enough to contain a complete idea and small enough to be specific, often a few hundred words.
• Use modest overlap between adjacent chunks so information at boundaries is not lost.
• Prepend the document title and section heading to each chunk to preserve context.
• Treat tables carefully; convert them to text that keeps row and column meaning.
• Store metadata such as effective date so outdated versions can be filtered or deprioritized.

Why use hybrid search and reranking?

Hybrid search combines keyword search, which is precise for names, codes and exact terms, with vector search, which finds passages with similar meaning even when wording differs. Together they retrieve better candidates than either alone.

A reranker then scores the top candidates against the question more carefully. Reranking is one of the most cost-effective quality improvements in RAG, because it ensures the model sees the best passages first.

How do you enforce permissions in RAG?

Enforce permissions at retrieval time, so the model never sees content the user is not allowed to access. Store access control information, such as allowed groups, with each chunk and filter search results by the signed-in user's identity.

Do not rely on prompts to hide sensitive content. If a passage reaches the model, assume it can appear in the answer. Also fix oversharing in source systems first; RAG faithfully surfaces whatever permissions allow.

How do you evaluate a RAG system?

Evaluate with a fixed set of real questions and expected answers, and rerun it after every change to chunking, models or prompts.

Build an evaluation set of 50 to 200 questions that includes:

• Common questions with clear answers in the documents
• Questions requiring information from multiple documents
• Questions the documents cannot answer, where the correct response is "I don't know"
• Questions with outdated or conflicting source material

Measure:

Automated LLM-based evaluators help at scale, but have subject-matter experts review a sample regularly.

How do you secure a RAG application?

Treat a RAG application as a system with access to sensitive data.

• Authenticate users with your identity provider and pass identity to retrieval filters.
• Defend against prompt injection, where malicious text inside a document tries to change the model's behavior. Treat retrieved content as data, not instructions, and limit what actions the system can take.
• Log questions and answers with appropriate retention and access controls.
• Use enterprise AI services with data protection terms.
• Apply rate limits and budgets to control abuse and cost.

Our API development best practices cover authentication and rate limiting for the services behind a RAG application.

Should you use a managed RAG service or build your own?

Start managed, and build custom only when you hit a limit.

When does RAG become an agent?

When the system needs to take actions or choose among several tools and data sources, it becomes an agent. Many agents use RAG as one of their tools. See AI agents for business for design patterns.

How Desai InfoTech builds RAG systems

Desai InfoTech builds RAG assistants evaluation-first: we agree on a question set and success criteria with your subject-matter experts before choosing tools, then iterate until the numbers meet the bar. Contact us to discuss a knowledge assistant for your organization.