Agentic RAG lets a model decide when and how to retrieve evidence, refine a query, or request another source inside a bounded agent loop.

What is the difference between RAG and tools?

RAG retrieves evidence from indexed knowledge. Tools query live systems or perform operations. A current balance belongs behind a tool. A policy explanation may come from RAG.

Should RAG use vector search only?

Usually no. Lexical search handles exact terms and identifiers well, while dense search handles semantic similarity. Hybrid retrieval often provides a stronger baseline.

How do you secure RAG?

Attach access metadata during ingestion and enforce tenant, subject, role, and time filters in storage before retrieved text reaches the model or shared caches.

When should a RAG system abstain?

Abstain when visible evidence is missing, stale, contradictory, below the relevance threshold, or insufficient to support the requested claim.

AI Agents Part 6: Agentic RAG

AI Agent Engineering From Scratch

Language models cannot reliably answer questions about private documents, recent policies, or changing business data. Retrieval-augmented generation, or RAG, supplies relevant source material.

A basic RAG system searches an approved document collection and returns useful passages. The application places those passages in the model's context. The model can then use current, verifiable evidence.

Agentic RAG lets the model decide when to search and whether to search again. It can also choose a source or revise a query. Use this decision loop only when basic retrieval fails on measured tasks.

Each added search increases response time, cost, and failure risk.

Give each information source one job

An agent can obtain information from documents, live systems, and previous runs. Each source needs separate update, access, and deletion rules.

Use RAG to find evidence in a document collection. Use tools to read live systems or perform actions.

Use memory to retain selected information across runs.

RAG

Use RAG for evidence in documents prepared for search:

policies.
manuals.
research papers.
case history.
product documentation.
contracts.
internal knowledge.

Tools and APIs

Use tools for current operational facts and actions:

account balance.
order status.
available inventory.
open incident state.
permission membership.
creating or updating a record.

Memory

Use memory for selected information carried across turns or runs:

user preferences.
project conventions.
previous task summaries.
durable notes with an explicit lifecycle.

The same model can use all three. Keep separate rules for updates, ownership, access, and deletion.

Keep live operational data in the system that owns it.

Retrieval starts before search

Ingestion prepares source documents for search. It fetches each document, extracts its text, divides the text into passages, and adds access data.

The system must record each document version. It must also update the search index when the source changes.

Parsing converts files, tables, and pages into searchable text. If parsing loses a table row, later search cannot return that row.

Each stored passage needs enough metadata to verify its source and enforce access rules. Store fields such as:

chunk_id
document_id
version_id
title
section
text
source_url
effective_at
superseded_at
content_hash
tenant_scope
allowed_roles
allowed_subjects

This metadata answers four questions:

What source did the passage come from?
Which version did the agent use?
Where can a reader verify it?
Who is allowed to see it?

Preserve document structure before splitting by size. Headings, lists, table boundaries, and page anchors often carry meaning. A passage such as "returns accepted for 30 days" is ambiguous without the product, region, or policy section around it.

Test parsing on real examples. Do not assume a successful ingestion job produced faithful text.

Make chunking preserve the question's answer

A chunk is a passage stored as one retrieval unit. Its boundaries decide what search can return.

Fixed-size chunks are easy to implement, but they can separate a rule from its condition. One chunk may say "refunds are allowed" while the next holds the regional exception.

Structure-aware chunking starts with headings, paragraphs, list items, and table rows. It then joins or splits those units within a size range.

Consider this policy:

Late delivery

EU consumer orders may receive a refund review when delivery is
more than five business days late.

Exceptions

Custom products and orders already refunded do not qualify.

A useful stored passage keeps the rule, its heading, and nearby exceptions together. It also stores the document and section identifiers.

Overlap can preserve sentences near a boundary. Too much overlap creates duplicate results and spends context on repeated text.

Choose chunk size and overlap through retrieval tests. Include questions that need a heading, a table row, a nearby definition, or an exception.

Document type matters. A legal policy, API reference, support article, and spreadsheet should not share one blind splitting rule.

Record the parser and chunker versions. A retrieval result is hard to reproduce if the text transformation changes without a version.

Start with a measurable baseline

A retriever searches the index and returns candidate passages. Start with a retriever that you can test and understand.

Lexical search matches words and phrases. It works well for names, codes, error messages, and exact quotations.

Dense search compares numerical text representations called embeddings. It can match text that expresses the same idea with different words.

Test both methods. Their strengths differ, so many systems combine their result lists.

A practical baseline often uses both:

retrieve lexical candidates.
retrieve dense candidates.
combine the ranked lists.
optionally rerank a small candidate set.
apply a relevance threshold.
return structured evidence.

Reciprocal Rank Fusion, or RRF, combines two ranked result lists. It gives more weight to passages near the top of either list.

Test the number of candidates and the RRF settings on real questions. No setting works for every document collection.

A reranker scores a small candidate set again with a more precise model. It can improve ordering, but it cannot recover a missing passage.

Keep lexical-only, dense-only, and combined results during evaluation. Otherwise a fused score can hide a weak component.

Follow one RAG request from question to answer

Assume the same user asks whether order ord_4821 qualifies for a late-delivery refund.

The agent needs two kinds of information:

the current order state from the order tool.
the applicable rule from the policy index.

The order tool returns:

{
  "order_id": "ord_4821",
  "region": "EU",
  "product_type": "standard",
  "delivery_delay_business_days": 8,
  "already_refunded": false
}

The model then proposes a retrieval plan. It can choose the query and source type, but it cannot choose its security scope:

{
  "retrieval_plan": {
    "query": "late delivery refund eligibility EU standard product",
    "collection": "customer-policies",
    "top_k": 8
  },
  "enforced_scope": {
    "tenant_scope": "support",
    "region": ["EU", "global"],
    "effective_at": "2026-08-24",
    "allowed_roles": ["support_agent"]
  }
}

Application code builds enforced_scope from the verified principal, current policy, and server clock. The model cannot supply, remove, or widen these filters.

The retrieval service runs lexical and dense search inside that allowed scope. It keeps each list for inspection before fusion.

A trace might show:

lexical
  1. returns-2026-08#late-delivery
  2. shipping-2026-04#delivery-estimates
  3. returns-2025-11#late-delivery

dense
  1. returns-2026-08#late-delivery
  2. returns-2026-08#exceptions
  3. support-playbook#refund-review

after version and permission filters
  1. returns-2026-08#late-delivery
  2. returns-2026-08#exceptions
  3. support-playbook#refund-review

The old 2025 policy disappears because it is no longer effective. Filtering happened before any passage entered model context.

A reranker can reorder the remaining passages for the full question. The retrieval service then returns structured evidence:

[
  {
    "source_id": "returns-2026-08#late-delivery",
    "document_id": "returns-policy",
    "version_id": "2026-08",
    "section": "Late delivery",
    "text": "EU consumer orders may receive a refund review when delivery is more than five business days late.",
    "score": 0.91
  },
  {
    "source_id": "returns-2026-08#exceptions",
    "document_id": "returns-policy",
    "version_id": "2026-08",
    "section": "Exceptions",
    "text": "Custom products and orders already refunded do not qualify.",
    "score": 0.87
  }
]

The generation step receives the order facts and these two passages. It can state that the order appears eligible for review.

It should not say that a refund exists. Creating one requires a separate tool, policy check, and approval.

The final structured answer might be:

{
  "answer": "Order ord_4821 appears eligible for a refund review because it was eight business days late.",
  "supporting_source_ids": [
    "returns-2026-08#late-delivery",
    "returns-2026-08#exceptions"
  ],
  "sufficient_evidence": true,
  "proposed_next_action": "request_refund_approval"
}

This case needs one document search and one live read. It does not need an open-ended research loop.

Separate retrieval planning from retrieval execution

The model may propose a query, source, and intent. Application code must apply collection limits, permission filters, candidate limits, and time bounds.

A useful retrieval plan has a schema:

{
  "intent": "find_policy_rule",
  "queries": [
    "late delivery refund eligibility EU",
    "refund exclusions custom product already refunded"
  ],
  "required_source_types": ["approved_policy"],
  "max_search_calls": 2
}

Validate this plan before execution. Reject unknown collections, excessive query counts, and filters that attempt to widen access.

Keep query text distinct from filters. The phrase tenant:other-company inside the query must not change the enforced tenant predicate.

Permissions come before generation

Access control must run before retrieved text enters model context.

Filtering after generation is too late. The denied passage may already influence the answer, another tool call, a memory write, or a model-generated query.

The application must identify the signed-in user or service before it searches. Security systems call this identity the principal.

Use the principal to restrict every search by:

tenant.
subject or group.
role.
document ownership.
effective date.
revocation state.

Apply the same scope to lexical indexes, vector indexes, reranking inputs, and caches.

A common mistake is to filter visible metadata in the application after retrieving a global candidate set. That still moves denied text across a boundary and may leak through logs or shared caches. Push the access predicate into storage wherever possible.

Test cross-tenant and revoked-access cases explicitly.

Return structured evidence

The application needs structured evidence for every selected passage.

Structured evidence lets the system display citations, check document versions, and trace incorrect answers.

Return fields such as:

{
  "source_id": "policy-2026-08#section-4",
  "version_id": "2026-08",
  "title": "Returns policy",
  "section": "Late delivery",
  "quote": "Eligible orders may be refunded after...",
  "source_url": "https://example.com/policies/returns#late-delivery",
  "retrieved_at": "2026-08-24T14:00:00Z"
}

Keep the quote small enough to inspect and large enough to support the claim. Preserve a reference to the full source when more context is needed.

A grounded answer can then separate claims from citations:

claim
supporting_source_ids
sufficient_evidence
missing_evidence

A schema makes citation errors visible. Tests must still verify that each cited passage supports the related claim.

Teach the system to abstain

A system should refuse to answer when its evidence is too weak.

This refusal prevents a fluent answer from hiding missing or contradictory sources.

Return an insufficient-evidence response when:

no visible passage meets the relevance threshold.
the source is stale.
two current sources disagree.
the parser confidence is low.
the question asks for a live fact.
the principal cannot access the required evidence.
the source supports only part of the claim.

A refusal can reduce the number of answered questions. Measure whether it also reduces unsupported answers.

Do not turn "access denied" into "the document does not exist." The user-facing response can stay generic while the protected trace records the real reason.

Add agent behavior for a named reason

Many questions need only one search. One search is easier to test, faster to run, and cheaper to operate.

Add a decision loop only when test failures show a specific need:

the question contains several subquestions.
the first query is too broad.
different sources must be consulted.
evidence is contradictory.
the model needs to inspect a cited source in more detail.
a document answer must be combined with a live tool result.

Set maximum values for query rewrites, search calls, selected sources, and context size.

Keep the retrieval plan observable:

original question
query used
filters applied
candidate IDs
reranker version
selected source IDs
abstention reason

Agentic retrieval should improve a measured failure mode. If the extra loop does not improve task outcomes, remove it.

Treat retrieved content as untrusted

A document can contain malicious instructions. For example, it can tell the agent to ignore earlier rules or send a file elsewhere.

The retriever may return that text because it is relevant to the query.

Treat retrieved content only as evidence. The application must continue to enforce policy.

Keep retrieved text in a marked data section. Do not let it change tool permissions, identity, approval requirements, or system instructions. Enforce those controls in the application.

RAG poisoning can also happen during ingestion. Protect source connectors, record provenance, detect unexpected bulk changes, and support deletion or index rebuilds.

Evaluate the pipeline in stages

An end-to-end answer tells you whether the system worked. Stage metrics tell you where it failed.

For ingestion, check:

parsing completeness.
table and OCR accuracy.
source lineage.
permission metadata.
version updates.
deletion propagation.

For retrieval, measure:

required-source recall: how often the top results contain the required passage.
result precision: how many returned passages are relevant.
required-source rank: where the required passage appears in the result list.
permission leakage: whether any user receives denied content.
freshness: whether results use the current source version.
performance for each query type.

For generation, check:

factual correctness.
citation support.
citation completeness.
contradiction handling.
abstention.
use of live tool facts where required.

For the full system, track task success, latency, cost, and repeated-run reliability.

Read failed traces. First, find the stage where the required passage disappeared.

Search might miss it. The reranker might remove it. The context builder might omit it.

The model might also ignore a passage that reached its context. Each failure needs a different fix.

Diagnose failures at the stage that caused them

A RAG system can produce a wrong answer even when its final response looks well cited.

Symptom	Likely stage	What to inspect
Correct document never appears	Ingestion or first retrieval	Parsed text, metadata, query, and candidate lists
Correct passage ranks below noise	Retrieval or reranking	Lexical, dense, fused, and reranked positions
Old policy supports the answer	Version filtering	Effective and superseded timestamps
Citation is relevant but does not prove the claim	Generation	Claim-to-source mapping
User receives another tenant's passage	Authorization	Storage predicate, reranker input, and cache scope
Agent searches repeatedly with similar queries	Planning loop	Query history, stop rule, and evidence threshold
Answer uses policy for a live balance	Source routing	Tool-versus-RAG classification
Answer ignores an exception	Chunking or context selection	Passage boundaries and selected evidence
Deleted document still appears	Index lifecycle	Connector event, deletion job, and cache invalidation

A plausible citation is not enough. The cited passage must support the nearby claim.

Store the query, applied filters, candidate IDs, scores, selected passages, and final claim mapping in one trace. Redact protected text from broad logs.

Decide when agentic retrieval is justified

Use a bounded agentic loop when one search cannot reliably select the needed evidence.

Good reasons include decomposition, source selection, contradiction checks, and combining documents with live tools.

Poor reasons include making the demo look autonomous or compensating for weak ingestion.

Set deterministic limits around model choices:

retrieval_limits:
  max_search_calls: 3
  max_query_rewrites: 2
  max_sources_in_context: 6
  max_passages_per_source: 3
  require_new_evidence_for_repeat_search: true
  stop_on_access_denied: true

The new-evidence rule prevents a loop that rephrases the same query without changing the evidence set.

Define stop reasons that appear in traces and metrics:

sufficient_evidence
no_new_evidence
search_budget_exhausted
contradictory_sources
access_denied
live_tool_required

The model can recommend a stop reason. Application code enforces the actual limit.

RAG build checklist

Source preparation

Which source system owns each document?
Are versions, effective dates, and deletions available?
Does parsing preserve headings, lists, tables, and exceptions?
Can the team reproduce a passage from its parser and chunker versions?

Retrieval

Which query types need lexical search?
Which query types improve with dense search?
Are the two result lists visible before fusion?
Can the reranker receive only authorized candidates?
Does the evidence schema preserve source and version IDs?

Security

Is the principal known before search starts?
Do storage filters enforce tenant, subject, role, and time scope?
Do caches include the same permission scope?
Is retrieved text isolated from instructions and tool permissions?

Answer policy

Which claims require citations?
What counts as sufficient evidence?
How does the system handle stale or contradictory sources?
When must it use a live tool instead of document search?
Which abstention reason does the user see?

Agent loop

Which measured failures justify another search?
What are the hard limits on queries, sources, and context?
Must a repeat search return new evidence?
Can every query and stop decision be reconstructed?

Evaluation

Does the test set include exact identifiers and paraphrased questions?
Does it include tables, exceptions, version changes, and denied sources?
Can metrics separate ingestion, retrieval, selection, and generation failures?
Does release testing compare latency and cost with the non-agentic baseline?

Ship the first version when it can retrieve authorized evidence, cite it, and abstain. Add planning behavior after traces show a repeatable retrieval failure.

The OpenNash guide to reliable agentic RAG architecture covers a larger production design.

Add a retrieval step only when a measured failure shows why you need it.

Previous: AI Agents Part 5: Context and Memory

Next: AI Agents Part 7: Evals and Observability turns traces, evidence, tool calls, and final state into tests you can use before and after release.