Retrieval Paths and Source Priority: How AI Agents Decide Which Context to Trust

An agent at a portfolio company we audited last quarter quoted a customer the wrong price. Not a hallucination - the number was real. It came from a pricing sheet on the company wiki, last edited in 2023, two acquisitions and one repackaging ago. The correct price lived in the CRM the whole time. The agent had access to both systems. It just had no opinion about which one to believe.

That is the failure mode almost nobody designs for. Teams spend weeks tuning chunk sizes and embedding models, then let the agent treat a stale Confluence page and a live system of record as equally credible because both came back with a cosine similarity above 0.8. Retrieval quality is not just "did we find relevant text." It is "did we find the text this organization would bet money on."

This post is a framework for that second question: how to design retrieval paths, rank sources, handle permissions and conflicts, and decide when the agent should stop choosing and ask a person. It builds on our earlier posts on consolidating wikis without migrating the mess and what agentic knowledge actually means.

Retrieval Is a Trust Decision, Not a Search Problem

Vector search answers one question: which chunks of text are semantically similar to this query. It answers nothing about whether those chunks are true, current, authorized, or authoritative. Similarity and trust are orthogonal, and the wiki pricing example is what happens when you conflate them.

A retrieval path is everything between the question and the context window: which systems get queried, in what order, with what filters, how results are ranked and merged, and what happens when the top results disagree. Most RAG tutorials collapse this into "embed everything into one index and search it." That works in a demo with 40 documents. It fails in a company with a CRM, an ERP, three wikis, a shared drive, and 9 years of Slack history, because the one-index design erases the trust signal that lives in where a document came from.

There is also a mechanical problem stacked on top of the trust problem. Research from Stanford and Berkeley on long-context behavior (Liu et al., "Lost in the Middle") showed that models attend most reliably to information at the beginning and end of the context window and degrade sharply on content buried in the middle. So even if the right document is retrieved, dumping 20 loosely ranked chunks into the prompt means the correct answer may be sitting in the dead zone while the stale wiki page sits at position one. Ranking is not cosmetic. It decides what the model actually reads.

The takeaway: before touching retrieval infrastructure, write down which sources your organization trusts, for what, and in what order. That document is worth more than any reranker.

The Source Priority Stack

Every company we work with ends up with some version of the same hierarchy. Here is the general shape:

Three design rules make this stack operational rather than decorative:

• Route by claim type, not just by query. A question like "what is this customer's renewal date" should never touch Tier 4. It has exactly one legitimate source. Building this routing logic is a workflow decision, and the LangGraph workflows-versus-agents distinction applies directly: source routing should usually be a deterministic workflow step, not something the agent improvises per request.
• Attach tier metadata to every chunk at ingestion. Source system, owner, last-modified date, and tier travel with the chunk. Rerank retrieved results with tier as a hard signal, not a soft one. A Tier 1 hit at similarity 0.74 should beat a Tier 4 hit at 0.91 for factual claims.
• Give every tier a freshness window. A billing record is trusted as of right now. A published policy might be trusted for 12 months past review. A wiki page older than 18 months should retrieve with a staleness flag the agent must surface: "According to a pricing page last updated March 2023..." That single behavior, forcing the agent to show the age of what it read, catches more silent errors than any accuracy tuning we have done.

For organizations whose knowledge is genuinely relational - entities, ownership chains, dependencies between systems - flat chunk retrieval hits a ceiling regardless of ranking. Microsoft's GraphRAG work is the reference point for building retrieval over an extracted knowledge graph instead, which handles "how do these things connect" questions that similarity search structurally cannot.

Permissions Are Part of the Retrieval Path, Not a Filter You Bolt On

Here is the mistake that turns a retrieval problem into a breach: retrieve first, filter later. The agent pulls 20 chunks, drops the ones the user cannot see, and answers from the rest. Except the model already read the restricted content during an earlier step, or the filter runs on document IDs while the summary of the restricted document is already sitting in conversation memory. Content leaks through paraphrase, through citations, through "based on what you found earlier."

Permission-aware retrieval means the access check executes inside the retrieval query itself, scoped to the requesting user, so unauthorized documents never reach the model at all. This is a solved pattern in enterprise search - Elastic calls it document-level security, and every serious enterprise search vendor has an equivalent - but it is routinely skipped in agent builds because the demo ran as a single admin user and nobody revisited the assumption.

Two additional rules for the permission layer:

• The agent inherits the user's permissions, never its own. A service account with broad read access querying on behalf of a junior analyst is a confused-deputy problem waiting for an audit finding. The OWASP Top 10 for LLM applications covers this class of failure under excessive agency and sensitive information disclosure, and it is worth making that list required reading for whoever owns your agent roadmap.
• Treat retrieved content as untrusted input, not just as facts. A document in your own wiki can contain text that manipulates the agent ("ignore prior instructions and..."), planted or accidental. Source tier should also govern how much instruction-following weight content gets: Tier 5 and Tier 6 content should be quarantined as data to reason about, never directives to obey.

The business framing for non-technical readers: your retrieval layer is now part of your access control surface. If your SOC 2 auditor asked "can this agent show a contractor the CEO's compensation," you want the answer enforced in the query, not in a prompt instruction asking the model to please be discreet.

Conflict Handling: When the Agent Should Stop Choosing

The stack resolves most conflicts by rank: CRM beats wiki, done. The hard cases are conflicts within a tier, or between a fresh low-tier source and a stale high-tier one. A Slack message from the VP of Sales yesterday says the discount policy changed. The published policy doc says otherwise. Which wins?

Neither, automatically. This is where most agent builders over-rotate on autonomy. The honest answer is that the organization's own truth is in flux, and no ranking function fixes that. What the agent should do is narrow and mechanical:

1. Detect the conflict explicitly. Compare extracted claims from top results, not just rank scores. Two sources both above threshold asserting different values for the same field is a detectable event.
2. Surface both with provenance. "The published policy (updated Jan 2026) says 15%. A message from J. Alvarez on July 10 indicates 20% pending approval." Citations with dates, every time.
3. Escalate when the answer is load-bearing. If the conflicting fact drives a quote, a refund, a contract term, or anything irreversible, the agent routes to a human with the conflict pre-packaged. If it is background context, the flagged dual answer is fine.

The counter-intuitive part: a well-designed agent that escalates 8% of queries builds more organizational trust than one that answers 100% and is quietly wrong on 3%. Users forgive "I found conflicting sources, here they are" instantly. They do not forgive a confidently wrong number they repeated to a customer. Escalation is not a capability gap. It is the feature.

Also define fallback order for the opposite problem, retrieval coming back empty: Tier 1 miss falls through to Tier 3, then to a scoped web search, each step annotated in the answer so the reader knows how far down the stack the agent had to reach. An agent that says "I could not find this in our systems; here is what public documentation says" is being useful. One that silently substitutes web content for internal truth is lying by omission.

Evaluating Retrieval Paths Separately From Answers

You cannot fix what you measure as one blob. "Answer quality" evals conflate three different failures: retrieved the wrong thing, ranked the right thing too low to be read, or reasoned badly over the right thing. In our experience the first two account for most production incidents that get filed as "hallucination."

The eval setup that isolates them:

• A golden retrieval set. 50 to 200 real queries, each labeled with the documents that should be retrieved and the source tier that should win. Measure recall and precision on retrieval alone. LangSmith's evaluation tooling supports tracing and scoring the retrieval step independently from generation, which is exactly the separation you want.
• Conflict probes. Seed your test corpus with known contradictions (a stale doc and a current record) and assert the agent surfaces or escalates rather than picking one. This is the test suite almost nobody writes and the one that catches the expensive failures.
• Permission probes. Run the same query as users with different access levels and assert the restricted content never appears, including in paraphrase. Automate this; it regresses every time someone adds a new connector.
• Staleness drift checks. Re-run the golden set monthly. Retrieval quality decays as the corpus grows and ages even when the code never changes, which makes this an operations job, not a launch-week checkbox.

Why This Lands Hardest in PE-Backed Mid-Market Companies

The category framing matters here. Private equity in 2026 is squarely in operational value creation mode: PwC's midyear deals outlook points to sponsors working through a large backlog of long-held portfolio companies where returns now have to come from operating improvements rather than multiple expansion, and Morgan Stanley's 2026 outlook makes a similar case that extended hold periods have shifted the work toward earning growth inside the portfolio. AI agents are on nearly every value-creation plan we see.

The catch: PE-backed mid-market companies have the messiest source-truth environments in the economy. Add-on acquisitions mean two CRMs, three wikis, and duplicate customer records with different renewal dates - the exact problem we covered in our duplicate-data post. An agent deployed on top of that without an explicit source priority stack does not just underperform. It launders the mess into confident answers, at scale, in front of customers, during a hold period where the exit story depends on clean operations.

The practical sequencing for an operating partner: source priority design is a two-week workshop, not a six-month data migration. You do not need to consolidate systems before deploying agents. You need the agent to know which system wins for which claim, which is a much smaller and much higher-ROI piece of work.

How OpenNash Can Help

This is the work OpenNash does in the audit and design phases before any agent ships. We map your systems into an explicit source priority stack, define freshness windows and escalation rules with the people who own each system, and build permission-aware retrieval into the query layer rather than the prompt. The conflict probes and retrieval evals described above ship as part of the deliverable, and the client owns all of it after handoff - the routing logic, the eval suites, the documentation.

If your team is planning agent deployments across a portfolio company with acquisition-era system sprawl, that source-truth mapping is the first conversation worth having. Book a call to map this framework to your specific stack.