Should You Block AI Crawlers? How AI Search Finds Your Website and What to Allow

A product manual page that three other websites link to can still be the best answer on the internet for "how do I reset the XR-200 controller." Traditional search engines have spent two decades learning to find pages like that. AI search systems are newer at it, and they waste a lot of their crawling on pages that return the wrong status code, redirect to the homepage, or exist under five different URLs. Cloudflare's write-up on its discovery work with OpenAI is about exactly this problem: how AI search finds pages, and which site-side mistakes burn crawl effort without producing anything worth citing.

If you own a marketing site, a publication, or a SaaS docs portal, you have two questions to answer. Should AI crawlers be allowed in at all? And if they are, will they find the right pages? This guide covers both in that order, ending with a policy you can ship this week.

How AI Systems Find and Use Your Pages

When someone asks ChatGPT, Claude, or Perplexity a question that needs current information, the answer usually comes from one of two retrieval paths:

1. A pre-built search index. The provider's search crawler visits your site ahead of time, stores the content, and retrieves it when a relevant question comes in. OpenAI's crawler for this is OAI-SearchBot.
2. A live fetch. The model opens a specific URL in the moment, often because the user pasted it or the system decided it needed a fresh copy. OpenAI labels these requests ChatGPT-User.

There is also a third kind of crawler that has nothing to do with answering questions today. Training crawlers such as GPTBot, ClaudeBot, and Common Crawl's CCBot collect content that may later be used to train models. OpenAI's crawler documentation lists GPTBot and OAI-SearchBot as separate agents with independent robots.txt settings. Other major providers split them the same way.

That split is the most useful idea in this whole topic. "Should I let AI in?" is three questions:

Google works a little differently. Googlebot crawls for both Search and AI features, and Google offers a separate token, Google-Extended, that controls whether your content is used for Gemini training and grounding. Blocking Google-Extended does not remove you from Google Search.

Once content is retrieved, the system still has to decide which source to trust. If your pricing page and an outdated forum thread both match a query, the result depends on source priority. That is the same problem we covered for internal agents in retrieval paths and source priority. Public AI search faces it at web scale, so clean, unambiguous pages win more often.

Should You Let AI Crawlers In? A Decision Matrix

For most businesses, the defensible default is to allow search and user-fetch crawlers and to block training crawlers unless you have a reason to want training exposure. The 2026 decision matrix from Digital Applied reaches the same baseline, and it holds up because it separates the lane that sends traffic from the lane that takes content without returning anything measurable.

The right answer still varies by what your content is worth and how you make money from it:

The docs portal row surprises people. Many teams block every AI bot by default out of caution, and then their customers ask ChatGPT how to configure the product and get an answer built from a 2023 Stack Overflow thread. For documentation, being invisible to AI is often worse than being trained on. When the model is going to answer questions about your product anyway, you are better off if it learned from your docs.

Publishers have the opposite problem. Allowing search crawlers produces citations, but a cited summary can answer the reader's question so completely that nobody clicks through. Whether search access pays off depends on how much referral traffic you get compared with how much reading you lose. Measure that before you commit.

Write the Policy in robots.txt (and Know Its Limits)

Robots.txt is still the main control, and the major AI crawlers say they respect it. It is also voluntary. RFC 9309, the standard that formalized the protocol, states plainly that these rules are not a form of access authorization. Treat robots.txt as your published policy and your CDN or firewall as the enforcement layer.

Here is a starting policy that allows search and blocks training:

# Search and answer engines
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /search

# Training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

Look at the repeated Disallow lines in the search group. This is the mistake we see most often in robots.txt files written for AI. Under RFC 9309, a crawler uses the group that names it most specifically and ignores the wildcard group entirely. If you add User-agent: OAI-SearchBot with only Allow: /, you have just given that bot access to your admin paths, cart pages, and internal search results, which your * group used to block. Copy your shared disallows into every named group.

A few more practical rules:

• Keep a crawler inventory. New user agents appear every few months. Directories such as GEO Toolbox's AI crawler list are useful for checking, but confirm each agent's behavior against the vendor's own documentation before you write a rule for it.
• Verify, don't trust, user agent strings. Anyone can send a request claiming to be GPTBot. OpenAI and other providers publish IP ranges, and CDN bot management tools can match against them. Block requests that claim a known AI user agent but come from outside its published ranges.
• Decide on user-initiated fetches deliberately. These requests are triggered by a person, so vendors handle them differently from automated crawling, and robots.txt may not govern them in the same way. If a page must never be fetched, protect it with authentication, not a robots rule.
• Treat llms.txt as optional. It is a proposed convention for giving models a curated map of your site. It costs little to add, but no major AI search provider has said it uses the file for ranking, so don't expect it to do anything on its own.

Find Out Who Is Already Crawling You

Don't write a policy before you look at your logs. Your access logs already show which AI crawlers visit, how often, and what they get back. On a standard nginx or Apache combined log, this gives you a first count:

grep -Eo 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Google-Extended|Bytespider' access.log \
  | sort | uniq -c | sort -rn

Then break each bot's hits down by HTTP status code:

grep 'OAI-SearchBot' access.log | awk '{print $9}' | sort | uniq -c | sort -rn

Three patterns are worth looking for:

1. Lots of training-bot traffic and little search-bot traffic. You are giving content to training pipelines without getting much search visibility back. Your policy should probably change.
2. Search bots hitting 3xx and 404 responses. Crawl effort is going to URLs that no longer exist or point somewhere else. Fix those URLs before you worry about rankings.
3. Search bots only reaching your homepage and top navigation. Deep pages, such as the manual page from the opening example, are not being discovered. That is a discoverability problem, and the next section covers it.

If you run on Cloudflare or a similar CDN, its bot analytics will show the same information already grouped by verified crawler. Use whichever one your team will check again in a month.

Fix the Pages That Waste Crawls

The Cloudflare and OpenAI post points to a handful of site problems that waste AI crawler effort. They are the same problems that have always hurt traditional search. AI crawlers just reveal them faster, because they are newer and have less history about your site.

Soft 404s. A missing page returns HTTP 200 with a "Sorry, nothing here" message. The crawler cannot tell that apart from real content, so it may index the error page or keep coming back to check. Return a real 404 for pages that never existed and a 410 for pages you removed on purpose. In a hypothetical docs portal, this means /docs/v2/install should not return 200 with a "page not found" template after version 2 is retired.

Catch-all redirects. Sending every dead URL to the homepage with a 301 tells crawlers that hundreds of different pages are all the same page. Redirect a retired URL only when it has a real successor, such as an old install guide pointing to the new one. Otherwise, return a 404.

Duplicate URLs. The same content is reachable at /guide, /guide/, /guide?ref=nav, and /v3/guide. Each copy costs a crawl and splits whatever signals the page earns. Pick one URL, add a rel="canonical" tag on the others, keep tracking parameters out of internal links, and redirect trailing-slash variants consistently.

Missing change signals. Crawlers need a reason to come back. Keep accurate <lastmod> dates in your XML sitemap, following the sitemaps protocol, and update them only when the content changes in a meaningful way. A sitemap that marks every page as modified today teaches crawlers to ignore the field. Push protocols like IndexNow can notify participating search engines when a page changes, which helps pages that rarely get linked.

Content hidden behind client-side rendering. If your docs only render after JavaScript runs, some crawlers will see an empty shell. Server-render or pre-render anything you want cited.

Back to the hypothetical manual page with almost no inbound links. You want it to get a real 200 at one canonical URL, appear in the sitemap with an honest lastmod, sit in a docs navigation that links to it, and contain the exact model number and error text people search for. None of that is AI-specific, which is why it works. The underlying work is the same discipline we described for internal agents in data and context engineering for AI agents. Retrieval quality depends on what you give the system to retrieve.

Measure Whether It Worked (Without Promising Rankings)

No one can promise you placement in AI answers. Providers do not publish ranking factors, and the same question can return different citations on different days. What you can measure:

• Crawl coverage. Track the share of your important URLs that search crawlers fetched in the last 30 days and the share of those fetches that returned 200.
• Referral traffic. Watch analytics for referrers from chatgpt.com, perplexity.ai, and similar domains. The numbers are small for most sites, but you can see the trend.
• Answer accuracy. Keep a fixed list of 20 to 30 questions your customers ask, run them through the main AI search products every month, and record whether you are cited and whether the answer is correct. A cited page with a wrong answer usually means your page is ambiguous or out of date.
• Policy drift. Re-check robots.txt after every CMS migration or CDN change. Platform defaults change, and an unnoticed block can remove you from AI search for months.

Frase's guide to AI visibility frames this as a measurement discipline separate from classic SEO, and that is the right way to run it. Give it a named owner, a monthly check, and a short list of fixed questions.

Turning a Crawler Policy Into a Maintained System

The policy itself is a single afternoon of work. Keeping it correct is the harder part: verifying crawlers against IP ranges, catching soft 404s after each docs release, running the monthly answer-accuracy check, and noticing when a new user agent shows up in the logs. Teams that handle this well tend to automate the checks rather than rely on memory.

OpenNash builds that kind of automation. Typical pieces are a log-analysis job that flags new AI user agents and spikes in 3xx or 404 responses, a scheduled agent that runs your fixed question set against AI search products and diffs the citations, and an approval step before any robots.txt change goes live. If your team already has a capable SEO lead and an ops engineer with an hour a month, you probably do not need OpenNash for this. Run the commands above and set a calendar reminder.

If you run a large docs portal or publish at volume, and wrong AI answers about your product are already turning into support tickets, book a call with OpenNash. Bring a week of access logs and your top 20 customer questions, and we will walk through which crawl problems to fix first.

The first step is simple: pull last week's access log tonight, run the user-agent count, and write down which of the three lanes each bot belongs to. Your robots.txt policy follows from that list.