Should I Block AI Crawlers on My Business Website? What to Allow

/ Ai Bots 
Nov 17, 2026 | 6 min read
AI-generated illustration — Empty office corridor lined with identical closed doors, with one different door standing half open at the end
AI-generated image
Olga LescuyerFrontend Developer,SEO specialist

We just saw AI bots crawling our website for the first time. Should we block them?

For most small businesses, no. Block the URLs that cost your server work, not the bots that can put you in ChatGPT, Claude and Perplexity answers. When AI crawlers pushed our own server to 100% CPU, one bot triggered 1,073 rebuilds of a single page in 23 hours. We still kept it allowed.

Why AI crawlers are suddenly showing up in your reports

Many owners, from a Zurich dental practice to a Lausanne architecture office, are only now seeing these visitors. Webflow’s LLM bot insights report shows which AI bots visit a site and which pages they request, based on the site’s server logs. It needs a Team or Enterprise plan with the Analyze add-on.

The report sorts bots into four groups: agentic bots that fetch a page to answer a user’s question, training bots, indexing bots, and bots with several purposes such as Googlebot and Bingbot. One detail saves a worried email. /manifest.json often appears among the most-visited paths, and Webflow says this is expected and unrelated to your content. The report also does not tell you whether a page was cited in an AI answer. It only shows that a bot asked for it.

Not all AI crawlers do the same job

“AI crawler” covers three different kinds of visitor, and each gets its own rule in robots.txt, the plain text file that tells bots what they may read. OpenAI states that each setting is independent. You can allow its search bot so you appear in results while blocking its training bot.

Bot typeExamplesWhat it doesWhat blocking it means
Search and indexingOAI-SearchBot, Claude-SearchBot, PerplexityBotFinds pages to show and link in AI search answersOpenAI: your site is not shown in ChatGPT search answers, though it can still appear as a navigational link. Anthropic: your content is not indexed for Claude’s search.
User-triggeredChatGPT-User, Claude-User, Perplexity-UserFetches a page when a person asks a questionAnthropic: Claude cannot retrieve your page for that user. OpenAI and Perplexity say robots.txt rules may not apply, because a person started the fetch.
TrainingGPTBot, ClaudeBot, Google-ExtendedCollects content that may train future AI modelsSignals that your future content should be left out of training. It does not remove you from search.

Anthropic’s help page gives the same split for its three bots and says they follow robots.txt. Perplexity says PerplexityBot is not used to train foundation models and recommends allowing it. Google-Extended is a special case. It has no user agent of its own and only controls how Google may use content for Gemini. Google says it does not affect inclusion in Google Search and is not a ranking signal.

When to allow AI crawlers

Allow the search and user-triggered bots if you want to be named in AI answers. These are the bots that read your page when someone asks ChatGPT or Perplexity for a recommendation in your field. Block them and you drop out of that answer. If you are unsure where you stand today, start by checking whether ChatGPT already recommends your business.

Allow them too if your pages are marketing pages: services, prices, team, case studies. Those pages exist to be read. We made this call for our own site. Our robots.txt still says yes to search, AI answers and AI training, because we want ChatGPT, Claude and Perplexity to read our pages and cite us.

When to block, and what exactly

Block training bots if your content is the product you sell, such as paid guides or research, and you do not want it used to build AI models. Because the rules are independent, blocking GPTBot or Google-Extended does not take you out of ChatGPT search or Google Search.

Block URL variants that create server work without adding content. Filter, tag and page-number links can turn one page into hundreds of addresses. A crawler that follows them all is doing nothing wrong, but your server pays for every visit.

Slow a heavy bot down before you shut it out. Anthropic supports the Crawl-delay rule for ClaudeBot. It was the only provider in the docs we read to mention it.

Avoid blocking by IP address. Anthropic warns that this may not work as an opt-out, because the bot can no longer read your robots.txt. User agent names can also be faked, as Google points out, so a bot calling itself GPTBot may not be OpenAI. If you use Cloudflare, Perplexity notes that Bot Fight Mode can still challenge its bots, and recommends firewall rules that match both user agent and IP. Changes are not instant either. OpenAI and Perplexity both mention around 24 hours.

What happened when AI crawlers overloaded our own server

This month our WordPress server kept hitting 100% CPU, once for 40 minutes. Our host saw about 2,400 requests a day and suspected a script. The logs showed otherwise: most of the load came from AI crawlers. In 23 hours, our blog list was rebuilt 1,344 times, and 1,073 of those rebuilds were for ClaudeBot. It followed every ?page= and ?tag= link, and our site rebuilt that page on every visit.

We did not block the AI bots. We blocked only the ?page= and ?tag= variants and gave ClaudeBot a crawl delay of 10 seconds. Search engines still find every post through the sitemap. We also keep an llms.txt file that tells AI tools who we are and links to our main pages. With four other small fixes, this was the result:

MetricBeforeAfter
Requests per day2,400+176
CPU faultsRepeated, once 40 minutes at 100%Stopped
AI bots allowed for search, answers and trainingYesYes

Should I block AI crawlers on my business website? A five-step check

  1. Open your bot report and list the pages AI bots request most. In Webflow: Insights, Site overview, AI discovery tab, LLM bots.
  2. Check whether those pages are cached or rebuilt on every visit.
  3. Block junk URL variants such as filters, tags and page numbers before you think about blocking a bot.
  4. Decide on AI training separately from search and answers, since each has its own rule.
  5. If one bot is still too heavy, slow it with Crawl-delay where the provider supports it, or write to the provider from an address on your domain, as Anthropic asks.

Most of these decisions belong in the build, not in a panic after a server alarm. A site made for AI-era search ships with a clean robots.txt, a sitemap, cached pages and an llms.txt file, and someone checks the bot reports after launch. Our guide to SEO, GEO and AEO strategy (ranking in search engines and getting cited by AI tools) shows where crawler access fits. To see where your own site stands, run Pixerry’s free site check. It reads your site the way search and AI engines do.

Run Pixerry's fast and free site check to see how Google and AI assistants read your site, and whether the right crawlers can reach your pages

Check my site
[ Faq ]

Frequently asked questions

Not from ChatGPT search. OpenAI says GPTBot and OAI-SearchBot are set independently, so you can block training and still allow the search bot. Disallowing GPTBot only signals that your content should not be used to train OpenAI’s foundation models.

Google says no. Google-Extended only controls whether your content may be used for Gemini training and grounding. It does not affect inclusion in Google Search and is not a ranking signal.

Not yet. Anthropic began recognising a machine-readable no-AI-training signal on 30 October 2026, based on an IETF draft, and is rolling it out over time. Google plans to support a similar signal in early 2027. For Anthropic, a robots.txt rule that names a specific bot still takes priority.

OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt update. Perplexity says changes can take up to 24 hours. Plan for that delay rather than expecting an instant result.

We keep one on our own site. It tells AI tools who we are and links to our main pages. We have not been able to confirm from an official source how widely AI tools use it, so treat it as a small extra, not a replacement for robots.txt and a sitemap.

Still have questions about this topic? Our team is happy to help you apply it to your project.

Get in touch