Are You Blocking AI Search? A Small Business Guide to robots.txt and AI Crawlers

By the Oski Marketing team · Updated 5 October 2026 · 6 min read

You might be blocking AI search without knowing it. Each AI company runs separate crawlers: some build the search index that decides whether your site can be shown and cited, others only collect training data. Check your robots.txt file and any security settings, allow the search crawlers, and decide separately whether you mind your content being used for training.

What is robots.txt and why does it matter for AI search?

Robots.txt is a small text file at the root of your website that tells crawlers which pages they may visit. Google describes it as a file that tells search engine crawlers which URLs the crawler can access. Well-behaved crawlers, including the main AI search crawlers, read it before they look at your site.

For AI search it matters because a single line can shut out an assistant completely. OpenAI says sites that opt out of its search crawler will not be shown in ChatGPT search answers. Many small business sites carry rules added years ago by a developer, a plugin or a hosting provider that nobody has looked at since.

What is the difference between AI search crawlers and training crawlers?

Search crawlers fetch your pages so an assistant can show and cite them in answers. Training crawlers collect content to train future AI models. Blocking a training crawler does not remove you from the matching assistant's search answers, but blocking a search crawler can. These are the names published by each company:

Sources: OpenAI, Anthropic, Perplexity and Google. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".

How do you check your robots.txt in five minutes?

You can check it in a browser with no tools. Follow these steps:

1. Type your web address followed by /robots.txt, for example yourbusiness.co.uk/robots.txt.
2. If you see "not found", you have no file and crawlers are allowed by default. Skip to the hosting check below.
3. Look for User-agent: * followed by Disallow: /. That blocks every crawler that obeys the file, including Google. Fix this first.
4. Search the page for each search crawler name: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot. If any appears above a Disallow: / line, it is blocked.
5. Note any training crawlers that are blocked, such as GPTBot, ClaudeBot or Google-Extended. That is a choice you can keep or change, and it does not stop search answers.
6. If you use Google Search Console, open its robots.txt report to see the version Google last fetched.

What does a sensible robots.txt look like for a local business?

For most local businesses, a sensible file allows search crawlers and makes a deliberate choice about training. If you are happy to be found and do not mind training use, you may need nothing at all. If you want to be found but would rather your content was not used for training, a file like this does it:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.yourbusiness.co.uk/sitemap.xml

This blocks three training crawlers by name and leaves everything else, including OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot, free to visit. Keep any existing rules your developer added for admin or checkout pages. If your website runs on a platform that manages robots.txt for you, change it in the platform's settings rather than uploading a file.

Can your hosting or security service block AI crawlers?

Yes, and it can do it without touching robots.txt. Cloudflare, which sits in front of many websites, announced in July 2025 that every new domain would be asked whether to allow AI crawlers when it signs up. Its managed robots.txt feature adds blocks for training crawlers such as GPTBot, ClaudeBot, Google-Extended and others, while marking search as allowed. Firewalls and bot protection can also turn crawlers away with an error page, which a robots.txt check will never show.

OpenAI's help centre makes the same point from the other side: to be eligible for ChatGPT search, you should allow its search crawler and confirm that your host or content delivery network allows traffic from OpenAI's published searchbot IP addresses. If your site uses Cloudflare or a similar service, ask whoever manages it to check the AI crawler and bot settings, and to confirm the search crawlers are not challenged or blocked.

Does robots.txt hide pages from search?

No, and this is a common mistake. Google says robots.txt is not a mechanism for keeping a web page out of Google, because a blocked page can still be indexed if other sites link to it. To keep a page out of search, use a noindex tag, a password or remove the page. Use robots.txt for managing crawlers, not for privacy.

Once crawler access is sorted, the rest of our AI search visibility checklist covers the other steps, and our guide to getting recommended by ChatGPT explains how assistants choose businesses. If you are launching a new site, add a robots.txt check to your website launch checklist. If you would rather someone checked your file and security settings for you, we do this as part of our website work at Oski Marketing.

What to do this week

• Open yourdomain.co.uk/robots.txt and look for any Disallow: / under User-agent: * or under a search crawler's name.
• Decide whether you want to block training crawlers, and write that choice down so future developers keep it.
• If your site uses Cloudflare or a similar service, ask whoever manages it to check the AI crawler settings.
• Make sure your robots.txt points to your sitemap, and that the sitemap address works.