What is robots.txt, and what can it actually do?
robots.txt is a plain text file at the root of your site (yoursite.com/robots.txt). It tells each crawler which parts of the site it may visit. Google, OpenAI and Anthropic all state that their crawlers follow it.
Three limits to keep in mind. The file is not a lock: a disallowed page stays open to anyone who has its address, and Google notes it may still show the address of a blocked page in its results, without a snippet. It only applies to the host it sits on: www.yoursite.com and shop.yoursite.com each need their own. And a crawler that opens a page because a user asked may not read it at all: OpenAI says so for ChatGPT-User.
Which AI crawlers visit your site, and what happens if you block them?
Each AI company now splits its crawlers by purpose. The table sums up their own documentation, read on October 30, 2026.
| Company | Name in robots.txt | What it does | If you block it |
|---|---|---|---|
| OpenAI | GPTBot | Collects pages that may be used to train models | You signal that your content should not be used for training. No effect on ChatGPT search. |
| OpenAI | OAI-SearchBot | Surfaces websites in ChatGPT search answers | Your site is no longer shown in those answers, except as a plain navigational link. |
| OpenAI | ChatGPT-User | Opens a page when a user asks | robots.txt rules may not apply, according to OpenAI. |
| Anthropic | ClaudeBot | Collects pages that may be used for training | Your future content is excluded from training data. |
| Anthropic | Claude-SearchBot | Crawls the web for Claude's search | Lower visibility in Claude's search results. |
| Anthropic | Claude-User | Opens a page when a user asks a question | Claude no longer reads your pages to answer its users. |
| Googlebot | Crawls the web for Google Search | You drop out of Google. Never block it by mistake. | |
| Google-Extended | A control token with no crawler of its own: Gemini model training and grounding of answers in Gemini apps | No effect on inclusion in Google Search or on ranking. Your pages are no longer used for training or grounding in Gemini. |
The practical takeaway: blocking a training crawler does not remove you from AI assistants' search answers. OpenAI states it plainly: the settings are independent, and a site can allow OAI-SearchBot while refusing GPTBot. Google-Extended is the exception, since one name covers both training and the use of your pages in Gemini's answers.
What file should a small business that wants to be cited write?
Our reading, for most small businesses: let every search crawler in, AI assistants' included, and decide on training separately. A site that sells a service has little to lose by being cited, and training is a matter of principle about your content. Here is a file that refuses training at OpenAI and Anthropic and leaves everything else open:
User-agent: *
Allow: /
Disallow: /client-area/
# Model training: refused
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
Sitemap: https://www.yoursite.com/sitemap.xml
If training does not bother you, keep only the first block: a file open to all lets AI crawlers in without naming them. Add Google-Extended to the refusing group only if you accept no longer being a source for Gemini's answers.
One trap comes up often when you add a group for a specific crawler. Google's documentation is explicit: a crawler follows only the group that names it, and the general rules (User-agent: *) are not added to it. If you create a group "User-agent: OAI-SearchBot" with "Allow: /", that crawler no longer reads the line protecting /client-area/. Copy the paths that must stay closed into every group.
What does the shared signal announced on October 30, 2026 change?
The signal looks like a made-up crawler name, __usage_ai-training, described in a draft standard published the same day at the IETF by an Anthropic engineer. You write it like any other group:
User-agent: __usage_ai-training
Disallow: /
This group states a preference against training, for the whole site or for some paths. It blocks no crawler: it says what you accept being done with your pages, not who may read them. The general "User-agent: *" group does not count as an answer: without a __usage_ai-training line, no preference is expressed.
The announced timeline: Google aims to support it in 2027, with no date. When the signal refuses training and the file does not name Google-Extended, Google plans to treat it as a refusal addressed to Google-Extended. Microsoft will start applying its new preferences (training on one side, use in answers on the other) on February 1, 2027, according to Search Engine Roundtable's report. Each company's own settings keep working.
Our advice: you can add the group now, since crawlers that do not know it simply ignore it, but keep your GPTBot and ClaudeBot lines. The text is still a draft, valid until May 3, 2027, and it may change before then.
What we found in the files of four AI agencies
On October 30, 2026, we read the robots.txt of four French agencies selling AI agents to businesses: ours and three agencies we track. Four files, three approaches.
- Two files open to all, naming no AI crawler, ours included. Ours closes only two paths (/api/ and the diagnostic report page) and lists the sitemaps of the French and English sites.
- One file that names 16 crawlers one by one to allow them (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot and others). The effect is the same as an open file; the intent is simply written down.
- One file that blocks 32 crawlers, including GPTBot, ClaudeBot, Google-Extended and GoogleOther, with an extra line accepting search and refusing training. Its comments show it is written by Cloudflare's tool, the provider serving the site.
The third case is the most useful for a business owner. The file lets OAI-SearchBot and Claude-SearchBot through, so the site stays readable for ChatGPT and Claude search. But it refuses Google-Extended which, according to Google's documentation, also removes the site from the sources of Gemini's answers. It may be a deliberate choice. What the file mainly shows is that a network provider setting can write robots.txt in place of the agency that built the site: the person who thinks they own it may never have read it.
How do you check and fix yours?
- Open yoursite.com/robots.txt in a browser, then the same address on each subdomain (shop, blog, client area).
- Look for "Disallow: /" under "User-agent: *" or "User-agent: Googlebot": it closes the whole site to search, and it is the first error to fix.
- Find the AI crawler names and compare them with the table above: does what is blocked match your choice?
- Ask your provider what writes this file: the site software, a plugin, or a service such as Cloudflare. A change made in the wrong place will be overwritten.
- After the change, read the live file again. OpenAI says it takes about 24 hours for ChatGPT search to take it into account.
It is the same approach as the PHP version to ask your web host about: know who holds each setting of your site, and have it in writing.
Do you need an AI agent to monitor this?
No. The file rarely changes: one review per quarter, plus one at each change of provider or hosting, is enough. The real question for a small business runs the other way: what access you give the AI tools working for you. It is covered in the four questions to ask before connecting an AI agent to your tools, and it is one of the points we settle in writing during a scoping phase.
To see what our agents handle in a small business, and what stays approved by a person: AI agents for small businesses.
FAQ
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is used for model training. ChatGPT search relies on OAI-SearchBot, and OpenAI states the two settings are independent: you can refuse one and allow the other.
Does blocking Google-Extended hurt my Google rankings?
No, according to Google: the token has no effect on a site's inclusion in Google Search and is not a ranking signal. It does remove your pages from Gemini model training and from the sources used in Gemini apps' answers.
Does robots.txt erase what has already been collected?
The documentation we read is about what comes next. Anthropic writes that blocking ClaudeBot excludes your future content from training; OpenAI, that blocking GPTBot indicates your content should not be used for training. Neither mentions removing what was collected before.
Should I add the __usage_ai-training signal now?
You can: crawlers that do not know it ignore it. Google plans to support it in 2027, and Microsoft says it will apply its new AI preferences from February 1, 2027. Until then, keep your GPTBot and ClaudeBot lines, which remain the recognized way to refuse training.
Sources, all read on October 30, 2026: Google Search Central, "Supporting a shared AI training control for websites", John Mueller, October 30, 2026 (developers.google.com) · K. Kelley (Anthropic), "Machine-Readable Training Preference Signal", IETF draft draft-kelley-aipref-training-preference-signal-00, October 30, 2026 (datatracker.ietf.org) · OpenAI, "Overview of OpenAI Crawlers" (developers.openai.com); OpenAI's October 30 announcement was not accessible, Unite.AI's coverage was read (unite.ai) · Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?", April 7, 2026 (support.claude.com) · Google, "Google's common crawlers", updated October 20, 2026, and "How Google interprets the robots.txt specification" (developers.google.com) · Search Engine Roundtable, "Microsoft Bing New AI New Controls With AI Training From AI Search Grounding", Barry Schwartz, October 30, 2026; Microsoft's own post was not read. The four robots.txt files were read by Équipage IA on October 30, 2026. No client case is cited.