Configure robots.txt for AI Crawlers

Also known as: AI Bot Access Control, Crawler Permissions, Bot Allow/Disallow

Audit and update robots.txt to explicitly allow or block specific AI user agents like GPTBot, ClaudeBot, and OAI-SearchBot based on your strategy.

What is Configure robots.txt for AI Crawlers?

Impact Mechanism:
RAG & Retrieval
Category:
Technical & Schema
Effort:
Low
Skill needed:
Technical & Dev
Evidence level:
Proven

How does it work?

  1. 1. Sort each AI bot by purpose

    Search bots: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot. User fetchers: ChatGPT-User, Claude-User, Perplexity-User. Training controls: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended.

  2. 2. Allow the search bots by name

    In /robots.txt at each host's root, add a group per bot, e.g. 'User-agent: OAI-SearchBot' then 'Allow: /'; a named group replaces the '*' group.

  3. 3. Decide on training bots separately

    To opt out of training, add 'Disallow: /' groups for GPTBot, ClaudeBot, Google-Extended and Applebot-Extended; this does not affect Google or Apple search.

  4. 4. Know what robots.txt cannot stop

    ChatGPT-User and Perplexity-User may ignore robots.txt because a person asked; robots.txt is a request, not access control, so use your firewall if needed.

  5. 5. Set Bing and Copilot use per page

    Bingbot feeds Copilot; from 1 Feb 2027 opt out per page with <meta name="Bingbot" content="ai-grounding:disallow">; robots.txt has no grounding setting.

  6. 6. Test the file and wait a day

    Open yoursite.com/robots.txt in a private window, check it in Search Console's robots.txt report and Bing's robots.txt tester; OpenAI and Perplexity need about 24 hours.