robots.txt for GPTBot and co., and what llms.txt can and can't do.

The best free resources

1. Overview of OpenAI Crawlers — — (OpenAI), OpenAI (live 2026-11)
   OAI-SearchBot vs GPTBot vs ChatGPT-User: allow search, decide on training separately

2. Does Anthropic crawl data from the web, and how can site owners block the crawler? — — (Anthropic), Anthropic (Claude Help Center) (updated 2026-11)
   ClaudeBot, Claude-User, Claude-SearchBot: what blocking each one costs

3. Perplexity Crawlers — — (Perplexity), Perplexity (live 2026-11)
   PerplexityBot vs Perplexity-User; user fetches generally ignore robots.txt

4. List of Google's common crawlers (Google-Extended) — — (Google), Google Search Central (live 2026-11)
   Google-Extended controls Gemini training/grounding, not Search or AI Overviews

5. The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals — João Tomé, Cloudflare (2025-08-29)
   ~80% of AI crawling is for training; the data behind block-or-allow decisions

6. We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read — Louise Linehan, Ahrefs (2026-06-15)
   Log data: 97% of llms.txt files got zero requests; no major AI platform has committed to reading it

7. The /llms.txt file — Jeremy Howard, llmstxt.org (2024-09-03 (mod. 2026-08-10))
   The spec itself, for readers who still want to add one

Strategies for this step

• Configure robots.txt for AI Crawlers

• Deploy an llms.txt File

See all GEO strategies →

Curated by effectiveGEOstrategy.com. Every link checked 2026-11-10. Know a better free resource? Tell us.