How to block AI crawlers on your own website
Everything a crawler can reach.
robots.txt entries for GPTBot, Google-Extended, ClaudeBot and the rest.
Everything a crawler can reach.
robots.txt entries for GPTBot, Google-Extended, ClaudeBot and the rest.
Add crawler directives to robots.txt at the root of every domain you own.
User-agent: GPTBot Disallow: /
User-agent: Google-Extended Disallow: /
User-agent: ClaudeBot Disallow: /
User-agent: CCBot Disallow: /
User-agent: Applebot-Extended Disallow: /
User-agent: Bytespider Disallow: /
User-agent: PerplexityBot Disallow: /
Google-Extended controls training use without affecting Search ranking. Blocking Googlebot itself would deindex you, so do not.
robots.txt is a request, not a wall. Server-level blocking by user agent or IP is the only enforcement, and the crawler list goes stale fast. Re-check it a few times a year.
robots.txt entries for GPTBot, Google-Extended, CCBot, ClaudeBot, and friends. Compliant crawlers only.
Every service on that list buries its opt-out somewhere different. Don't Train On Me puts all of them in one checklist in a panel beside your browser, walks you through them one at a time, and helps you confirm your privacy settings on the ones that expose a setting it can read. Free, no account, and nothing you tick leaves your machine.