robots.txt for AI crawlers

Allow discovery without exposing private app areas.

Named AI user-agents

Reference crawlers your buyers care about: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bingbot. Use explicit Allow/Disallow only for sensitive paths—not the entire site.

Public vs private

Disallow /admin, /dashboard, checkout, and API routes. Keep marketing, blog, FAQ, and legal pages crawlable.

User-agent: GPTBot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

User-agent: ClaudeBot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

User-agent: PerplexityBot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

User-agent: *
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Pair with llms.txt

robots.txt controls fetch permission; llms.txt curates which URLs matter most for AI summaries. Use both for a complete GEO technical base.