robots.txt for AI search, training, and user access

Distinguish each platform user agent while keeping private application areas blocked.

Author: By Equipe MencionAI
Published: Published on June 5, 2026
Reviewed: Reviewed on September 9, 2026

Separate each purpose

Googlebot controls participation in Search, including AI Overviews and AI Mode. Google-Extended covers some generative uses outside Search. OAI-SearchBot and Claude-SearchBot support search; GPTBot and ClaudeBot support training; ChatGPT-User and Claude-User support user-triggered visits. Their rules are independent.

Public vs private

Disallow /admin, /dashboard, checkout, and API routes. Keep marketing, blog, FAQ, and legal pages crawlable.

User-agent: Googlebot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

User-agent: Google-Extended
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

User-agent: ChatGPT-User
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

User-agent: GPTBot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

User-agent: Claude-SearchBot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

User-agent: Claude-User
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

User-agent: ClaudeBot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

User-agent: *
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Validate the published policy

robots.txt controls crawling, not authorization for private content. Test every user agent and keep server-side authentication. llms.txt can describe public URLs, but it does not replace sitemaps, internal links, or access controls.