AI Crawler Access Tester: The Same Test Lumos Runs on Every Page Weekly

Why crawler access is the first GEO gate

Before AI engines can cite your page, they have to read it. ChatGPT, Claude, Gemini, and Perplexity all dispatch crawlers — each with a distinct user-agent string and purpose — and each one checks your robots.txt before fetching content. Block them there and nothing else matters: not your schema, not your llms.txt, not the quality of your prose. You're invisible.

In our 2026 audit of 1,000 mid-market sites, 41% were blocking at least one of the 13 major AI crawlers, almost always by accident. The usual culprits: a User-agent: * Disallow: / left over from a staging site, a Cloudflare bot-protection toggle enabled by default, or a CMS plugin that "secured" the site without anyone noticing AI bots were caught in the net.

The 13 AI crawlers this tool checks

Each bot has a specific role, and some sites should allow them all while others may legitimately block a subset:

• GPTBot — OpenAI's training crawler. Reads pages to train future GPT models. Blocking this opts you out of GPT training but does not affect ChatGPT citations.
• OAI-SearchBot — ChatGPT's real-time search crawler. This is the one that gets you cited in ChatGPT answers. Must be allowed for ChatGPT visibility.
• ChatGPT-User — fires when a ChatGPT user clicks a link inside an answer. Allowing it is essentially required.
• ClaudeBot — Anthropic's main crawler for Claude, including Claude.ai search-time answers.
• anthropic-ai — Anthropic's training crawler (separate from ClaudeBot).
• Claude-Web — Anthropic's web fetch user-agent when Claude visits a page on a user's behalf.
• PerplexityBot — Perplexity's main crawler. Must be allowed to appear in Perplexity answers.
• Perplexity-User — fires when a Perplexity user clicks a link.
• Google-Extended — Google's training opt-out flag for Gemini. Blocking this opts you out of Gemini training but does not block AI Overviews (those use Googlebot).
• Googlebot — classic Google + powers AI Overviews and Bard/Gemini search-time answers.
• Applebot-Extended — Apple Intelligence training opt-out.
• Bingbot — Bing + Copilot. Must be allowed for Copilot citations.
• Bytespider — TikTok / Doubao training crawler. Optional; many western brands choose to block it.

What good results look like

A site optimized for GEO will show all 13 bots as Allowed on the root path. Exceptions:

• Some brands intentionally block GPTBot, anthropic-ai, Google-Extended, and Applebot-Extended (the training-only bots) while keeping search-time bots (OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, Bingbot) allowed. This is the "opt-out of training, opt-in to citation" pattern.
• A few brands block Bytespider to avoid TikTok training without affecting western AI engines.

If you see Allowed across the board, you've cleared the first gate. If you see Blocks, fix them — the next sections show what to do.

Common mistakes

Blocking by accident via User-agent: * Disallow. A blanket disallow catches every bot, including AI crawlers. Add explicit Allow rules for the AI user-agents above.

Blocking GPTBot but not OAI-SearchBot. A common pattern, but ensure you mean it. If your goal is ChatGPT visibility, OAI-SearchBot is what matters — GPTBot only affects training.

Cloudflare AI-bot toggle. Cloudflare's dashboard added an "Block AI bots" toggle in 2024 that defaults to ON for new sites. robots.txt can say Allow all you want — Cloudflare will still 403 the request.

Serving robots.txt as HTML or behind auth. Both cause AI crawlers to give up. Force text/plain and public access.

Stale rules. A robots.txt from 2018 won't mention any of the AI user-agents that emerged in 2023-2024. Default-allow behavior usually saves you, but generic blocks can still catch them.

Trusting the tool result for one path. This tester checks your root (/). If you've blocked AI bots on /blog or /docs, you need to test those paths too — those are where your highest-citability content usually lives.

After you test

1. If all 13 show Allowed: you're cleared at the robots.txt layer for the root. Move on to schema, citability, and per-path checks.
2. If any show Blocked: edit your robots.txt to add explicit Allow: rules for the blocked user-agents, then re-run this tester.
3. If your CDN or WAF blocks despite robots.txt allowing: check Cloudflare → Security → Bots → "AI Scrapers and Crawlers"; in Akamai/Imperva, look for bot-management rules tagged "AI" or "scraper."
4. Pair with Page Citability Checker — access is necessary but not sufficient.

What Lumos does continuously (vs this one-shot test)

This page tests your root path on demand. Useful, but a single snapshot. The Lumos platform takes the same test and runs it as continuous infrastructure:

• Every URL on your site, not just /. The tester here checks /. Lumos runs the 13-bot test against every URL it discovers — blog posts, product pages, docs, anything in your sitemap. That's where the regressions hide: /blog blocked while / stays open.
• Weekly, not when you remember. Configurations drift. A CMS plugin updates, a Cloudflare toggle flips on, a content team adds a robots rule. Lumos re-runs the full bot-access test every week so you catch the change in days, not quarters.
• Day-of alerts. When any of the 13 bots flips from Allowed to Blocked, Lumos sends an alert with the diff — which bot, which paths, what changed in robots.txt or the network response. No more "we noticed traffic dropped in ChatGPT three months ago."
• robots.txt + network layer in one report. This standalone tool only inspects robots.txt. The Lumos platform also fetches each URL as each user-agent and reports the actual HTTP response — so Cloudflare 403s and WAF blocks surface alongside robots.txt rules.
• Tied to your citation and visibility data. When a bot becomes blocked, Lumos correlates the day with your visibility score in that engine — so you see the business impact, not just the technical event.

This standalone tool is the same logic, run once at the root. If you want the continuous, every-URL, alerted version, the Lumos platform is built for it.