AI crawler access directory for GEO audits
AI Crawler Access Directory for GEO Audits
A source-backed directory of major search and AI crawler signals that SaaS teams should review before testing AI search visibility.
What this directory covers
This directory summarizes the crawler and fetcher signals that matter during a SaaS GEO audit. It focuses on public documentation from Google, OpenAI, Perplexity, and Anthropic. The goal is not to list every bot on the web; it is to help teams avoid accidentally blocking the systems they expect to discover, retrieve, or cite their public pages.
- Use this directory as a review companion to the Robots and AI Crawler Checker.
- Verify official documentation before changing production robots.txt or WAF rules.
- Separate search visibility crawlers from model-training crawlers when a platform documents that distinction.
- Remember that passing crawler checks does not prove page quality, originality, or future citations.
Crawler and fetcher review table
The table below is intentionally conservative. It describes how a publisher should think about access during an AI search visibility audit, not how to manipulate retrieval systems. If a site uses a firewall, CDN bot fight mode, geo blocks, or private staging protection, test access with logs after any robots.txt change.
| Provider | Documented agents to review | Visibility relevance | Audit note |
|---|---|---|---|
| Googlebot and related Google crawlers/fetchers | Google Search discovery, rendering, indexing, and product fetches | Do not block Googlebot on public pages that need Search or AI Overview discovery; verify Google requests when security rules are strict | |
| OpenAI | OAI-SearchBot, GPTBot, ChatGPT-User | OAI-SearchBot is relevant to ChatGPT search visibility; GPTBot relates to model training; ChatGPT-User is user-triggered | Use OAI-SearchBot when thinking about ChatGPT search access; do not assume GPTBot access alone controls search visibility |
| Perplexity | PerplexityBot, Perplexity-User | PerplexityBot supports search-result surfacing; Perplexity-User supports user-requested fetches | If Cloudflare or AWS WAF is enabled, combine user-agent review with official IP sources and log monitoring |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User | Claude-SearchBot supports search relevance; Claude-User supports user-directed retrieval; ClaudeBot relates to model training | Blocking search or user agents can reduce visibility in user-facing retrieval while still leaving other bots unaffected |
Source-backed notes
Crawler guidance changes. A useful audit should name the source used, the date it was reviewed, and the production file or WAF rule that was changed. This keeps the work verifiable for future editors and avoids cargo-cult robots.txt snippets copied from old blog posts.
https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers
OpenAI crawlers and user agents
Used for OAI-SearchBot, GPTBot, ChatGPT-User, published IP ranges, and the distinction between search, model training, and user-triggered access.
https://platform.openai.com/docs/bots
Perplexity crawlers
Used for PerplexityBot, Perplexity-User, WAF configuration notes, official IP sources, and the reminder to monitor logs.
https://docs.perplexity.ai/docs/resources/perplexity-crawlers
Anthropic crawler help article
Used for ClaudeBot, Claude-User, Claude-SearchBot, Crawl-delay support, robots.txt controls, and visibility caveats.
https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
Robots.txt review patterns
A GEO audit should begin with the simplest public-page baseline: allow crawling, expose a sitemap, and avoid page-level noindex unless the page should stay out of search. Selective opt-outs may be legitimate, but they should be documented because blocking the wrong agent can remove a page from the answer surface a team wants to test.
| Pattern | Example intent | Review before publishing |
|---|---|---|
| Visibility-friendly baseline | Allow public marketing pages and submit sitemap | Confirm pages are public, canonical, and not blocked by meta robots |
| Training opt-out with search access | Block documented training agents while allowing search agents where supported | Confirm platform-specific distinctions in official docs |
| WAF allowlist | Allow verified crawler IPs through Cloudflare or AWS WAF | Use official IP endpoints and monitor logs instead of user-agent matching alone |
| Private or staging protection | Prevent unpublished pages from being crawled | Do not test AI search visibility on pages that require login or block public crawlers |
How this reduces low-value content risk
A crawler directory creates original utility because it turns official platform documentation into an actionable SaaS audit workflow. Instead of repeating a generic definition of robots.txt, the page explains which access signals to review, which sources to verify, where WAF rules can interfere, and why crawler access still has to be paired with helpful visible content.
- The page is source-backed and dated instead of copied from a generic SEO checklist.
- The table connects each crawler group to a specific audit decision.
- The limitations prevent users from treating crawler access as a citation guarantee.
- The page links naturally to the crawler checker, source notes, and full SaaS GEO workflow.
Use this with a tool
Turn this page into a concrete review by starting with the visibility checker, schema generator, or crawler checker.