Free Tool

Can AI read your website?

Check in seconds whether ChatGPT, Claude, Perplexity, Google and Bing can reach your page: robots.txt rules, firewall blocks, and how much of your content exists without JavaScript.

Run the check

Free, no sign-up. We check one page (the address you enter) and its robots.txt. Results are cached for 10 minutes.

Why this matters

Three reasons AI assistants miss a website.

1. robots.txt rules

A Disallow rule written for one bot, or a blanket rule for all of them, can shut out the crawlers AI search depends on. Each company uses separate crawlers for search and for training, so a rule meant for one can catch the other.

2. Firewall and CDN blocks

Bot protection can turn crawlers away before robots.txt is even read. Cloudflare began blocking known AI crawlers by default on new domains in 2025 (NBC News).

3. JavaScript-only content

OpenAI’s and Anthropic’s crawlers fetch JavaScript files but do not run them, according to a Vercel and MERJ study (Vercel). Pages built entirely in the browser can look empty to them.

Know the crawlers

Search crawlers and training crawlers are not the same.

Blocking a training crawler keeps your content out of model training. Blocking a search crawler keeps you out of the answers.

CrawlerUsed forIf you block it
GooglebotGoogle Search, including AI Overviews and AI ModeYou disappear from Google Search and its AI answers
BingbotBing and Microsoft Copilot; also a source for ChatGPT searchYou disappear from Bing and Copilot, and likely weaken ChatGPT search
OAI-SearchBotChatGPT search resultsChatGPT search stops showing and citing your pages
Claude-SearchBotClaude search resultsClaude search stops indexing your pages
PerplexityBotPerplexity search resultsPerplexity stops surfacing your pages
GPTBot, ClaudeBot, CCBotTraining AI modelsYour content is not used for training; AI search is unaffected if search bots stay allowed
Google-ExtendedA robots.txt control for Gemini, not a crawlerGemini stops using your content; Google Search and AI Overviews are unaffected

← Swipe to see the full table →

Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User as separate crawlers, each with its own robots.txt token (Anthropic). OpenAI and Perplexity work the same way.

How to fix it

Let AI search crawlers in, on purpose.

  • robots.txt: add explicit groups for the search crawlers you want, as in the example.
  • CDN or firewall: in Cloudflare, review Security > Bots and AI Crawl Control, and allow the AI search crawlers.
  • Rendering: make sure your main content is in the HTML the server sends, through server-side rendering or static generation.
  • Then re-run this check to confirm.

Example robots.txt groups that allow AI search crawlers:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://yourbusiness.com/sitemap.xml
FAQ

AI crawler questions.

Block training crawlers if you do not want your content used to train models; that is a legitimate choice. Do not block the search crawlers (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want to appear in AI answers, because they are how assistants find and cite you.
Mostly not. A Vercel and MERJ study found that OpenAI’s and Anthropic’s crawlers fetch JavaScript files but do not execute them (Vercel). If your content only appears after JavaScript runs, those crawlers see an almost empty page. Googlebot does render JavaScript.
robots.txt is a request, not a gate. Firewalls and CDNs can block bots before robots.txt matters. Cloudflare, for example, began blocking known AI crawlers by default on new domains in 2025, regardless of robots.txt. Check your CDN’s bot settings. Our guide Is your website blocking ChatGPT? walks through each blocker.
Close, not identical. We request your page with each crawler’s user-agent from our own servers. Some sites verify crawlers by IP address and may treat the real crawler differently, in either direction. Treat a block here as a strong signal worth checking, not as proof.
It is optional. llms.txt is a proposed summary file for AI tools, and the major search engines do not require it. It is cheap to add, but it does not replace crawlable pages and clear facts.
From our guides

Researched from vendor pages and official sources, with dates and prices checked.

All guides on SEO and AI search →
Go further

Crawlable is step one. Being recommended is the goal.

Our AI Visibility Audit adds what this tool cannot: what ChatGPT, Gemini and Perplexity actually say about your business, which sources they cite, and a ranked fix list.

Get an AI visibility auditSee SEO, AEO & GEO services
  • Fixed quote within one business day
  • Free review, no obligation
  • You own everything we build
Chat