Is your website blocking ChatGPT? How to check, and the hidden blockers to fix

The four blockers

When a business asks us why ChatGPT or Perplexity never mentions it, the cause is usually one of four technical blockers, not the quality of the business. Each one has a different fix, so it is worth knowing which one you have.

1. robots.txt rules that tell AI crawlers to stay out. Sometimes written on purpose, often added by a plugin, a theme or a security tool.
2. Firewall or CDN blocking, where bot protection refuses AI crawlers before robots.txt is even read.
3. Content that needs JavaScript, so crawlers that do not run scripts see an empty page.
4. noindex or snippet limits, which keep pages out of Google’s AI Overviews and AI Mode.

Which crawlers matter

Every AI company runs more than one crawler, and they do different jobs. The ones that decide whether you can be found and cited are the search crawlers. The training crawlers only decide whether your content is used to train future models.

Two details catch people out. First, blocking GPTBot does not remove you from ChatGPT search: OpenAI says the settings are independent (OpenAI). Second, Google-Extended has no effect on Google Search or AI Overviews: Google says it does not affect inclusion in Search, and AI Overviews use ordinary Googlebot crawling (Google).

Blocker 1: robots.txt

Open yourdomain.com/robots.txt in a browser. Look for any group that names one of the crawlers above, or a catch-all group, followed by Disallow: /:

User-agent: OAI-SearchBot
Disallow: /

A crawler follows the group that names it most specifically, and only falls back to the User-agent: * group when nothing names it (RFC 9309). So a site can welcome all bots under * and still block one AI crawler by name further down the file. Read the whole file.

If you want to appear in AI search but keep your content out of model training, this is a reasonable starting point. The search crawlers are not named, so they fall through to the last group and are allowed:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

On jeedesigns.com we made the opposite choice and allow every AI crawler, training included, because we want our guides and our facts to be learned and cited. While writing this guide we checked our own file and found it named every major AI crawler except Claude-SearchBot. The catch-all group already allowed it, but we added it by name so a future edit cannot quietly exclude it.

After you change robots.txt, OpenAI says ChatGPT search takes about 24 hours to adjust (OpenAI).

Blocker 2: firewalls and CDNs

robots.txt is a request. A firewall is a gate. If your CDN or security plugin refuses AI crawlers, they never get as far as reading robots.txt, and the file can say “allow” while the server answers “403 Forbidden”.

The most common case is Cloudflare. Since July 1, 2025, every new domain added to Cloudflare is asked whether to allow AI crawlers and starts with them blocked by default (Cloudflare). If your site moved to Cloudflare after that date and nobody changed the setting, AI crawlers are probably blocked. The fix is in your domain’s bot or AI crawler settings in the Cloudflare dashboard, where you can allow the search crawlers you want.

Security plugins, hosting firewalls and bot-protection services can do the same. To test from your own computer, request your home page with a crawler’s name as the user agent and look at the status code:

curl -I -A "OAI-SearchBot" https://yourdomain.com/

A 200 means the page was served. A 403, 429 or 503, or a challenge page, means something in front of your site is refusing that crawler. The test is close to a real crawler visit, not identical: some firewalls verify real crawlers by their IP address and may treat them differently in either direction.

Blocker 3: content that needs JavaScript

Many modern sites send a nearly empty HTML page and build the content in the browser with JavaScript. People see a normal page. Most AI crawlers do not.

Vercel and MERJ analysed AI crawler traffic and found that none of the major AI crawlers rendered JavaScript, including OpenAI’s, Anthropic’s, Meta’s, ByteDance’s and Perplexity’s. ChatGPT’s crawler fetched JavaScript files in 11.50% of its requests and Claude’s in 23.84%, but neither ran them. Google’s Gemini, which uses Googlebot’s infrastructure, and Applebot do render JavaScript (Vercel, December 2024).

To check, turn JavaScript off in your browser (in Chrome: Developer Tools, then the command menu, then “Disable JavaScript”) and reload your page. What you see now is roughly what most AI crawlers see. If your services, prices, opening hours or answers disappear, the fix is server-side rendering or static generation, so the HTML arrives complete. This site is generated that way, which is why this page reads the same with JavaScript off.

Blocker 4: noindex and snippet limits (Google)

Google’s AI Overviews and AI Mode only show pages that are indexed and eligible to appear in Search with a snippet. Google says there are no extra requirements, no special files and no special schema (Google).

That makes the blockers the ordinary ones: a noindex tag left on from a staging site, or nosnippet, data-nosnippet and max-snippet settings that stop Google quoting your text. Check the page source for noindex and nosnippet, and check the page in Google Search Console’s URL Inspection tool.

Test all four in one go

Our free AI crawler checker runs these tests together. It reads your robots.txt the way RFC 9309 describes, requests your page as five live AI crawlers from our server, checks how much of your text is readable without JavaScript, and looks for noindex, structured data and an llms.txt file. It gives you a score out of ten and the specific fix for each failed check.

If it finds a blocker you cannot fix yourself, or your site is readable but AI assistants still never mention you, the problem is no longer access. It is evidence and reputation, which is what our AI Visibility Audit covers.