This post shows you how you can check if your web host blocks AI crawlers. If you want your website to show up in AI-generated results (like ChatGPT), you don’t want to block these bots! Some hosts block AI crawlers by default and don’t tell you unless you ask. So here we explain how to check your host’s response to IA crawlers using your own computer.
What do AI crawlers do?
The crawlers scan your website’s public content: they visit your pages, read your copy and follow the links on your site. This content is then added to the AI tools’s internal index (a big database). When someone asks a question in a tool like ChatGPT, the information from the index can be used to answer the question.
So you can look at AI crawlers as information-gathering robots.
What happens if your host blocks them?
On one hand your website’s content is protected from being scraped or reused by AI. But the downside is that you may lose AI-driven exposure for your website. Your content won’t be quoted or linked in AI answers and new blog posts or updates won’t get picked up by AI.
Can AI still recommend my website if crawlers are blocked?
Yes, but your chances are reduced.
If AI crawlers can’t scan your site directly, they may reference other websites that mention you, such as online directories and social media pages.They might also pull information from older datasets from before the crawler was blocked, meaning outdated information could show up instead of current content.
All this to say it’s less than ideal.
Should you block AI crawlers?
It depends on your goals!
- If you’re an artist and don’t want your style copied, blocking AI bots makes sense.
- If you work in the legal field and want to avoid misrepresentation by AI incorrectly summarizing legal advice etc, that also can be a good resaon to block the crawlers
But for most small businesses and entrepeneurs who want visibility, allowing AI crawlers gives you a better chance of getting recommended.
It’s a trade-off: you allow AI to scan and repurpose your content and in exchange you have a better chance of getting quoted and referred.
How to test if your web host blocks AI crawlers
You can run a simple test using the Terminal (macOS) or PowerShell (Windows).
Open your terminal
- For Windows: Press the Windows Key + X. Then choose Windows PowerShell, Terminal or Windows Terminal.
- For Mac: Press Cmd + Space, type “Terminal” and hit Enter.
Run a command for each AI crawler
Here is an example for GPTBot, used by OpenAI and ChatGPT.
curl -A “GPTBot” -I https://yourwebsitehere.com
You can copy and paste this command in your terminal and press Enter.
When successful, the result will show you a result that includes this:
HTTP/2 200
Here’s an example of what that looks like in the Mac Terminal:
What the status codes mean
| Response Code | Meaning |
| 200 | ✅ The crawler is allowed |
| 403 | ❌ The crawler is blocked |
| 301 | 🔁 This URL is redirected, so instead check and test the new URL |
| 503 | ⚠️ This error may be caused by a CAPTCHA (used by some hosts!), other possibility is that the server is temporarily unavailable. |
Keep in mind that a 403 error can also be because of timing restrictions, so if you want to be 100% sure, you can test it a couple of times instead of just once.
Common AI crawlers to test
Use the same curl command as above, but replace "GPTBot" with any of these:
- GPTBot: used by ChatGPT/OpenAI
- CCBot: used by Common Crawl, which is a data source for OpenAI, Meta, Amazon and others.
- PerplexityBot: used by Perplexity.ai
- ClaudeBot: used by Claude
- Applebot: used by Apple for Siri and Spotlight suggestions
- Facebookexternalhit: used by Facebook and Instagram
- Amazonbot: used by Amazon for Alexa and internal AI systems.
If you’re not feeling like testing them all, focus on GPTBot and CCBot, because they cover the biggest platforms.
Why do some web hosts block AI crawlers?
Most hosts say it’s because they want to protect your copy and content from being scraped. If this is the case, they’ll usually offer a setting in your dashboard to opt in or out. But some hosts block bots by default and don’t tell you, likely so they can save on server resources.
Contact your web host and ask to allow them!
Web hosts have the ability to lift their imposed block on AI crawlers! So if you notice after testing that your host blocks bots, ask them to allow specific bots (like GPTBot and CCBot).
Many will update your settings if you ask. Clients of ours have done so successfully.
And if they refuse to cooperate? You can always switch providers.
Bonus tip: using Cloudflare?
If you use Cloudflare, be aware that as of July 1, 2025, they block AI crawlers by default. To allow them, you’ll need to manually create an “Allow” rule in your Cloudflare dashboard.
