AI Search
AI crawler
An AI crawler is a bot that reads your site for an AI product, such as GPTBot, ClaudeBot, or PerplexityBot. If you block it in robots.txt, that assistant cannot cite you.
Each assistant uses named user-agents. Many sites block them by accident, copied from a template, and quietly opt out of AI answers.
Allowing them is a real choice, not a default. Blocking them protects content but removes you from AI search.
See what is broken on your website
Get a free technical SEO audit. Real checks on your real site, with fixes your developer can act on. No sales call required.
We explain every finding in your audit in plain language, like this.