Skip to content
AI Search

AI crawler

An AI crawler reads your site for products such as ChatGPT, Claude, or Perplexity. Blocking it in robots.txt can prevent citations.

Each assistant uses named user-agents. Many sites block them by accident, copied from a template, and quietly opt out of AI answers.

Allowing them is a real choice, not a default. Blocking them protects content but removes you from AI search.

Related terms

robots.txtllms.txtGenerative Engine Optimisation (GEO)
All terms

See what is broken on your website

Get a free technical SEO audit. Real checks on your real site, with fixes your developer can act on. No sales call required.

We explain every finding in your audit in plain language, like this.