AI crawler
An AI crawler fetches content for an AI provider. Agents can have different purposes, including search discovery, retrieval or model training.
Review the provider's current documentation for each user-agent. Training controls and search-discovery controls should not be treated as interchangeable.
Choose permissions according to your content policy and the agent's role. Allowing access does not guarantee a citation; robots.txt is not private access protection.
A practical example
A business permits a search crawler while separately restricting an agent used for model training.
A common mistake
Assuming every agent from one provider has the same purpose or control rules.
How to check it
Read the provider's current agent documentation. Review robots rules against your chosen search, retrieval and training policy.
See what is broken on your website
Get a free technical SEO audit. Real checks on your real site, with fixes your developer can act on. No sales call required.
We explain every finding in your audit in plain language, like this.