Free tool
Robots.txt Generator & Checker
Build a robots.txt from a safe preset, or paste your site and read the one it already serves. Small file, big blast radius: one wrong line can hide your whole site.
Allow rules are listed first, so a specific Allow can carve an exception out of a broader Disallow (one path per line).
Save as /robots.txt
User-agent: *
Allow: /
Sitemap: https://yourwebsite.com/sitemap.xmlGet this written up for your developer
The result is on the page already. Add your email and we will send the fix in plain language, plus the next checks worth running.
What this means
robots.txt tells crawlers where they may and may not go. The most expensive mistake in it is a single line, Disallow: / under User-agent: *, usually left over from staging. It blocks every crawler from every page, and it is invisible unless you look.
Two things belong in almost every robots.txt: allow the pages you want indexed, and declare your Sitemap: so crawlers find your URLs quickly. Everything else is fine-tuning.
Deciding whether to allow AI crawlers is a real choice, not a default. Blocking them keeps you out of AI answers; allowing them is how you get cited. Our AI Crawler Checker shows where your site stands today.
See what is broken on your website
Get a free technical SEO audit. Real checks on your real site, with fixes your developer can act on. No sales call required.
This checked one thing. The full audit checks forty.