Free tool
Robots.txt Generator and Robots.txt Checker
Build a robots.txt from a safe preset, or paste your site and read the one it already serves. Small file, big blast radius: one wrong line can hide your whole site.
Allow rules appear first. A specific Allow can create an exception inside a broader Disallow. Enter one path per line.
Save as /robots.txt
User-agent: *
Allow: /
Sitemap: https://yourwebsite.com/sitemap.xmlGet this written up for your developer
The result is on the page already. Add your email and we will send the fix in plain language, plus the next checks worth running.
What this means
robots.txt tells crawlers where they may go. A dangerous mistake is Disallow: / under User-agent: *. Staging configurations often leave that rule behind. It blocks every crawler from every page.
Allow the pages you want indexed. Declare your Sitemap:so crawlers find those URLs quickly. Everything else is fine-tuning.
Deciding whether to allow AI crawlers is a real choice, not a default. Blocking them keeps you out of AI answers; allowing them is how you get cited. Our AI Crawler Checker shows where your site stands today.
See what is broken on your website
Get a free technical SEO audit. Real checks on your real site, with fixes your developer can act on. No sales call required.
This checked one thing. The full audit checks forty.