robots.txt
robots.txt tells obeying crawlers which paths they may access. It controls crawling, not reliable removal from search or private access protection.
It controls crawling, not indexing. A blocked page can still appear in results without a description if other sites link to it.
The most expensive mistake is a site-wide Disallow left over from staging. It is invisible in a browser, so it survives until someone reads the file.
A practical example
A staging rule such as Disallow: / is accidentally copied to the live website.
A common mistake
Blocking a page in robots.txt while expecting Google to read its noindex tag.
How to check it
Open /robots.txt, identify the applicable user-agent group, and check its rules against important paths.
Related terms
See what is broken on your website
Get a free technical SEO audit. Real checks on your real site, with fixes your developer can act on. No sales call required.
We explain every finding in your audit in plain language, like this.