Technical SEO
robots.txt
robots.txt is a file at the root of your site that tells crawlers where they may and may not go. One wrong line can hide the whole site from search.
It controls crawling, not indexing. A blocked page can still appear in results without a description if other sites link to it.
The most expensive mistake is a site-wide Disallow left over from staging. It is invisible in a browser, so it survives until someone reads the file.
Related terms
See what is broken on your website
Get a free technical SEO audit. Real checks on your real site, with fixes your developer can act on. No sales call required.
We explain every finding in your audit in plain language, like this.