robots.txt

It controls crawling, not indexing. A blocked page can still appear in results without a description if other sites link to it.

The most expensive mistake is a site-wide Disallow left over from staging. It is invisible in a browser, so it survives until someone reads the file.