Key findings
- A page can return a healthy 200 and still be blocked from search by a header.
- The X-Robots-Tag header overrides the meta tag, and most tools never show it.
- This usually hits a whole template, not one page, from a leftover staging rule.
- One curl command finds it, and one line fixes it.
Most SEO tools read the page the way a person does. They load the HTML, run the scripts, and score what appears. A crawler does something different first. It reads the HTTP response, and the response can say do not index this before a single pixel is drawn.
Where the instruction hides
There are two places a page can tell Google to drop it. One is a meta tag in the HTML. The other is an HTTP header called X-Robots-Tag, sent with the response before the body. The header wins, and most dashboards never show it.
GET /products/blue-widget HTTP/1.1
HTTP/1.1 200 OK
content-type: text/html
x-robots-tag: noindex ← the page is fine, the header is notA 200 status looks healthy. The page loads, the content is there, the screenshot in the monthly report looks correct. Google still drops the URL, because the header told it to.
How a whole template gets it
This rarely happens to one page. It happens to a template. A staging rule that adds noindex ships to production. A CDN rule matches more paths than intended. A framework default is left on. One line in the wrong place, and every product page inherits it.
The tell is scale. If organic traffic to one page type fell off a cliff on a single day, look at the response header for that template before you look at anything else.
How to check it yourself
You do not need a tool. Ask the server for the header directly and read what it returns to a crawler.
curl -sI https://yourwebsite.com/a-real-page | grep -i x-robots-tag
# nothing printed → good, the page is indexable
# x-robots-tag: noindex → the page is being droppedRun it against a page that should rank and is not. Then run it against one that ranks fine. If only the broken one returns the header, you have found it in one command.
The fix, and who owns it
- Find the rule that sets the header. It lives in the server config, the CDN, or the framework middleware, not in the page.
- Scope it correctly, or remove it. Staging rules should never match a production hostname.
- Confirm with the same
curl. The header should be gone. - Request re-indexing in Search Console for a sample of URLs, then watch coverage recover.
This is the kind of finding a written audit hands your developer: the exact URL, the exact header, and the one line to change. No dashboard, no guesswork.
See this on your own site, for free.
The audit runs this exact check against your live pages.
CraftDigitally Team
We run and grow our own B2B sites, then write up what we learn. Every claim here is one we check against a live site first.
