Google has started emailing us (admin contact for Google Analytics account) about URLs like http://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Live_(Online)_Teen_Wolf_Season_5_Episode_13_:_Codo Which is was spam and was deleted by the content team but the URL remains. These URLs need to be removed from the site or Google needs to be prevented from accessing them. This is affecting our standing in the search results.
This seems legit, the expected behavior for when a document is logically deleted should be: 1. if the user is a super user, show the restore page with a HTTP status code of 200 2. if the user is not a super user, show the 404 page
Grabbing this to look into now.
Looks like that url is already returning an HTTP 404 and the page is tagged with x-robots-tag: noindex. I'll fix it so that the http response content is the regular 404 page if the user is not a super user.
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/74152c73ac9c6aaa54be462ad05aa2f5bff95e3f Fix bug 1244308: Return 404 content if user can't restore the document Previously, MDN would return an HTTP 404 with an HTML document that has some stuff on it and lets people restore the document. That's not great because somehow those pages are getting indexed or otherwise making search bots grumpy. Instead of doing that, this changes MDN to return a straight-up regular 404 page unless the user can restore documents in which case they get the HTML page that lets them restore documents. https://github.com/mozilla/kuma/commit/b4b8e479e8e8400ab9bbbb96042fb66381b63d77 Merge pull request #3777 from mozilla/1244308-404 Fix bug 1244308: Return 404 content if user can't restore the document