Closed Bug 607996 Opened 14 years ago Closed 14 years ago

check all staging sites for proper robots.txt

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jslater, Unassigned)

Details

Per http://twitter.com/cramforce/statuses/28971217655, it looks like we don't have the proper robots.txt file enabled, so search engines are indexing our staging sites. I'm not sure if this is happening only on MDN or on all our sites, but it should be fixed. Can you check the staging sites to see what we do and don't have, and then implement robots.txt where needed? Thanks-
Committed a robots.txt for Django app. Filed It Bug#608003 Neither of our staging servers are configured to disallow indexing.
http://mdn.staging.mozilla.com/robots.txt is fixed... Working on deki...
Please reopen for other staging servers, but deki and django are all set now. curl http://mdn.staging.mozilla.com/robots.txt User-agent: * Disallow: / curl http://developer-stage.mozilla.org/robots.txt User-agent: * Disallow: /
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.