Closed
Bug 539961
Opened 16 years ago
Closed 16 years ago
[dashboard][robots] create a robots app to serve robots.txt
Categories
(Mozilla Localizations :: Infrastructure, defect, P1)
Mozilla Localizations
Infrastructure
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: Pike, Unassigned)
Details
We should serve a robots.txt file to control which parts of the dashboard get hit by search engines.
Until the site is more stable, including perf fixes and a permanent URL, I'd suggest to disallow all.
|
Reporter | |
Comment 1•16 years ago
|
||
Found
(r'^robots\.txt$', lambda r: HttpResponse("User-agent: *\nDisallow: /*", mimetype="text/plain"))
on http://news.e-scribe.com/431, that should be good enough for now.
|
||
Comment 2•16 years ago
|
||
yup, that's matching my experience with robots.txt.
Additionally, we may want to block via .htaccess spiders that are ignoring robots.txt.
I'll work on a magical recepe for that.
|
|
||
Comment 3•16 years ago
|
||
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•