Closed Bug 539961 Opened 15 years ago Closed 15 years ago

[dashboard][robots] create a robots app to serve robots.txt

Categories

(Mozilla Localizations :: Infrastructure, defect, P1)

Product:
Mozilla Localizations
Component:
Infrastructure
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Pike, Unassigned)

Details

We should serve a robots.txt file to control which parts of the dashboard get hit by search engines.

Until the site is more stable, including perf fixes and a permanent URL, I'd suggest to disallow all.
Found

(r'^robots\.txt$', lambda r: HttpResponse("User-agent: *\nDisallow: /*", mimetype="text/plain"))

on http://news.e-scribe.com/431, that should be good enough for now.
yup, that's matching my experience with robots.txt.

Additionally, we may want to block via .htaccess spiders that are ignoring robots.txt.

I'll work on a magical recepe for that.
http://hg.mozilla.org/l10n/django-site/rev/c7997ff49b35
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.