Closed Bug 665231 Opened 13 years ago Closed 13 years ago

Add robots.txt

Categories

(Websites Graveyard :: markup.mozilla.org, defect)

Product:
Websites Graveyard
Component:
markup.mozilla.org
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: wenzel, Assigned: wenzel)

Details

(Whiteboard: [qa-is-happy]) 

Googlebot is sucking in most of Markup, and it shouldn't.

robots.txt should restrict Google's access to the parts that are actual content, like the about page.
https://github.com/mozilla/markup/commit/6d10746

Added! Matt, to test this, check out if robots.txt is accessible on dev, and if it looks reasonable :)
Assignee: nobody → fwenzel
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Target Milestone: --- → 1.1
QA verified ... looks reasonable to me
Status: RESOLVED → VERIFIED
Whiteboard: [qa-]
How is it qa- if you verified it ;)
haha ... true dat :)
Whiteboard: [qa-] → [qa-is-happy]
Product: Websites → Websites Graveyard
You need to log in before you can comment on or make changes to this bug.