Stop search engines from indexing Contributors and Off Topic fora

VERIFIED FIXED in 1.4.1

Status

support.mozilla.org
General
VERIFIED FIXED
9 years ago
8 years ago

People

(Reporter: cilias, Assigned: jsocol)

Tracking

unspecified
1.4.1

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: sumo_only, URL)

Attachments

(1 attachment)

(Reporter)

Description

9 years ago
It was determined last month that the misplaced traffic in the Contributors forum was caused by visitors arriving via Google search results. Now that we have pretty forum URLs, we should prevent Google from indexing the contributors forum.

Comment 1

9 years ago
James says this is super simple and almost zero risk so let's just do it in 1.4.1
Assignee: nobody → james
Target Milestone: 1.4.2 → 1.4.1
(Assignee)

Comment 2

9 years ago
Created attachment 405571 [details] [diff] [review]
robots.txt patch

Based on my research, Google, Yahoo and MSN/Bing (at least) support the wildcard matching. Bing's "Robots.txt validator" didn't complain about "Allow: " directives but it was not clear from other sources that Bing supports it.

These lines should block the Contributors' and Off Topic forums in all locales in crawlers that support wildcards, and will hopefully be ignored by anything else.
Attachment #405571 - Flags: review?(morgamic)
Attachment #405571 - Flags: review?(morgamic) → review+
(Assignee)

Comment 3

9 years ago
r53170
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Summary: Stop Google from indexing Contributors forum → Stop search engines from indexing Contributors and Off Topic fora
https://support-stage.mozilla.org/robots.txt matches the patch; verified FIXED based on code inspection (I don't think there's a way to tell until Google's index updates itself based on these rules in prod).
Status: RESOLVED → VERIFIED
(Assignee)

Comment 5

9 years ago
r53245 (prod)
(Assignee)

Updated

8 years ago
Whiteboard: sumo_only
You need to log in before you can comment on or make changes to this bug.