It was determined last month that the misplaced traffic in the Contributors forum was caused by visitors arriving via Google search results. Now that we have pretty forum URLs, we should prevent Google from indexing the contributors forum.
James says this is super simple and almost zero risk so let's just do it in 1.4.1
Assignee: nobody → james
Target Milestone: 1.4.2 → 1.4.1
Created attachment 405571 [details] [diff] [review] robots.txt patch Based on my research, Google, Yahoo and MSN/Bing (at least) support the wildcard matching. Bing's "Robots.txt validator" didn't complain about "Allow: " directives but it was not clear from other sources that Bing supports it. These lines should block the Contributors' and Off Topic forums in all locales in crawlers that support wildcards, and will hopefully be ignored by anything else.
Attachment #405571 - Flags: review?(morgamic)
Attachment #405571 - Flags: review?(morgamic) → review+
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Summary: Stop Google from indexing Contributors forum → Stop search engines from indexing Contributors and Off Topic fora
https://support-stage.mozilla.org/robots.txt matches the patch; verified FIXED based on code inspection (I don't think there's a way to tell until Google's index updates itself based on these rules in prod).
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.