621754 - Block search engines from indexing /{locale}/products/download.html

Status: RESOLVED FIXED

(www.mozilla.org :: General, defect)

Description

[Alex Buchanan [:abuchanan]]

Attachment: robots.txt patch

From bug 621559, search engines are indexing this page, but it's a very poor landing page.

We should patch robots.txt to block search engines from indexing these pages.

Patch attached

Comment 1

[Alex Buchanan [:abuchanan]]

On trunk, 

Sending        robots.txt
Transmitting file data .
Committed revision 79977.

Comment 2

[Alex Buchanan [:abuchanan]]

Stephend, unless you know of a tool that checks robots.txt rules, I don't know how you can QA this. 

The real test would be to make sure it disappears from search results once this hits production and enough time has passed that search engines have re-indexed the site (1-2 days?)

Comment 3

[Alex Buchanan [:abuchanan]]

Example search query,
http://www.google.com/search?q=download+firefox&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:unofficial&client=firefox-a

/products/download.html is the second result for me

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #2)
> Stephend, unless you know of a tool that checks robots.txt rules, I don't know
> how you can QA this. 
> 
> The real test would be to make sure it disappears from search results once this
> hits production and enough time has passed that search engines have re-indexed
> the site (1-2 days?)

Well, I can at least verify the changes hit staging, first:

http://www-trunk.stage.mozilla.com/robots.txt

User-agent: *
Disallow: /*/products/download.html

qa-verified-trunk

Will verify post-push in a few days, as well.

Comment 5

[Alex Buchanan [:abuchanan]]

r79989 on production

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

robots.txt change is in, on prod:

http://www.mozilla.com/robots.txt; will wait a few days (comment 2, comment 4), until I verify for realz.

Comment 7

[Pascal Chevrel (relman team) -> :pascalc]

fyi, r80114 has the l10n part of the patch, we are not using that page yet (except for betas), therefore those pages are not idexed yet Google, so that's a preventive change.