892189 - Add robots.txt to block spiders to download.mozilla.org

Status: VERIFIED FIXED

(Webtools :: Bouncer, defect)

Description

[Chris More [:cmore]]

Can we add a robots.txt file to http://download.mozilla.org to block search engines from spidering that URL. That URL should not be showing up in search as the query paramaters passed to it may not be specific to the search query. It is showing up on bing, I've blocked it on bing, but that is just temporary.

For example, if you search bing for firefox there is a "deep link" called "Download" that points to: https://download.mozilla.org/?product=firefox-stub&os=win&lang=en-US. That is the windows build and my search came from Mac.

To block all spiders to all URLs on d.mo., the robotx.txt file should be:

User-agent: *
Disallow: /

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Perhaps this should be handled by WebOps as an server setup issue ?

Comment 2

[Chris More [:cmore]]

jakem: should we switch components for this? I didn't know a better place to start witht his one.

Comment 3

[Chris More [:cmore]]

cturra: can you help here? We are getting a lot of invalid downloads from bing.com and robots.txt should be used to prevent this for all search engines.

Comment 4

[Daniel Maher [:phrawzty]]

Webops does not manage content. Moving to the webtools component.

Comment 5

[Chris More [:cmore]]

(In reply to Daniel Maher [:phrawzty] (afk 24-07-2013 through 04-08-2013) from comment #4)
> Webops does not manage content. Moving to the webtools component.

That's where it was initially and :nthomas said to pass it over to WebOps and now WebOps is saying pass it over to the Webtools team. 

Laura/Jakem/nthomas: Can we just decide who is going to do this change? Bing is serving up links to en-US windows builds to all users regardless of their locale or operating system. A robot.txt is the way to prevent this from happening. d.m.o should not be indexed by robots.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

/CC bsavage as someone who touched bouncer this year

Apologies if that was wrong, it was a guess based on download.m.o being a php app. Perhaps this is a simple as dumping the robots.txt at https://github.com/mozilla/tuxedo/tree/master/bouncer/php/, maybe it needs some Apache work too.

Comment 7

[Brandon Savage [:brandon]]

Fixed in https://github.com/mozilla/tuxedo/commit/226d71cfa6b85103aaf66631b6288e2d2afb92fc

All this needs now is to be deployed.

Comment 8

[Chris More [:cmore]]

(In reply to Brandon Savage [:brandon] from comment #7)
> Fixed in
> https://github.com/mozilla/tuxedo/commit/
> 226d71cfa6b85103aaf66631b6288e2d2afb92fc
> 
> All this needs now is to be deployed.

Thanks, Brandon! What is the normal deployment for this?

Try searching for site:download.mozilla.org on both Google and Bing. Google totally ignores the domain as it should and bing has been indexing it all along.

Comment 9

[Brandon Savage [:brandon]]

Let me talk with :laura and :jakem this morning and see about getting this shipped. Assuming that we are already serving the latest master (before this change) it should quite literally be trivial.

Comment 10

[Chris More [:cmore]]

(In reply to Brandon Savage [:brandon] from comment #9)
> Let me talk with :laura and :jakem this morning and see about getting this
> shipped. Assuming that we are already serving the latest master (before this
> change) it should quite literally be trivial.

Can we get an update here?

Comment 11

[Brandon Savage [:brandon]]

This has been pushed to production.

Comment 12

[Matt Brandt [:mbrandt]]

QA verified on stage and prod - automation also passes. Thanks to Brandon for adding a test for robots.txt

Comment 13

[Chris More [:cmore]]

(In reply to Brandon Savage [:brandon] from comment #11)
> This has been pushed to production.

Thanks!