857621 - https://wiki.mozilla.org/robots.txt disallows all

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Kartikaya Gupta (email:[e-mail])]

Attachment: Google results screenshot

https://wiki.mozilla.org/robots.txt disallows everything. I was searching for something on google today and got the attached (I added the red outline for emphasis). Is this intended?

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This is making it hard to find things on wikimo - and killing the site's google-fu.  it's probably too late for the latter..

Comment 2

[Tantek Çelik]

This seems like a major regression.

Comment 3

[Andrei Hajdukewycz [:sancus]]

Seriously.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Andrei, "critical" means "all work stops".  Think of it as "I'm willing to have a meeting with Mitchell to explain this".  So, not this bug.

Callek, what do you think?  Who can make this call?  Sadly, it's probably too late for the site's pagerank..

Comment 5

[Justin Wood (:Callek)]

Personally I'm ok if google indexes/crawls wiki.m.o, nothing on there should ever be private that a crawl would expose it.

The only downside is it makes it a larger target of spammers, (since they can get their pagerank artificially increased), but I think we're in a better state (right now) re: newly incomming spam than we used to be.

Final say, lies with webops, imho.

Comment 6

[Andrei Hajdukewycz [:sancus]]

(In reply to Dustin J. Mitchell [:dustin] from comment #4)
> Andrei, "critical" means "all work stops".  Think of it as "I'm willing to
> have a meeting with Mitchell to explain this".  So, not this bug.

I thought that was blocker :p 

Anyway, I think it is pretty critical for wikimo to be google searchable, because if it's not that makes it considerably harder for anyone to find anything on it, especially people looking for things they don't *know* are even on the wiki.

Comment 7

[Chris Turra [:cturra]]

i agree this really isn't a blocker -- blockers are reserved for scenarios like "omg, the site is down!!111one"

i have applied a new robots.txt file for the wiki that should be a little less restrictive for bots. note, i have applied Disallow list of the top known "scraper" bots, plus many of our "Special:..." pages. *interesting note on that, i chose not to use wildcard matching in here since only the Google and Microsoft's Bing bots honor wildcards.

Comment 8

[Andrei Hajdukewycz [:sancus]]

Thanks! I apologize for the somewhat aggressive severity setting :p

Comment 10

[Matt Basta [:basta]]

This seems to have regressed. The current robots.txt disallows all robots;

    User-agent: *
    Disallow: /

Comment 11

[Chris Turra [:cturra]]

looks like a recent mediawiki updated the robots.txt back to the vanilla one that is shipped with the product. i have updated this again and will look into how we can make sure this gets applied with future upgrades.