Closed Bug 737099 Opened 12 years ago Closed 11 years ago

Adjust robots.txt file on Mozillians.org

Categories

(Participation Infrastructure :: Phonebook, defect)

defect
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: cmore, Assigned: giorgos)

References

Details

The robots.txt file on the production Mozillians.org website is blocking all spiders from scanning content.

https://mozillians.org/robots.txt

We normally block all spiders and search engines from -dev -stage sites and then just blank out the robots.txt file on production sites. I don't know if the robots.txt file to disallow all to the entire website was done on purpose or not. Since most of the site is behind authentication, I can see the rationale, but there is also no harm in allowing search engines to find that out naturally.

Thanks!
Assignee: nobody → timw
OS: Mac OS X → All
Hardware: x86 → All
Whiteboard: [rel:04/04/2012]
Whiteboard: [rel:04/04/2012]
Component: mozillians.org → Phonebook
Product: Websites → Community Tools
QA Contact: mozillians-org → phonebook
Version: unspecified → other
William, thoughts on this one? I tend to agree with Cmore, the about pages etc should be spidered. Users profiles will not be spidered as they require auth.
Assignee: timw → williamr
Flags: needinfo?(williamr)
Yeah, let the search engines go wild on spidering. They can't get to anything that shouldn't be able to and when we have more public views, they will find them. Google search for "Mozillians" and notice that the description can't even be pulled because we are blocking all search engines. This is silly and since SEO is so important, we shouldn't be stopping search engines.
Depends on: 820850
This is now fixed!

Prod allows search engine to crawl everything, while Stage and Dev blocks everything.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Thanks Giorgos - this is great! I'm giving you proper credit as the assignee of the bug since you resolved it :)
Assignee: williamr → giorgos
Flags: needinfo?(williamr)
QA verified on prod - https://mozillians.org/robots.txt
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.