The robots.txt file on the production Mozillians.org website is blocking all spiders from scanning content. https://mozillians.org/robots.txt We normally block all spiders and search engines from -dev -stage sites and then just blank out the robots.txt file on production sites. I don't know if the robots.txt file to disallow all to the entire website was done on purpose or not. Since most of the site is behind authentication, I can see the rationale, but there is also no harm in allowing search engines to find that out naturally. Thanks!
Assignee: nobody → timw
OS: Mac OS X → All
Hardware: x86 → All
Component: mozillians.org → Phonebook
Product: Websites → Community Tools
QA Contact: mozillians-org → phonebook
Version: unspecified → other
William, thoughts on this one? I tend to agree with Cmore, the about pages etc should be spidered. Users profiles will not be spidered as they require auth.
Assignee: timw → williamr
Yeah, let the search engines go wild on spidering. They can't get to anything that shouldn't be able to and when we have more public views, they will find them. Google search for "Mozillians" and notice that the description can't even be pulled because we are blocking all search engines. This is silly and since SEO is so important, we shouldn't be stopping search engines.
Commit pushed to master at https://github.com/mozilla/mozillians https://github.com/mozilla/mozillians/commit/d846ec18e598221c233889df81e0cae7301ec38d [bug 737099] Add robots.txt
Commit pushed to master at https://github.com/mozilla/mozillians https://github.com/mozilla/mozillians/commit/10f1939bc036ae3b7473649615e4d69bcd35fc57 [bug 737099] Move /robots.txt to /media/robots-prod.txt
This is now fixed! Prod allows search engine to crawl everything, while Stage and Dev blocks everything.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Thanks Giorgos - this is great! I'm giving you proper credit as the assignee of the bug since you resolved it :)
Assignee: williamr → giorgos
QA verified on prod - https://mozillians.org/robots.txt
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.