Closed
Bug 737099
Opened 12 years ago
Closed 11 years ago
Adjust robots.txt file on Mozillians.org
Categories
(Participation Infrastructure :: Phonebook, defect)
Participation Infrastructure
Phonebook
Tracking
(Not tracked)
VERIFIED
FIXED
People
(Reporter: cmore, Assigned: giorgos)
References
Details
The robots.txt file on the production Mozillians.org website is blocking all spiders from scanning content. https://mozillians.org/robots.txt We normally block all spiders and search engines from -dev -stage sites and then just blank out the robots.txt file on production sites. I don't know if the robots.txt file to disallow all to the entire website was done on purpose or not. Since most of the site is behind authentication, I can see the rationale, but there is also no harm in allowing search engines to find that out naturally. Thanks!
Updated•12 years ago
|
Assignee: nobody → timw
OS: Mac OS X → All
Hardware: x86 → All
Whiteboard: [rel:04/04/2012]
Updated•12 years ago
|
Whiteboard: [rel:04/04/2012]
Updated•12 years ago
|
Component: mozillians.org → Phonebook
Product: Websites → Community Tools
QA Contact: mozillians-org → phonebook
Version: unspecified → other
Comment 1•12 years ago
|
||
William, thoughts on this one? I tend to agree with Cmore, the about pages etc should be spidered. Users profiles will not be spidered as they require auth.
Assignee: timw → williamr
Flags: needinfo?(williamr)
Reporter | ||
Comment 2•12 years ago
|
||
Yeah, let the search engines go wild on spidering. They can't get to anything that shouldn't be able to and when we have more public views, they will find them. Google search for "Mozillians" and notice that the description can't even be pulled because we are blocking all search engines. This is silly and since SEO is so important, we shouldn't be stopping search engines.
Comment 3•11 years ago
|
||
Commit pushed to master at https://github.com/mozilla/mozillians https://github.com/mozilla/mozillians/commit/d846ec18e598221c233889df81e0cae7301ec38d [bug 737099] Add robots.txt
Comment 4•11 years ago
|
||
Commit pushed to master at https://github.com/mozilla/mozillians https://github.com/mozilla/mozillians/commit/10f1939bc036ae3b7473649615e4d69bcd35fc57 [bug 737099] Move /robots.txt to /media/robots-prod.txt
Assignee | ||
Comment 5•11 years ago
|
||
This is now fixed! Prod allows search engine to crawl everything, while Stage and Dev blocks everything.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Comment 6•11 years ago
|
||
Thanks Giorgos - this is great! I'm giving you proper credit as the assignee of the bug since you resolved it :)
Assignee: williamr → giorgos
Flags: needinfo?(williamr)
Comment 7•11 years ago
|
||
QA verified on prod - https://mozillians.org/robots.txt
Status: RESOLVED → VERIFIED
You need to log in
before you can comment on or make changes to this bug.
Description
•