Adjust robots.txt file on Mozillians.org

VERIFIED FIXED

Status

VERIFIED FIXED
7 years ago
5 years ago

People

(Reporter: cmore, Assigned: giorgos)

Tracking

Details

(Reporter)

Description

7 years ago
The robots.txt file on the production Mozillians.org website is blocking all spiders from scanning content.

https://mozillians.org/robots.txt

We normally block all spiders and search engines from -dev -stage sites and then just blank out the robots.txt file on production sites. I don't know if the robots.txt file to disallow all to the entire website was done on purpose or not. Since most of the site is behind authentication, I can see the rationale, but there is also no harm in allowing search engines to find that out naturally.

Thanks!

Updated

7 years ago
Assignee: nobody → timw
OS: Mac OS X → All
Hardware: x86 → All
Whiteboard: [rel:04/04/2012]
Whiteboard: [rel:04/04/2012]

Updated

7 years ago
Component: mozillians.org → Phonebook
Product: Websites → Community Tools
QA Contact: mozillians-org → phonebook
Version: unspecified → other

Comment 1

6 years ago
William, thoughts on this one? I tend to agree with Cmore, the about pages etc should be spidered. Users profiles will not be spidered as they require auth.
Assignee: timw → williamr
Flags: needinfo?(williamr)
(Reporter)

Comment 2

6 years ago
Yeah, let the search engines go wild on spidering. They can't get to anything that shouldn't be able to and when we have more public views, they will find them. Google search for "Mozillians" and notice that the description can't even be pulled because we are blocking all search engines. This is silly and since SEO is so important, we shouldn't be stopping search engines.
(Assignee)

Updated

6 years ago
Depends on: 820850
(Assignee)

Comment 5

6 years ago
This is now fixed!

Prod allows search engine to crawl everything, while Stage and Dev blocks everything.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Thanks Giorgos - this is great! I'm giving you proper credit as the assignee of the bug since you resolved it :)
Assignee: williamr → giorgos
Flags: needinfo?(williamr)
QA verified on prod - https://mozillians.org/robots.txt
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.