Status

Mozilla Reps
reps.mozilla.org
VERIFIED FIXED
6 years ago
6 years ago

People

(Reporter: stephend, Assigned: giorgos)

Tracking

0.2 - Skon
Dependency tree / graph

Details

(URL)

(Reporter)

Description

6 years ago
There's no robots.txt for Mozilla Reps:

http://reps-dev.allizom.org/robots.txt

Not sure what we'd put in it (Chris, suggestions/recommendations?), but here are examples from other projects:

http://support.mozilla.org/robots.txt
https://addons.mozilla.org/robots.txt
http://www.mozilla.org/robots.txt

Comment 1

6 years ago
This is a list of sites that have a robots.txt where search engines are blocked:

https://wiki.mozilla.org/Websites/Domain_List/robots-blocked

Most comment robots.txt for non-prod sites is:

User-agent: *
Disallow: /

Though, some sites do specific disallows on certain directories, but that doesn't quite make sense for a dev/stage site.
Group: mozilla-reps
(Assignee)

Comment 2

6 years ago
I don't think that there is a directory that we want to disallow on 0.1. We want all profiles to be searchable. Moving this to 0.2.
Version: 0.1 - Solkar → 0.2 - Skon

Comment 3

6 years ago
Checked with cmore:

For dev and stage you want to have:

User-agent: *
Disallow: /

For production if you do not have anything you want to disallow just have an empty robots.txt file.
(Assignee)

Updated

6 years ago
Depends on: 741323
(Assignee)

Updated

6 years ago
Depends on: 741326

Comment 4

6 years ago
Commit pushed to master at https://github.com/mozilla/remo

https://github.com/mozilla/remo/commit/a8af13e19ee279261c0d4ca1ba70d36cff1b0d89
Merge pull request #111 from glogiotatidis/robots

Add robots.txt. Fix bug 727149.

Updated

6 years ago
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Assignee)

Updated

6 years ago
Assignee: nobody → giorgos
(Reporter)

Comment 5

6 years ago
Verified FIXED; thanks.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.