There's no robots.txt for Mozilla Reps: http://reps-dev.allizom.org/robots.txt Not sure what we'd put in it (Chris, suggestions/recommendations?), but here are examples from other projects: http://support.mozilla.org/robots.txt https://addons.mozilla.org/robots.txt http://www.mozilla.org/robots.txt
This is a list of sites that have a robots.txt where search engines are blocked: https://wiki.mozilla.org/Websites/Domain_List/robots-blocked Most comment robots.txt for non-prod sites is: User-agent: * Disallow: / Though, some sites do specific disallows on certain directories, but that doesn't quite make sense for a dev/stage site.
I don't think that there is a directory that we want to disallow on 0.1. We want all profiles to be searchable. Moving this to 0.2.
Version: 0.1 - Solkar → 0.2 - Skon
Checked with cmore: For dev and stage you want to have: User-agent: * Disallow: / For production if you do not have anything you want to disallow just have an empty robots.txt file.
Commit pushed to master at https://github.com/mozilla/remo https://github.com/mozilla/remo/commit/a8af13e19ee279261c0d4ca1ba70d36cff1b0d89 Merge pull request #111 from glogiotatidis/robots Add robots.txt. Fix bug 727149.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Verified FIXED; thanks.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.