866873 - [socorro-crashstats] Stage should not be spidered by bots

Status: RESOLVED FIXED

(Socorro :: Webapp, task, P4)

Description

[Peter Bengtsson [:peterbe]]

If you go to: https://crash-stats-django.allizom.org/robots.txt it should return a static content in "text/plain" containing exactly this::

User-agent: *
Disallow: *

Comment 1

[Daniel Maher [:phrawzty]]

Hello,

As a general rule webops does not manage the content of websites.  If you wish to include a robots.txt in your webapp, you are free to do so - how you implement that functionality within the context of your application is up to you.

If you need any further information, please don't hesitate to ask.

Comment 2

[Peter Bengtsson [:peterbe]]

Good point. I was just thinking about the fact that this is going to need to be different per instance and there's no other way to do that other than to set by configuration. 

I'll build a solution within socorro-crashstats and then I can bounce it back to you guys to vary the settings/local.py accordingly.

Comment 3

[Peter Bengtsson [:peterbe]]

Pull request: https://github.com/mozilla/socorro-crashstats/pull/333

Comment 4

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro-crashstats

https://github.com/mozilla/socorro-crashstats/commit/1136c7dbbdf01c0e068d045f8e85492a895826f6
bug 866873 - robots.txt accordingly

https://github.com/mozilla/socorro-crashstats/commit/d70e4e19e8c8070f5416fbae305739127497e390
Merge pull request #333 from peterbe/bug866873-robotstxt-accordingly

bug 866873 - robots.txt accordingly

Comment 5

[Peter Bengtsson [:peterbe]]

The code to switch /robots.txt "on and off" is now in place but since we still don't run Django in production, we'll leave it off on all sites for now. 

Once we switch to Django for production (for crash-stats.mozilla.com) then we can enable this.