bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

[socorro-crashstats] Stage should not be spidered by bots

RESOLVED FIXED

Status

Socorro
Webapp
P4
normal
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: peterbe, Assigned: phrawzty)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [triaged 20130430])

(Reporter)

Description

5 years ago
If you go to: https://crash-stats-django.allizom.org/robots.txt it should return a static content in "text/plain" containing exactly this::

User-agent: *
Disallow: *
(Reporter)

Updated

5 years ago
Summary: [socorro-crashstats] Stage should not be indexed by bots → [socorro-crashstats] Stage should not be spidered by bots
(Assignee)

Updated

5 years ago
Assignee: server-ops-webops → dmaher
Status: NEW → ASSIGNED
Priority: -- → P4
Whiteboard: [triaged 20130430]
(Assignee)

Comment 1

5 years ago
Hello,

As a general rule webops does not manage the content of websites.  If you wish to include a robots.txt in your webapp, you are free to do so - how you implement that functionality within the context of your application is up to you.

If you need any further information, please don't hesitate to ask.
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → WONTFIX
(Reporter)

Comment 2

5 years ago
Good point. I was just thinking about the fact that this is going to need to be different per instance and there's no other way to do that other than to set by configuration. 

I'll build a solution within socorro-crashstats and then I can bounce it back to you guys to vary the settings/local.py accordingly.
Status: RESOLVED → REOPENED
Component: Server Operations: Web Operations → Webapp
Product: mozilla.org → Socorro
QA Contact: nmaul
Resolution: WONTFIX → ---
(Reporter)

Comment 5

5 years ago
The code to switch /robots.txt "on and off" is now in place but since we still don't run Django in production, we'll leave it off on all sites for now. 

Once we switch to Django for production (for crash-stats.mozilla.com) then we can enable this.
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.