730136 - robots.txt is missing

Status: VERIFIED FIXED

(Websites :: Firefox Flicks, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

There's no robots.txt file: https://firefoxflicks-dev.allizom.org/robots.txt

Comment 1

[Ben (:bensternthal)]

will do for 2013 site.. actually this change might be a good test for our chief setup.

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

BTW, https://github.com/mozilla/marketing-project-template found this, I believe.

Comment 3

[Ben (:bensternthal)]

This looks to be added but if i read the file correctly it is preventing all spiders which is not what we want.

AFAIK we should not restrict the flicks site from being spidered.

Comment 4

[Osmose [:osmose, :mkelly]]

It was added correctly: There's a setting "ENGAGE_ROBOTS" that determines if robots are allowed or denied. We set it to False on dev and stage to keep robots from spidering the dev and stage sites as they were causing tracebacks (a seperate bug that should be fixed).

If you're concerned I can have IT reenable it for testing, but I'm not too worried about it myself, the feature is pretty much copied verbatim from playdoh and has unit tests.

Comment 5

[Ben (:bensternthal)]

No need this is fine, and thanks for explaining.

Comment 6

[Rebecca Billings [:rbillings]]

Verified