423822 - Make services outage page send an appropriate http status code

Status: VERIFIED FIXED

Description

[Dave Townsend [:mossop] (he/him)]

Not sure this is the right place for this but the blocklist service is on AMO so...

It seems that in times of outage the blocklist service returns a Service Outages page like most of mozilla's webservers. Unfortunately it is returned as a regular page with 200 status code. This causes the blocklist service to assume it has been fed valid data and saves it to the users profile. This doesn't cause a major issue, the invalid page is just ignored, but it does mean that for the next day the user is without any blocklist.

It would seem to me that a services outage page should be sent with something other than a 200 status code, possibly a 503, but I'm not really sure. Basically we should be properly indicating that the service failed.

Comment 1

[Wil Clouser [:clouserw]]

There is a disabled page built in to AMO but I don't think it's ever been used in production.  I think IT usually just changes the apache config to point elsewhere.  CCing oremj.

Comment 2

[Michael Morgan [:morgamic]]

Dave does the client look for a particular code?

Comment 3

[Dave Townsend [:mossop] (he/him)]

The code looks for a status of 200 before assuming the data it has received is correct.

Comment 4

[Michael Morgan [:morgamic]]

Jeremy - what happens in the event of an outage?

Comment 5

[Michael Morgan [:morgamic]]

-> IT for feedback.

Comment 6

[Jeremy Orem [:oremj]]

In the event of an outage the backend servers get switched to the static cluster.  The following vhost gets put in use:

<VirtualHost *:80>
    ServerName outages.mozilla.com
    ServerAlias outage-org.nslb.sj.mozilla.com
    ServerAlias addons.mozilla.org
    DocumentRoot /data/www/outages.mozilla.com
    <Directory /data/www/outages.mozilla.com>
        SetEnv SERVER_NAME outages.mozilla.com
        RewriteEngine On
        RewriteCond %{SCRIPT_FILENAME} !^.*\.(js|css|png|jpg)$
        RewriteCond %{SCRIPT_FILENAME} !^.*prefetch.php$
        RewriteRule .* en-US/outages.html
        Action init /includes/prefetch.php
        AddHandler init .html
        ErrorDocument 403 /includes/prefetch.php
        ErrorDocument 404 /includes/prefetch.php
    </Directory>
</VirtualHost>

Comment 7

[Jeremy Orem [:oremj]]

So the end result is the content at http://www.mozilla.com/en-US/outages.html.

Comment 8

[Michael Morgan [:morgamic]]

So why not a 503 instead of a 200 OK?

Given:
        RewriteRule .* en-US/outages.html

During an outage people will think the page they got back was normal, and that's what this bug is about -- mismatched expectations.  :D

Comment 9

[Jeremy Orem [:oremj]]

We could set the status via the rewrite rule to 503, but it would probably be a better idea just to set 503 in en-US/outages.html (since it is really getting processed as php). 

Comment 10

[Michael Morgan [:morgamic]]

I think that's all we need, unless there's a reason not to have it.

Comment 11

[Jeremy Orem [:oremj]]

Should this be moved to the mozilla.com component for the fixes?

Comment 12

[Reed Loden [:reed] (use needinfo?)]

Added:
 header('HTTP/1.1 503 Service Temporarily Unavailable');
 header("Retry-After: 120");

Sending        outages.html
Transmitting file data .
Committed revision 13212.

Comment 13

[Stephen Donner [:stephend]]

http://web-sniffer.net/?url=http%3A%2F%2Fwww.mozilla.com%2Fen-US%2Foutages.html&submit=Submit&http=1.1&gzip=yes&type=GET&uak=0 yields a 503; Verified FIXED.

Comment 14

[Dave Townsend [:mossop] (he/him)]

Seems to be giving a 200 code right now using the test in comment 13

Comment 15

[Wil Clouser [:clouserw]]

Looks like reed backed this out 4 months ago, not sure why.  Reed?

Comment 16

[Reed Loden [:reed] (use needinfo?)]

(In reply to comment #15)
> Looks like reed backed this out 4 months ago, not sure why.  Reed?

It killed the NetScaler at the Firefox 3 launch. I think we can add this back now, but I'm pretty sure the outages page has moved to outages.mozilla.com.

Comment 17

[Reed Loden [:reed] (use needinfo?)]

r13212 added it. r15972 removed it. r19843 added it back.

Comment 18

[Stephen Donner [:stephend]]

Verified FIXED; thanks for pushing to prod, Wil.

http://web-sniffer.net/?url=http%3A%2F%2Fwww.mozilla.com%2Fen-US%2Foutages.html&submit=Submit&http=1.1&gzip=yes&type=GET&uak=0:

HTTP Status Code: HTTP/1.1 503 Service Temporarily Unavailable