848454 - [elmo] ALLOWED_HOSTS set up for l10n.mozilla.org and l10n-dev-sj.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P3)

Description

[Peter Bengtsson [:peterbe]]

We're now using Django 1.4.4 on prod and dev. 

Setting ALLOWED_HOSTS = ['l10n.mozilla.org'] and ALLOWED_HOSTS = ['l10n-dev-sj.mozilla.org'] on /home/dashboard/site/elmo/settings/local.py /home/dashboard/test/elmo/settings/local.py does not work. 

At the time of writing ALLOWED_HOSTS is set on dev and NOT set on prod and that's the only way we can make it work.

Comment 1

[Brandon Burton [:solarce]]

Background:

l10n-dashboard1.webapp.scl3.mozilla.com is an unmanaged (by IT) Ubuntu 10.04 VM, so it's configs do not match our typical RHEL6/Puppetized configs.

When ALLOWED_HOSTS was added to l10n-dev-sj.mozilla.org this caused Zeus to fail the server, which is the only backend pool, because the health check at the time (simple HTTP) expected a 2xx, 3xx, or 4xx response.

What happened is that with ALLOWED_HOSTS enabled, requests to IP:81/ were returning a 500 error and the health check was failing. Additionally, because the l10n-dev Virtual Host configuration file is the first that Apache reads (Apache reads these in alphanumerically) and has a valid :81 Virtual Host block, it handles requests for just the server's IP

In order to restore service I updated the backend pool health check to use a PING check.

Current Status:

1. Apache is functional, with dev having ALLOWED_HOSTS and prod not currently, as :peterbe outlined above
2. Zeus is using a PING check for the backend health check, and this is working so that Zeus serves traffic
3. the l10n-dev vhost is what responds to requests for just IP, which is what Zeus would do for an HTTP health check

Next Steps:

1. there is no reason not to enabled ALLOWED_HOSTS on prod now
2. Do we want to leave the backend health check set to PING? Do we want to make it do a TCP port check? 3. Do we want to add another virtual host that responds for requests to IP that would keep Zeus sending traffic even if any application vhosts goes into a 500.
 * I think one later of the two makes sense so that application errors get sent through Zeus to the browser.

Comment 2

[Axel Hecht [:Pike]]

FTR, we shouldn't attempt to do further changes until pwn2own is over, a downtime here shouldn't concern us when trying to do a chemspill for Firefox.

Comment 3

[Peter Bengtsson [:peterbe]]

:solarce,

We've enabled `ALLOWED_HOSTS` on prod **and** dev now. Yippi!

However, we struggle to understand what the next action is regarding the Zeus health check. It think partly we don't understand the tech as you do and partly we kinda wish you would just suggest the best course of action. So, what do you think it should be? And can you proceed to set that up please?

Comment 4

[Brandon Burton [:solarce]]

(In reply to Peter Bengtsson [:peterbe] from comment #3)
> :solarce,
> 
> We've enabled `ALLOWED_HOSTS` on prod **and** dev now. Yippi!
> 
> However, we struggle to understand what the next action is regarding the
> Zeus health check. It think partly we don't understand the tech as you do
> and partly we kinda wish you would just suggest the best course of action.
> So, what do you think it should be? And can you proceed to set that up
> please?

I think using a TCP port healthcheck on port 81 is simplest and best, let me know a good time (where the site might go down for a minute or two) and I'll flip the Zeus config over to use it

Comment 5

[Brandon Burton [:solarce]]

This has been solid with the ping check so I am going to RF and we can tweak the healthcheck if we find it causes an issue in the future.