1271346 - The Zeus HTTP-Monitor status check always 301 redirects

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Ed Morley [:emorley]]

There are thousands of entries in the prod gunicorn logs like:

127.0.0.1 - - [03/May/2016:06:36:51 -0700] "GET / HTTP/1.1" 301 - "-" "HTTP-Monitor/1.1"
127.0.0.1 - - [03/May/2016:06:36:51 -0700] "GET / HTTP/1.1" 301 - "-" "HTTP-Monitor/1.1"
127.0.0.1 - - [03/May/2016:06:36:51 -0700] "GET / HTTP/1.1" 301 - "-" "HTTP-Monitor/1.1"
127.0.0.1 - - [03/May/2016:06:36:54 -0700] "GET / HTTP/1.1" 301 - "-" "HTTP-Monitor/1.1"
127.0.0.1 - - [03/May/2016:06:36:55 -0700] "GET / HTTP/1.1" 301 - "-" "HTTP-Monitor/1.1"
127.0.0.1 - - [03/May/2016:06:36:55 -0700] "GET / HTTP/1.1" 301 - "-" "HTTP-Monitor/1.1"
127.0.0.1 - - [03/May/2016:06:36:55 -0700] "GET / HTTP/1.1" 301 - "-" "HTTP-Monitor/1.1"
127.0.0.1 - - [03/May/2016:06:36:58 -0700] "GET / HTTP/1.1" 301 - "-" "HTTP-Monitor/1.1"

These are from Zeus' health checks, which is fine, however they ideally shouldn't be redirecting (otherwise we're not actually checking the real site index). Compare to other requests for the site root:

127.0.0.1 - - [03/May/2016:06:36:54 -0700] "GET / HTTP/1.1" 200 1964 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"

I imagine the reason is that the requests are not being seen as secure (eg the X-FORWARDED-PROTO header isn't being set to 'https') which is causing the Django security middleware to redirect them to the https URL.

I believe this is also the cause of the strangely high transaction times for the security middleware on:
https://rpm.newrelic.com/accounts/677903/applications/5585473/transactions?type=app#id=5b225765625472616e73616374696f6e2f46756e6374696f6e2f646a616e676f2e6d6964646c65776172652e73656375726974793a53656375726974794d6964646c65776172652e70726f636573735f72657175657374222c22225d

Kendall, I don't suppose you could check the setup for the health check for Zeus? I'm guessing since it has to query each node individually it doesn't use the normal HTTPS URL and instead makes requests to specific nodes with the 'Host' header set. If so, I guess we just need to also set 'X-FORWARDED-PROTO' and it should all work?

Also, once these are not 301-ing all the time, is there an option we can enable to make sure the checks fail if the HTTP response != 200 ? :-)

Comment 1

[Kendall Libby [:fubar] (he/him)]

(In reply to Ed Morley [:emorley] from comment #0)
>
> Kendall, I don't suppose you could check the setup for the health check for
> Zeus? I'm guessing since it has to query each node individually it doesn't
> use the normal HTTPS URL and instead makes requests to specific nodes with
> the 'Host' header set. If so, I guess we just need to also set
> 'X-FORWARDED-PROTO' and it should all work?

It's not currently using the Host header, it's just looking to see if apache answers on port 80 on each node. I *can* set a host header, but there isn't any way to set an env variable. I can make apache not log the requests, too, if that's useful.

I suspect the high transaction times are actually due to zeus using a simple http check, which just ensures something is listening on port 80. I've changed it over to using a full http check, which actually looks for a response.

> Also, once these are not 301-ing all the time, is there an option we can
> enable to make sure the checks fail if the HTTP response != 200 ? :-)

Yes; alternatively, one can look for a regexp/string in the body output.

Comment 2

[Ed Morley [:emorley]]

Thank you for the explanation. Yeah could you set a Host header please :-)

(In reply to Kendall Libby [:fubar] from comment #1)
> but there isn't any way to set an env variable.

Just to clarify - by 'X-FORWARDED-PROTO' I was meaning in the HTTP header, not an environment variable? (We may both be meaning the same thing, it just seemed strange that Zeus would let you set the Host but not other headers)

Comment 3

[Kendall Libby [:fubar] (he/him)]

(In reply to Ed Morley [:emorley] from comment #2)
> Thank you for the explanation. Yeah could you set a Host header please :-)

done.

> (In reply to Kendall Libby [:fubar] from comment #1)
> > but there isn't any way to set an env variable.
> 
> Just to clarify - by 'X-FORWARDED-PROTO' I was meaning in the HTTP header,
> not an environment variable? (We may both be meaning the same thing, it just
> seemed strange that Zeus would let you set the Host but not other headers)

six of one, half dozen of the other; zeus let's me set a Host header and the path, but that's it. :-(  we have other monitors that look for /robots.txt to work around app issues (I'm looking at you, gitmo); would that be of use here?

Comment 4

[Ed Morley [:emorley]]

(In reply to Kendall Libby [:fubar] from comment #3)
> > Just to clarify - by 'X-FORWARDED-PROTO' I was meaning in the HTTP header,
> > not an environment variable? (We may both be meaning the same thing, it just
> > seemed strange that Zeus would let you set the Host but not other headers)
> 
> six of one, half dozen of the other; zeus let's me set a Host header and the
> path, but that's it. :-(  we have other monitors that look for /robots.txt
> to work around app issues (I'm looking at you, gitmo); would that be of use
> here?

Ah ok.
Unfortunately robots.txt doesn't help us, since it's served by gunicorn too.

Let's just ignore this for now, it's another thing that will go away once we move to Heroku :-)

Thank you for your help anyway.