615577 - 500 errors on Jpake stage

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Tarek Ziadé (:tarek)]

From Richard:to confirm, you've run the server on a redhat server at mozilla with observe mode enabled using the rpms built by hg tip, and you are not seeing sporadic death after 5-10 minutes?

I cannot reproduce the problem so far, and I did not get any debug information from stage. We need to backport the deployment done in stage on our dev, environment so it's similar in every way. The dev server currently has no daemontools configured for instance. There's also no nagios health check done on dev (I can reproduce it via a script though.)

I also suggest to set the debug flag on in stage so we can get verbose traceback on errors for feedback. This can be done today and will let us fix the bug right away, until the dev environment is up to date.

Comment 1

[:Atoll]

weave-dev-jpake01 and 02 are both fully configured for daemontools and nginx.  you can simply "killall nginx gunicorn" to trigger a restart, or do it the daemontools way.  they probably have an older jpake installed and configured, so be sure to check for cfg.rpmnew files after an upgrade.

Comment 2

[:Atoll]

also, they are dev-setup.services.mozilla.com when you are connected to the mozilla office vpn, and zeus currently shows them as "up and available".

Comment 3

[Tarek Ziadé (:tarek)]

Oh great, I didn't know the dev infra was up to date. I will try to reproduce the issue now in dev, via a script.

Comment 4

[Tarek Ziadé (:tarek)]

Richard was able to get the TB in stage with the debug flag, and I found and fix the issue.

Pushed at
http://hg.mozilla.org/services/server-key-exchange/rev/f8062109b396

Comment 5

[:Atoll]

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/gunicorn/workers/sync.py", line 101, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/usr/lib/python2.6/site-packages/keyexchange/filtering/middleware.py", line 255, in __call__
    self._check_ip(ip, environ)
  File "/usr/lib/python2.6/site-packages/keyexchange/filtering/middleware.py", line 184, in _check_ip
    self.callback(ip, environ)
  File "/usr/lib/python2.6/site-packages/keyexchange/wsgiapp.py", line 297, in blacklisted
    signature=_BLACKLISTED)
  File "/usr/lib/python2.6/site-packages/services/cef.py", line 178, in log_failure
    'dest': _convert(environ['HTTP_HOST']),
KeyError: 'HTTP_HOST'

Comment 6

[Tarek Ziadé (:tarek)]

Added a specific bug 615681. I suggest to do this for every specific issue.