975687 - aus2.webapp.phx1 timeouts

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Description

[Rick Bryce [:rbryce]]

<nagios-phx1> Sat 00:14:03 PST aus2.webapp.phx1.mozilla.com:PHX Zeus - Port 81 is CRITICAL: aus2.webapp.phx1.mozilla.com down(2) (http://m.mozilla.org/PHX+Zeus+-+Port+81)

Server was under no load to speak of.  Apache connections were well under the limit at the time.  

The error_log shows lots if balrog.wsgi errors prior to my sighup.

Sat Feb 22 08:12:57 2014] [error] [client 10.8.81.215] Script timed out before returning headers: balrog.wsgi

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Ben, any ideas on what how to debug here ?

Comment 2

[[e-mail] (:bhearsum)]

There's nothing in the application log (/var/log/aus.log) nor did Sentry receive any tracebacks. This makes me pretty certain that we didn't get a Traceback from the wsgi script.

Is it just this node that had issues? Maybe the connection to the db was somehow degraded for a period of time?

Comment 3

[[e-mail] (:bhearsum)]

Is this still happening?

Comment 4

[Rick Bryce [:rbryce]]

(In reply to Ben Hearsum [:bhearsum] from comment #3)
> Is this still happening?

Has not happened since.

Comment 5

[[e-mail] (:bhearsum)]

Going to resolve this for now then, because I'm not sure how else to investigate.