968965 - MozTrap-dev.allizom.org is down/returning a Zeus 500 error

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

https://moztrap-dev.allizom.org/ is returning a 500 from Zeus

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Stephen Donner [:stephend] from comment #0)
> https://moztrap-dev.allizom.org/ is returning a 500 from Zeus

And it just came back up; Web Ops, what fixed this (again)?  Apache restart?  Stuck WSGI processes?

Comment 2

[Chris Turra [:cturra]]

i have dropped the importance to normal here for a couple reasons: 1) it has resolved itself* and (2) this was only impacting dev, not stage or prod.

*there are several auto deploy jobs in dev that can restart apache (usually gracefully, but not always). there doesn't appear to have been anyone in webops that was working on the system at the time nor to take action to resolve this. my best guess after some quick look through logs is that your tests caught a deploy that may have taken a bit of time to restart apache. if zeus cannot connect to the web head, it returns a 500 (ugly, i know). 

i am going to mark this bug as r/invalid since there is nothing that was technically "fixed." this said, if your automation continues to see this, please reopen. detailed information about when it started/stopped would be good also to see if we can pin it down to one application doing deployments.

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

Thx, :cturra; marking as WFM because it technically is.  We don't run tests against this project or its dev/staging servers all that frequently, so we only noticed it around 10:46am PST; it appears to have lasted until 11:01am PST (http://qa-selenium.mv.mozilla.com:8080/view/MozTrap/job/moztrap.dev.saucelabs/572/ + http://qa-selenium.mv.mozilla.com:8080/view/MozTrap/job/moztrap.dev/437/, respectively).