691811 - Unhork support-release.allizom.org

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[James Socol [:jsocol, :james]]

support-release.allizom.org (mrapp-stage04) is in a bad way right now, which is blocking final QA of our release this afternoon.

It's sending Apache-style 500 Server Errors, which means we're not getting emailed stack traces, and they're probably ending up in the Apache error logs, and I don't have access to mrapp-stage04 anymore.

May be database related, since the last error we did get said "Lock wait timeout exceeded," but probably not, since those would likely still send email tracebacks.

Looking forward to the days of unshared staging environments, but for today we need this fixed ASAP to unblock QA. Thanks!

Comment 1

[Jeremy Orem [:oremj]]

Saw a lot of Premature end of script headers: kitsune.wsgi, which usually means the wsgi workers have all died off. This seems to happen occasionally in dev/staging envs. I think it is due to the frequent restarts.

Ping me if you are still having problems.

Btw, to fix I just restarted apache.

Comment 2

[James Socol [:jsocol, :james]]

Thanks, Jeremy!

For posterity, we tracked this down to RabbitMQ being a jerk with connections, and our default settings for Celery of TIMEOUT=4sec, RETRIES=100.