Unhork support-release.allizom.org

VERIFIED FIXED

Status

--
blocker
VERIFIED FIXED
7 years ago
4 years ago

People

(Reporter: jsocol, Assigned: oremj)

Tracking

Details

(Reporter)

Description

7 years ago
support-release.allizom.org (mrapp-stage04) is in a bad way right now, which is blocking final QA of our release this afternoon.

It's sending Apache-style 500 Server Errors, which means we're not getting emailed stack traces, and they're probably ending up in the Apache error logs, and I don't have access to mrapp-stage04 anymore.

May be database related, since the last error we did get said "Lock wait timeout exceeded," but probably not, since those would likely still send email tracebacks.

Looking forward to the days of unshared staging environments, but for today we need this fixed ASAP to unblock QA. Thanks!
Assignee: server-ops → dgherman
(Assignee)

Updated

7 years ago
Assignee: dgherman → jeremy.orem+bugs
(Assignee)

Comment 1

7 years ago
Saw a lot of Premature end of script headers: kitsune.wsgi, which usually means the wsgi workers have all died off. This seems to happen occasionally in dev/staging envs. I think it is due to the frequent restarts.

Ping me if you are still having problems.

Btw, to fix I just restarted apache.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Reporter)

Comment 2

7 years ago
Thanks, Jeremy!

For posterity, we tracked this down to RabbitMQ being a jerk with connections, and our default settings for Celery of TIMEOUT=4sec, RETRIES=100.
Status: RESOLVED → VERIFIED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.