[stage] Stage is throwing sporadic 500 errors

VERIFIED FIXED in 61

Status

--
blocker
VERIFIED FIXED
5 years ago
5 years ago

People

(Reporter: mbrandt, Unassigned)

Tracking

unspecified

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 years ago
Stage sporadically throws 500 errors however they don't appear to be logged by sentry. Errormill is empty.

[14:43:54.002] GET https://crash-stats.allizom.org/home/frontpage_json?product=Firefox&versions=25.0b1 [HTTP/1.1 500 INTERNAL SERVER ERROR 60736ms]
60,736 milliseconds. I'm guessing what's happening there is that the middleware is unavailable and our code that talks to the middleware is retrying patiently until it eventually has to give up.
(Reporter)

Comment 2

5 years ago
The 500's are making it difficult to verify release bugs and are hurting our automated tests.
Seeing lots of timeouts:

[Wed Sep 25 14:02:36 2013] [error] [client 10.8.81.216] Script timed out before returning headers: webservices.py
(In reply to Robert Helmer [:rhelmer] from comment #3)
> Seeing lots of timeouts:
> 
> [Wed Sep 25 14:02:36 2013] [error] [client 10.8.81.216] Script timed out
> before returning headers: webservices.py

This is from socorro-mware1.stage.webapp:/var/log/httpd/error_log btw ^

Presumably it is in turn timing out on something else, postgres?
We're seeing high load on the DB, https://nagios.mozilla.org/phx1/cgi-bin/status.cgi?host=socorro1.stage.db.phx1.mozilla.com

Top cpu process are all postgres processes running SELECT

[root@socorro1.stage.db.phx1 ~]# top -b -n 1 | head -n 12  | tail -n 5
23383 postgres  20   0 9033m 4.6g 4.3g D 51.8 19.6  46:20.40 postgres
23594 postgres  20   0 9023m 4.7g 4.4g R 41.4 20.0  30:35.37 postgres
23700 postgres  20   0 9023m 4.7g 4.4g D 41.4 20.1  30:26.27 postgres
23470 postgres  20   0 9023m 5.1g 4.8g R 37.5 21.5  30:44.65 postgres
23201 postgres  20   0 9023m 4.0g 3.7g D 34.9 17.0  33:32.88 postgres
We reopened bug 889041 and backed it out to resolve this, BTW. Going to work in there and re-land when it's ready.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 7

5 years ago
Thank you all for pouncing on stage and banishing the goblins. Stage no longer 500's and sentry has tracebacks in it again.

QA verified
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.