914039 - [stage] Internal server error(500) on https://crash-stats.allizom.org/

Status: VERIFIED FIXED

(Socorro :: Webapp, task)

Description

[Trif Andrei-Alin[:AlinT]]

Attachment: 500_socorro_stage.png

Accessing https://crash-stats.allizom.org/ returns 500 Internal Server Error

Comment 1

[Matt Brandt [:mbrandt]]

I wasn't able to consistently reproduce a 500 error on stage but they do indeed exist and are playing havoc with our automation. Can we please make stage stable again. Noming to blocker.

[09:03:17.583] GET https://crash-stats.allizom.org/topcrasher/products/Firefox/versions/26.0a1?days=7 [HTTP/1.1 500 INTERNAL SERVER ERROR 122304ms]

Not sure which error is most helpful but here are a few from errormill that occurred during the timeframe I was reproducing AlinT's experiences.

https://errormill.mozilla.org/webtools/socorro-stage/group/97549/
https://errormill.mozilla.org/webtools/socorro-stage/group/99799/
https://errormill.mozilla.org/webtools/socorro-stage/group/97559/

Comment 2

[Selena Deckelmann :selenamarie :selena]

Issue was caused by failed Postgres database stage refresh.

I applied passwords this morning, reset crontabbers config and stage started working again.

Comment 3

[Lonnen :lonnen]

so this this resolved now?

Comment 4

[Matt Brandt [:mbrandt]]

(In reply to Chris Lonnen :lonnen from comment #3)
> so this this resolved now?

Stage appears to be in a healthier state; are we still waiting for the bug 914155 to be resolved?

Comment 5

[Laura Thomson :laura]

Working now.

Comment 6

[Matt Brandt [:mbrandt]]

QA verified - we are back in business

Comment 7

[Trif Andrei-Alin[:AlinT]]

Attachment: Screenshot of the issue

This is reproducible again.
I've attached a screenshot of the issue

Comment 8

[Matt Brandt [:mbrandt]]

Bumping to resolved fixed - stage is up again and we were able to get a passing build - http://qa-selenium.mv.mozilla.com:8080/view/Socorro/job/socorro.stage/172/

We are tracking a different behavior where we're seeing spurious 500's on the reports as well as a potential perf regression on queries (unconfirmed & still investigating). Tracker bug is bug 922662.

Comment 9

[Trif Andrei-Alin[:AlinT]]

It seems this is fixed.
I cannot longer reproduce 500s on the stage site
Bumping to VF