432609 - socorro processor not running?

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

+++ This bug was initially created as a clone of Bug #431135 +++

The socorro status page says there are no processors running and no reports pending but I've submitted 3 today:

http://crash-stats.mozilla.com/report/pending/94903e2b-1c36-11dd-b631-001cc4e2bf68
http://crash-stats.mozilla.com/report/pending/6f1287d1-1c31-11dd-ae7e-001cc45a2ce4
http://crash-stats.mozilla.com/report/pending/d25a9847-1c2e-11dd-a5c8-001cc45a2ce4

They stay in pending mode. #developers is trying to isolate which recent checkin is causing Linux builds to crash so a working Socorro would be very helpful.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Dropping severity one position, looks like there may be some crashes getting processed. eg 
  http://crash-stats.mozilla.com/report/index/545b7e8a-1b6e-11dd-a187-001a4bd46e84

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I ran the "reports in the last two hours" query 
 http://crash-stats.mozilla.com/?do_query=1&query_search=signature&query_type=contains&query=&date=&range_value=2&range_unit=hours
That's only returning windows crashes - I'd expect at least one crash on the other platforms. Mac/Linux crashes seem to have ceased sometime between 4 and 5 hours before this post.

Only the first of the crashes I gave will have a valid stack, the other two are from an hourly build.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Ok, convinced now this really is busted.

Comment 4

[Aravind Gottipati [:aravind]]

restarted them.  Looking into why this didn't page the oncall.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Probably because I futzed with the severity when thought it might just be my reports.

Comment 6

[Aravind Gottipati [:aravind]]

No, we have monitoring on that status page, should have been paged about it when it started happening.

Comment 7

[Aravind Gottipati [:aravind]]

Ah, it was set to notify only during the day :(

Comment 8

[Michael Morgan [:morgamic]]

Weak!

Comment 9

[Michael Morgan [:morgamic]]

(though it's really our fault and we'll figure out why it's dying randomly)