1185278 - Crash processing ceased

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[JP Schneider [:jp]]

Attachment: processors-not-processing.log

Last evening, 7/17 at approximately 8pm PDT, processors stopped processing.  

https://www.dropbox.com/s/bidoexcfiua0rmv/Screenshot%202015-07-18%2013.25.35.JPG

Surprisingly, the alert did not trigger for this, though it appears settings are correct.  
https://dl.dropboxusercontent.com/u/2273146/processing-alert-settings.pdf

Upon noticing this, I grabbed the attached log, and then killed off all current processors.  The new processors started processing correctly.

As rhelmer mentioned in irc, the two issues we want to address are why systemd didn't restart the processors, and why the alert did not trigger.

Comment 1

[Robert Kaiser]

We should see if we need to backfill for this - reports_clean usually gets updates hourly or so and if we miss a few hours, data will look missing in reports.

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to Robert Kaiser (:[e-mail]) from comment #1)
> We should see if we need to backfill for this - reports_clean usually gets
> updates hourly or so and if we miss a few hours, data will look missing in
> reports.

Good point, thanks... filed bug 1185309 for that.

Comment 3

[JP Schneider [:jp]]

We did the backfill, we changed the settings in datadog to make the alert trigger properly, and we have figured out/resolved why processors were not restarting.