Status

Socorro
Infra
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: jp, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
Created attachment 8635733 [details]
processors-not-processing.log

Last evening, 7/17 at approximately 8pm PDT, processors stopped processing.  

https://www.dropbox.com/s/bidoexcfiua0rmv/Screenshot%202015-07-18%2013.25.35.JPG

Surprisingly, the alert did not trigger for this, though it appears settings are correct.  
https://dl.dropboxusercontent.com/u/2273146/processing-alert-settings.pdf

Upon noticing this, I grabbed the attached log, and then killed off all current processors.  The new processors started processing correctly.

As rhelmer mentioned in irc, the two issues we want to address are why systemd didn't restart the processors, and why the alert did not trigger.

Comment 1

2 years ago
We should see if we need to backfill for this - reports_clean usually gets updates hourly or so and if we miss a few hours, data will look missing in reports.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #1)
> We should see if we need to backfill for this - reports_clean usually gets
> updates hourly or so and if we miss a few hours, data will look missing in
> reports.

Good point, thanks... filed bug 1185309 for that.
(Reporter)

Comment 3

2 years ago
We did the backfill, we changed the settings in datadog to make the alert trigger properly, and we have figured out/resolved why processors were not restarting.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.