Created attachment 8635733 [details] processors-not-processing.log Last evening, 7/17 at approximately 8pm PDT, processors stopped processing. https://www.dropbox.com/s/bidoexcfiua0rmv/Screenshot%202015-07-18%2013.25.35.JPG Surprisingly, the alert did not trigger for this, though it appears settings are correct. https://dl.dropboxusercontent.com/u/2273146/processing-alert-settings.pdf Upon noticing this, I grabbed the attached log, and then killed off all current processors. The new processors started processing correctly. As rhelmer mentioned in irc, the two issues we want to address are why systemd didn't restart the processors, and why the alert did not trigger.
We should see if we need to backfill for this - reports_clean usually gets updates hourly or so and if we miss a few hours, data will look missing in reports.
(In reply to Robert Kaiser (:firstname.lastname@example.org) from comment #1) > We should see if we need to backfill for this - reports_clean usually gets > updates hourly or so and if we miss a few hours, data will look missing in > reports. Good point, thanks... filed bug 1185309 for that.
We did the backfill, we changed the settings in datadog to make the alert trigger properly, and we have figured out/resolved why processors were not restarting.