Closed Bug 1137752 Opened 10 years ago Closed 10 years ago

Reprocess crashes from brief filer outage from 8am-8:20am 2/27/2015

Categories

(Socorro :: Infra, task)

Product:
Socorro
Component:
Infra
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: selenamarie, Assigned: selenamarie)

Details

The NFS filer needed to be patched and was restarted around 8am PT today. It caused about ~4000 timeout errors in the processors. I'm going to reprocess those crashes as that number of errors seems significant enough to cause issues with reporting.
Crashes to reprocess turned out to be 8122.
System reprocessing speed was about 10 crashes per second. Now verifying reports_clean (may need to re-run for 7-9am).
Note for future travellers: Docs on monitoring rabbit are here: https://mana.mozilla.org/wiki/display/NAGIOS/WebOps+Rabbit+Unread+Messages Added a script for directly pushing crashes into rabbit PR: https://github.com/mozilla/socorro/pull/2646
Assignee: nobody → sdeckelmann
backfilled reports_clean: breakpad=# select backfill_reports_clean('2015-02-27 14:00:00+00', '2015-02-27 17:00:00+00');
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/65f69ff9deba8f5e5e9b3a8a7f5b62c95fae5450 Fixes bug 1137752 adds reprocessing submission script https://github.com/mozilla/socorro/commit/8771d58afeebe2d10c502ec0b9caea64044c426f Merge pull request #2646 from selenamarie/bug1137752-reprocess-crashlist-script Fixes bug 1137752 adds reprocessing submission script - "socorro reprocess_crashlist"
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.