Last Comment Bug 836845 - Crashes backed up on collectors
: Crashes backed up on collectors
Status: RESOLVED FIXED
:
Product: Socorro
Classification: Server Software
Component: General (show other bugs)
: unspecified
: All All
-- blocker (vote)
: ---
Assigned To: Selena Deckelmann :selenamarie :selena use ni?
:
:
Mentors:
Depends on: 834403 834481 834485 834546 834812
Blocks: 836619
  Show dependency treegraph
 
Reported: 2013-01-31 11:21 PST by Selena Deckelmann :selenamarie :selena use ni?
Modified: 2013-04-01 12:29 PDT (History)
9 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description User image Selena Deckelmann :selenamarie :selena use ni? 2013-01-31 11:21:19 PST
similar to #834403

All collectors have slowed to 20-25 crashes pushed into HBase per minute.
Comment 1 User image Selena Deckelmann :selenamarie :selena use ni? 2013-01-31 11:23:47 PST
This bug triggered investigation that revealed we had a lot of crashes backed up on collector05 and then multiple collectors were very slow: https://bugzilla.mozilla.org/show_bug.cgi?id=836619
Comment 2 User image Selena Deckelmann :selenamarie :selena use ni? 2013-01-31 11:51:44 PST
Need to run backfill for 1/30 once crashmover catches up
Comment 3 User image Laura Thomson :laura 2013-01-31 12:41:58 PST
Collectors and crash movers have once again been rolled back to Socorro version 33 (as we did in bug 834403, which makes them fast again.  Lars is attempting to reproduce in stage.  Problem appears to be a crash mover performance issue of some kind, as we are not seeing HBase errors.

Once crash movers are caught up, selena will run backfill.
Comment 4 User image Jake Maul [:jakem] 2013-01-31 17:22:22 PST
Collectors are all caught up. :)
Comment 5 User image Selena Deckelmann :selenamarie :selena use ni? 2013-01-31 19:18:38 PST
Waiting to run backfill until the replica is done being recreated.
Comment 6 User image Selena Deckelmann :selenamarie :selena use ni? 2013-02-01 06:41:07 PST
Running backfill
Comment 7 User image Selena Deckelmann :selenamarie :selena use ni? 2013-04-01 12:29:19 PDT
Backfill was run long ago!

Note You need to log in before you can comment on or make changes to this bug.