834403 - Crashes backed up on collectors since Tuesday 1/24, and collectors not catching up

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Laura Thomson :laura]

collector04 is behaving normally.

All the others are running extremely slowly and experiencing many HBase connection failures, too.  collector01 has nearly 200k crashes backed up on disk.

I also note that the crashes are all in date/[date]/00/00 instead of being distributed over subdirectories, which is surely not helping.

Problem appears to have started around 7am PT on Tuesday 1/22

Current status is instrumenting a collector (01) to get more information.

Comment 1

[Selena Deckelmann :selenamarie :selena]

Still debugging the issue. 

I recommend that we come up with two new monitors: 

* "acceptable number of timeouts when talking to hbase" - in the past this seems to stay below 10-15/hr
* "how many files we have per directory" on each collector

Comment 2

[Robert Helmer [:rhelmer]]

Attachment: debug logging for instrumenting collector01

This is running on collector01, just eyeballing the times it looks like removing files from the filesystem is taking the longest by far.

Comment 3

[Selena Deckelmann :selenamarie :selena]

Update: 

:jakem has gotten all collectors back online with fresh new primaryCrashStore directories.

We took sp-collector03 out of zeus pool and we are processing the old crashes. This is proceeding at a rate of about 2k crashes per minute, on a backlog of about 490k.

Comment 4

[Laura Thomson :laura]

More info: we rolled collectors and crash movers back to Socorro 33 (pre-multi-dump).  We will need to roll them forward again once the storage bug is fixed.

Comment 5

[Laura Thomson :laura]

I should also point out that once we complete backlog processing, we will need to backfill the days 1/22- 1/24

Comment 6

[Selena Deckelmann :selenamarie :selena]

sp-collector03 has completed backlog. :jakem is starting up processing for backlog on sp-collector02.

Comment 7

[Selena Deckelmann :selenamarie :selena]

Another note - socorro team needs sudo access as apache/socorro user on sp-collectorXX nodes.

Comment 8

[Jake Maul [:jakem]]

sp-collector02 is completed. sp-collector06 was already completed earlier. 01 is underway.

Comment 9

[Jake Maul [:jakem]]

01 and 04 are completed. 05 is underway... but proving more problematic than the others did.

Comment 10

[Laura Thomson :laura]

Where did we get to?  Still need to run backfill after this completes, too.

Comment 11

[Selena Deckelmann :selenamarie :selena]

:mpressman is running backfill now per IRC.

Comment 12

[Selena Deckelmann :selenamarie :selena]

Backfill was complete by 1/29.