978523 - collectors sporadically choking

Status: RESOLVED WORKSFORME

(Socorro :: Backend, task)

Description

[Lonnen :lonnen]

Twice so far we've had a collector go critical because input was far greater than output.

Socorro Crash Volume is CRITICAL: CRITICAL: 1803 is NOT within 20% of 843 (http://m.mozilla.org/Socorro+Crash+Volume)

Bug 978416 tracked the initial incident, and tonight collector6 presented with the error above. I had rbyrce drain it in zeus. There were only 17 connections to it at the time.

Comment 1

[Lonnen :lonnen]

rbryce added collector6 back to the pool and all status is green now.

Comment 2

[Laura Thomson :laura]

The time we had a lot of trouble with this was when there was a change to the file system walking code the collectors/crashmovers use - don't remember exactly what, but I can dig up the bug (jakem or selena or lars may remember faster than me). Did any of that change lately?

Another thing that has previously caused this is (a higher than normal rate of) hbase/thrift timeouts: those will be in logs. If that's the issue, kick thrift.

Comment 3

[Laura Thomson :laura]

The time I was thinking of was bug 849529 / bug 836845. Root cause in bug 849566. I suspect that is unrelated, fwiw.

While I was spelunking I came across another instance where tmary was running an MR job at the time, causing HBase timeouts, which is another possibility.

Comment 4

[K Lars Lohn [:lars] [:klohn]]

I've looked at the logs on collector6 and compared the HBase timeout rates with the other collectors and see nothing unusual.  investigation continues.

Comment 5

[K Lars Lohn [:lars] [:klohn]]

Attachment: hour by hour comparison of collector06 and crashmover06

I've gone through the logs for collector06 and crashmover06 hour by hour for the  entire week of 2014-02-23 through 2014-03-01. I found no time where they were out of sync between total collected vs total moved into HBase.  

My conclusion is that this is a false alarm.

Comment 6

[K Lars Lohn [:lars] [:klohn]]

Attachment: focusing on 2014-02-28 - this is a 10 minute analysis

Focusing on 2014-02-28, I've compared colletor06 totals with crashmover06 totals at 10 minute intervals and again found no trouble.  The crashmovers were never anything more than trivially behind the collectors.

Comment 7

[Lonnen :lonnen]

I filed bug 800009 to get documentation for that alert, but the bug is restricted even to me after filing.

Comment 8

[Lonnen :lonnen]

ericz: It checks the number of crashes submitted to hbase vs incoming crashes IIRC 
ericz: But I'm not working today so I can look at it more tomorrow.  :)

Comment 9

[Lonnen :lonnen]

We're out of the data center now, and this monitor is no longer active.