730986 - Socorro staging hbase having connection blips

Status: RESOLVED FIXED

(Mozilla Metrics :: Hadoop/HBase Operations, defect)

Description

[Laura Thomson :laura]

The logged error looks like this:
http://lonnen.pastebin.mozilla.org/1493971

Specifically on the HBase side, this looks like the root cause:
IOError(message='org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1b45e2d5 closed')

It seems to recover on its own, but this doesn't seem like a good thing.

Comment 1

[Laura Thomson :laura]

Happening again, but it looks actively down now.  Upping severity, we have a release tomorrow and it would be good it it was fixed.

Comment 2

[T [:tmary] Meyarivan]

AFAIK HBase Master and at least one HBase RS are alive (node7.generic.metrics.sjc1.mozilla.com) - thrifttester service reports that the service is alive - not sure which parts are broken 

--

Comment 3

[T [:tmary] Meyarivan]

Restarting HBase daemons on all nodes - hopefully this fixes the connection issues (temporary)

--

Comment 4

[Daniel Einspanjer [:dre] [:deinspanjer]]

Just to make sure, this is the old hbase staging cluster in sjc1, not the new one in phx, correct?

Comment 5

[T [:tmary] Meyarivan]

(In reply to Daniel Einspanjer :dre [:deinspanjer] from comment #4)
> Just to make sure, this is the old hbase staging cluster in sjc1, not the
> new one in phx, correct?

Yes.

-

HBase Thrift service is up on at least 4 nodes ATM 

--