socorro staging access to hbase is down

RESOLVED FIXED in Unreviewed

Status

Mozilla Metrics
Metrics Operations
--
major
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: lars, Assigned: tmary)

Tracking

unspecified
Unreviewed
x86_64
Linux

Details

(Reporter)

Description

6 years ago
the processors in Socorro staging are unable to talk to HBase.  Every thread is getting a socket read error:

2012-06-19 12:36:38,262 CRITICAL - Thread-2 - File "/data/socorro/thirdparty/thrift/transport/TSocket.py", line 108, in read
    raise TTransportException(type=TTransportException.END_OF_FILE, message='TSocket read 0 bytes')
2012-06-19 12:36:38,262 CRITICAL - Thread-2 - FatalException: the connection is not viable.  retries fail:
2012-06-19 12:36:38,263 CRITICAL - Thread-2 - major failure in crash storage - retry in 300 seconds
(Reporter)

Comment 1

6 years ago
hbaseHost=10.8.100.209
hbasePort=9090

Updated

6 years ago
Assignee: server-ops → nobody
Group: metrics-private
Severity: normal → major
Component: Server Operations: Web Operations → Metrics Operations
Product: mozilla.org → Mozilla Metrics
QA Contact: cshields → metrics-operations
Version: other → unspecified

Updated

6 years ago
Group: metrics-private
(Assignee)

Comment 2

6 years ago
Please check/verify - should be up now

--
(Assignee)

Updated

6 years ago
Assignee: nobody → tmeyarivan
(Reporter)

Comment 3

6 years ago
the monitor running on socorroadm.stage.private.phx1.mozilla.com are having no trouble with HBase.

the processors running on socorro-processor1.stage.metrics.phx1.mozilla.com still _cannot_ connect to HBase.
(Reporter)

Comment 4

6 years ago
I was about to write that the problem is resolved.  Suddenly, the situation is reversed, the processor is talking with HBase, but the monitor is giving connection errors.
(Reporter)

Comment 5

6 years ago
and just as suddenly, everything is fine...
(Reporter)

Comment 6

6 years ago
and it's all broken again moments later...
(Assignee)

Comment 7

6 years ago
As per last IRC msg, its working as expected

--
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.