822375 - Thrift pages

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[Laura Thomson :laura]

Thrift paged and recovered this morning.  Since we are looking into assorted connection problems on the Socorro side, I thought I'd get a bug on file to understand why.

This was the message:

Service: Socorro - Thrift connection
Host: sp-admin01.phx1.mozilla.com
Address: 10.8.75.29
State: CRITICAL

Date/Time: 12-17-2012 05:43:03

Additional Info:

CHECK_NRPE: Socket timeout after 10 seconds.

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

Tmary, could you look around in our server logs and see if this outage correlates with any events on our side?

Comment 2

[Corey Shields [:cshields]]

fwiw, I disabled timeouts in zeus for the thrift VIPs.

Comment 3

[T [:tmary] Meyarivan]

(In reply to Laura Thomson :laura from comment #3)
> State: CRITICAL
> Date/Time: 12-17-2012 05:43:03
> CHECK_NRPE: Socket timeout after 10 seconds.

"10 seconds" <-- seems like the NRPE cmd is timing out ? (IIRC 10s is the default timeout for check_nrpe) 

Re HBase Thrift events etc, none around that time

-

Re check_thrift_connection.sh, it hangs sometimes (forever ?) while processing exceptions etc

From trace logs:

"""
connect(5, {sa_family=AF_FILE, path="/var/run/abrt/abrt.socket"}, 27
"""

"""
#0  0x000000326a20e890 in __connect_nocancel () from /lib64/libpthread.so.0
#1  0x00007faf268cb0d8 in ?? () from /usr/lib64/python2.6/lib-dynload/_socketmodule.so
#2  0x00007faf268cd15c in ?? () from /usr/lib64/python2.6/lib-dynload/_socketmodule.so
#3  0x000000326c6de706 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0
#4  0x000000326c6e0797 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0
#5  0x000000326c6debe4 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0
#6  0x000000326c6dfccf in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0
#7  0x000000326c6dfccf in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0
#8  0x000000326c6e0797 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0
#9  0x000000326c66edb0 in ?? () from /usr/lib64/libpython2.6.so.1.0
#10 0x000000326c644303 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
#11 0x000000326c6d8dd3 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.6.so.1.0
#12 0x000000326c6fccd2 in PyErr_PrintEx () from /usr/lib64/libpython2.6.so.1.0
#13 0x000000326c6fd297 in PyRun_SimpleFileExFlags () from /usr/lib64/libpython2.6.so.1.0
#14 0x000000326c709c32 in Py_Main () from /usr/lib64/libpython2.6.so.1.0
#15 0x0000003269a1ecdd in __libc_start_main () from /lib64/libc.so.6
#16 0x0000000000400649 in _start ()
"""

Comment 4

[Daniel Maher [:phrawzty]]

This bug is quite old.  Anybody know if it's still valid?

Comment 5

[Lonnen :lonnen]

With a goal of moving from hbase to s3 next Q, I'm not sure that we would worry about it if it was. This is over a year old and we can reopen it if it crops up again.