1243814 - buildbot-master111 hung running maybe_reconfig.sh during checkconfig

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Jordan Lund (:jlund)]

discovered: 09:28:25 <nagios-releng> Thu 09:28:25 PST [4002] buildbot-master111.bb.releng.scl3.mozilla.com:buildbot is CRITICAL: PROCS CRITICAL: 2 processes with command name buildbot (http://m.mozilla.org/buildbot)


troubleshoot:
[[e-mail] tests1-windows]# ps -ef | grep buildbot
cltbld    1450     1  0  2015 ?        00:27:48 /builds/buildbot/queue/bin/python /builds/buildbot/queue/tools/buildbot-helpers/command_runner.py -q /dev/shm/queue/commands -r 10 -t 60 -m 60 -j4
cltbld    1462     1  0  2015 ?        00:45:53 /builds/buildbot/queue/bin/python /builds/buildbot/queue/tools/buildbot-helpers/pulse_publisher.py --passwords /builds/buildbot/queue/passwords.py -q /dev/shm/queue/pulse
cltbld    8804  8794  0 09:00 ?        00:00:00 /bin/sh -c /builds/buildbot/tests1-windows/tools/buildfarm/maintenance/maybe_reconfig.sh /builds/buildbot/tests1-windows 2>&1 | logger -t maybe_reconfig.sh
cltbld    8808  8804  0 09:00 ?        00:00:00 /bin/bash /builds/buildbot/tests1-windows/tools/buildfarm/maintenance/maybe_reconfig.sh /builds/buildbot/tests1-windows
cltbld    8998  8995  0 09:00 ?        00:00:00 /bin/sh -c cd master && /builds/buildbot/tests1-windows/bin/buildbot checkconfig
cltbld    8999  8998  0 09:00 ?        00:00:01 /builds/buildbot/tests1-windows/bin/python /builds/buildbot/tests1-windows/bin/buildbot checkconfig
root     13952 11776  0 09:42 pts/0    00:00:00 grep buildbot
cltbld   30717     1 20 Jan15 ?        2-15:25:42 /builds/buildbot/tests1-windows/bin/python /builds/buildbot/tests1-windows/bin/buildbot start /builds/buildbot/tests1-windows/master
[[e-mail] tests1-windows]# strace -s 99 -ffp 8999
Process 8999 attached - interrupt to quit
recvfrom(5, ^C <unfinished ...>
Process 8999 detached
[[e-mail] tests1-windows]# lsof -i | grep 8999
buildbot   8999 cltbld    5u  IPv4 56765415      0t0  TCP buildbot-master111.bb.releng.scl3.mozilla.com:49767->ec2-54-215-155-53.us-west-1.compute.amazonaws.com:http (ESTABLISHED)
[[e-mail] tests1-windows]# kill 8999

not sure what was happening. there was DNS work being done right before this. could be related. looks like it hung while receiving a connection ^

will leave open incase this has happened before or happens again

Comment 1

[Chris Cooper [:coop] (he/him)]

Hasn't happened again.