768143 - kitten.py should make it clearer if it can't reach a host

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Armen [:armenzg]]

bear could somehow tell but I could not from the output.

(production)[buildduty@cruncher production]$ python kitten.py talos-r3-fed-003 --debug
INFO    debug level is on
DEBUG   Starting
talos-r3-fed-003: enabled
        colo: scl1
      distro: fedora12
        pool: tests-scl1-linux
  trustlevel: try
DEBUG   calling [ping -c 5 talos-r3-fed-003.build.scl1.mozilla.com.]
DEBUG   waiting for remote shell to respond
DEBUG   waiting for remote shell to respond
DEBUG   Running ls -l /builds/slave/buildbot.tac*
DEBUG   waiting for remote shell to respond
DEBUG   Running ls -l /builds/slave/twistd.pid
DEBUG   waiting for remote shell to respond
DEBUG   Running ps ww `cat /builds/slave/twistd.pid`
DEBUG   waiting for remote shell to respond
DEBUG   Running tail -200 /builds/slave/twistd.log
DEBUG   waiting for remote shell to respond
DEBUG   Running tail -10 /builds/slave/twistd.log
DEBUG   waiting for remote shell to respond
INFO        tacfile: NOT FOUND; buildbot: NOT running
        fqdn: talos-r3-fed-003.build.scl1.mozilla.com.
   reachable: True
    buildbot: NOT running
     tacfile: NOT FOUND
    lastseen: unknown
      master: 
       error: current master is different than buildbot.tac master []
       IPMI?: False
DEBUG   Finished
INFO    process shutting down
DEBUG   running all "atexit" finalizers with priority >= 0
DEBUG   running the remaining "atexit" finalizers

Comment 1

[Mike Taylor [:bear]]

changed output to show more clearly that the host was unreachable

https://github.com/mozilla/briar-patch/commit/9cb9df55409cf80169a2fb5a9023ee7f0d3afaed

Comment 2

[Armen [:armenzg]]

This is the new output.
I am not sure if I managed to pull all the changes in since git said that there were some local changes.

(production)[buildduty@cruncher production]$ python kitten.py talos-r3-fed-003 --debug
INFO    debug level is on
DEBUG   Starting
talos-r3-fed-003: enabled
        colo: scl1
      distro: fedora12
        pool: tests-scl1-linux
  trustlevel: try
DEBUG   calling [ping -c 5 talos-r3-fed-003.build.scl1.mozilla.com.]
DEBUG   waiting for remote shell to respond
DEBUG   waiting for remote shell to respond
DEBUG   Running ls -l /builds/slave/buildbot.tac*
DEBUG   waiting for remote shell to respond
DEBUG   Running ls -l /builds/slave/twistd.pid
DEBUG   waiting for remote shell to respond
DEBUG   Running ps ww `cat /builds/slave/twistd.pid`
DEBUG   waiting for remote shell to respond
DEBUG   Running tail -200 /builds/slave/twistd.log
DEBUG   waiting for remote shell to respond
DEBUG   Running tail -10 /builds/slave/twistd.log
DEBUG   waiting for remote shell to respond
ERROR   exception during wait()
Traceback (most recent call last):
  File "/home/buildduty/production/briar-patch/releng/remote.py", line 268, in wait
    self.channel.sendall("\r\n")
  File "/home/buildduty/production/lib/python2.6/site-packages/ssh/channel.py", line 788, in sendall
    raise socket.error('Socket is closed')
error: Socket is closed
INFO        tacfile: NOT FOUND; buildbot: NOT running
        fqdn: talos-r3-fed-003.build.scl1.mozilla.com.
   reachable: True
    buildbot: NOT running
     tacfile: NOT FOUND
    lastseen: unknown
      master:
       error: current master is different than buildbot.tac master []
       IPMI?: False
DEBUG   Finished
INFO    process shutting down
DEBUG   running all "atexit" finalizers with priority >= 0
DEBUG   running the remaining "atexit" finalizers

Comment 3

[Armen [:armenzg]]

Do you mind looking at the checkout and see if you can update it to the latest code?
I would like to keep on testing kitten.py this week.

Comment 4

[Mike Taylor [:bear]]

landed in production