868414 - kittenherder misinterprets reboot output

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[John Hopkins (:jhopkins)]

When kittenherder reboots a machine, it can misinterpret the status of a successful reboot and add the build slave to the recovery list.
Example log snippet:

-----
^G^M
Broadcast message from cltbld@talos-linux32-ix-037^M
        (/dev/pts/1) at 5:34 ...^M
^M
^MThe system is going down for reboot NOW!^M
cltbld@talos-linux32-ix-037:~$ 
cltbld@talos-linux32-ix-037:~$ 
cltbld@talos-linux32-ix-037:~$ 
cltbld@talos-linux32-ix-037:~$ 
INFO    reboot failed, forcing recovery flag
----

Note above the reboot command succeeded but kittenherder thought it failed.

Comment 1

[John Hopkins (:jhopkins)]

Attachment: [briar-patch] allow ignoring of run_cmd output, assume reboot will succeed

I've added an argument to run_cmd, fetch_output=True, to allow one to avoid fetching output which is what we need to do when running the 'sudo reboot' command.  In that case, no output is generated and the SSH connection is closed immediately.  This was confusing kittenherder into thinking the reboot had failed.

Now the UNIX type reboots will run a test command ("echo test") just prior to the "sudo reboot" command.  If the test command returns the expected output, I'm assuming the reboot will complete as well and return True.  If the test does not return the expected output, I immediately return False without attempting to reboot.

Comment 2

[John Hopkins (:jhopkins)]

Comment on attachment 745291 [details] [diff] [review]
[briar-patch] allow ignoring of run_cmd output, assume reboot will succeed

Landed in https://github.com/mozilla/briar-patch/commit/88f5e63ac98142a92b28ddaccc5dfbfd9fe712f5 and deployed.