Closed Bug 868414 Opened 7 years ago Closed 7 years ago

kittenherder misinterprets reboot output

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

x86
Linux
task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jhopkins, Assigned: jhopkins)

References

Details

Attachments

(1 file)

When kittenherder reboots a machine, it can misinterpret the status of a successful reboot and add the build slave to the recovery list.
Example log snippet:

-----
^G^M
Broadcast message from cltbld@talos-linux32-ix-037^M
        (/dev/pts/1) at 5:34 ...^M
^M
^MThe system is going down for reboot NOW!^M
cltbld@talos-linux32-ix-037:~$ 
cltbld@talos-linux32-ix-037:~$ 
cltbld@talos-linux32-ix-037:~$ 
cltbld@talos-linux32-ix-037:~$ 
INFO    reboot failed, forcing recovery flag
----

Note above the reboot command succeeded but kittenherder thought it failed.
I've added an argument to run_cmd, fetch_output=True, to allow one to avoid fetching output which is what we need to do when running the 'sudo reboot' command.  In that case, no output is generated and the SSH connection is closed immediately.  This was confusing kittenherder into thinking the reboot had failed.

Now the UNIX type reboots will run a test command ("echo test") just prior to the "sudo reboot" command.  If the test command returns the expected output, I'm assuming the reboot will complete as well and return True.  If the test does not return the expected output, I immediately return False without attempting to reboot.
Attachment #745291 - Flags: review?(coop)
Attachment #745291 - Flags: review?(coop) → review+
Comment on attachment 745291 [details] [diff] [review]
[briar-patch] allow ignoring of run_cmd output, assume reboot will succeed

Landed in https://github.com/mozilla/briar-patch/commit/88f5e63ac98142a92b28ddaccc5dfbfd9fe712f5 and deployed.
Attachment #745291 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.