Closed Bug 868414 Opened 7 years ago Closed 7 years ago
kittenherder misinterprets reboot output
When kittenherder reboots a machine, it can misinterpret the status of a successful reboot and add the build slave to the recovery list. Example log snippet: ----- ^G^M Broadcast message from cltbld@talos-linux32-ix-037^M (/dev/pts/1) at 5:34 ...^M ^M ^MThe system is going down for reboot NOW!^M cltbld@talos-linux32-ix-037:~$ cltbld@talos-linux32-ix-037:~$ cltbld@talos-linux32-ix-037:~$ cltbld@talos-linux32-ix-037:~$ INFO reboot failed, forcing recovery flag ---- Note above the reboot command succeeded but kittenherder thought it failed.
I've added an argument to run_cmd, fetch_output=True, to allow one to avoid fetching output which is what we need to do when running the 'sudo reboot' command. In that case, no output is generated and the SSH connection is closed immediately. This was confusing kittenherder into thinking the reboot had failed. Now the UNIX type reboots will run a test command ("echo test") just prior to the "sudo reboot" command. If the test command returns the expected output, I'm assuming the reboot will complete as well and return True. If the test does not return the expected output, I immediately return False without attempting to reboot.
Attachment #745291 - Flags: review?(coop)
Attachment #745291 - Flags: review?(coop) → review+
Comment on attachment 745291 [details] [diff] [review] [briar-patch] allow ignoring of run_cmd output, assume reboot will succeed Landed in https://github.com/mozilla/briar-patch/commit/88f5e63ac98142a92b28ddaccc5dfbfd9fe712f5 and deployed.
Attachment #745291 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.