Closed
Bug 868414
Opened 8 years ago
Closed 8 years ago
kittenherder misinterprets reboot output
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: jhopkins, Assigned: jhopkins)
References
Details
Attachments
(1 file)
1.30 KB,
patch
|
coop
:
review+
jhopkins
:
checked-in+
|
Details | Diff | Splinter Review |
When kittenherder reboots a machine, it can misinterpret the status of a successful reboot and add the build slave to the recovery list. Example log snippet: ----- ^G^M Broadcast message from cltbld@talos-linux32-ix-037^M (/dev/pts/1) at 5:34 ...^M ^M ^MThe system is going down for reboot NOW!^M cltbld@talos-linux32-ix-037:~$ cltbld@talos-linux32-ix-037:~$ cltbld@talos-linux32-ix-037:~$ cltbld@talos-linux32-ix-037:~$ INFO reboot failed, forcing recovery flag ---- Note above the reboot command succeeded but kittenherder thought it failed.
Assignee | ||
Comment 1•8 years ago
|
||
I've added an argument to run_cmd, fetch_output=True, to allow one to avoid fetching output which is what we need to do when running the 'sudo reboot' command. In that case, no output is generated and the SSH connection is closed immediately. This was confusing kittenherder into thinking the reboot had failed. Now the UNIX type reboots will run a test command ("echo test") just prior to the "sudo reboot" command. If the test command returns the expected output, I'm assuming the reboot will complete as well and return True. If the test does not return the expected output, I immediately return False without attempting to reboot.
Attachment #745291 -
Flags: review?(coop)
Updated•8 years ago
|
Attachment #745291 -
Flags: review?(coop) → review+
Assignee | ||
Comment 2•8 years ago
|
||
Comment on attachment 745291 [details] [diff] [review] [briar-patch] allow ignoring of run_cmd output, assume reboot will succeed Landed in https://github.com/mozilla/briar-patch/commit/88f5e63ac98142a92b28ddaccc5dfbfd9fe712f5 and deployed.
Attachment #745291 -
Flags: checked-in+
Assignee | ||
Updated•8 years ago
|
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Updated•8 years ago
|
Product: mozilla.org → Release Engineering
Updated•3 years ago
|
Product: Release Engineering → Infrastructure & Operations
Updated•1 year ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•