978247 - slaveapi errors out on some ssh commands

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Justin Wood (:Callek)]

For some reason I haven't seen this in production in a while, however I just setup my own staging run of slaveapi and saw this while testing

I added a log.debug("DATA: %s" % repr(data)) just before the failing line and got:

    2014-02-28 12:42:17,813 - INFO - 10.26.56.16 - Connection as root succeeded!
    2014-02-28 12:42:17,814 - DEBUG - 10.26.56.16 - Running reboot
    2014-02-28 12:42:23,832 - DEBUG - DATA: 'reboot\r\n\r\necho $?\r\n\r\nLast login: Mon Dec 16 09:01:30 2013 from admin1a.private.s
    cl3.mozilla.com\r\r\nThis is t-mavericks-r5-003.test.releng.scl3.mozilla.com (10.26.56.16)\r\n[[e-mail]
    g.scl3.mozilla.com ~]# \r\n[[e-mail] ~]# \r\n[[e-mail]
    3.mozilla.com ~]# echo SHELL_READY\r\nSHELL_READY\r\n[[e-mail] ~]# \r\n[root@t-maveri
    cks-r5-003.test.releng.scl3.mozilla.com ~]# reboot\r\n[[e-mail] ~]# \r\n[root@t-maver
    icks-r5-003.test.releng.scl3.mozilla.com ~]# echo $?\r\n0\r\n[[e-mail] ~]# \r\n[root@
    t-mavericks-r5-003.test.releng.scl3.mozilla.com ~]# '
    2014-02-28 12:42:23,832 - DEBUG - 10.26.56.16 - Caught exception while running command:
    Traceback (most recent call last):
      File "/home/jwood/slaveapi/repo/slaveapi/clients/ssh.py", line 125, in run_cmd
        output, status = data.split("echo $?\r\n")
    ValueError: too many values to unpack
    2014-02-28 12:42:23,840 - WARNING - t-mavericks-r5-003 - Eating exception during SSH reboot. 


(the sporadic word wrap was my console not the actual data), looks like the issue is that the ssh terminal on at least some hosts (mavericks in this case) was echoing back to the user, giving more than one result for "echo $?" in the data