622926 - runslave.py does not recognize hostnames like 'try-mac64-slave11'

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

It's looking for 'darwin' in the hostname, not 'mac'.

This has all of the mac64 (and all mac?) builders down at the moment.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m622926-puppet-manifets-r1.patch

Quick fix.

Longer-term, this logic should not be in runslave.py.  Maybe it could extract the basedir from the downloaded buildbot.tac?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

OK, committed in 50e8212bf394, rolled out to all puppet masters, and then all affected machines rebooted with:

env.hosts = [ 'try-mac-slave%02d' % s for s in range(1, 48) ] + [ 'try-mac64-slave%02d' % s for s in range(1, 27) ]
def reboot():
    hostname = env.host_string.split('.')[0]
    if 'try-mac64-slave08' in hostname: return
    env.host_string = 'root@%s.build.mozilla.org' % hostname
    try:
        with hide('stdout'):
            if run('ps auxww | grep [b]uildbot || true').strip() != '': 
                print "buildbot still running; skipping"
                return
            run('/sbin/reboot', pty=True)
    except SystemExit:
        pass
    except:
        print sys.exc_info()
        raise

    for k in connections.keys():
        connections[k].close()
        del connections[k]

Based on the output of this script, none of the try-mac slaves were affected, but I'm not sure why..

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Solved: the try-mac slaves were not able to execute /sbin/reboot due to a bug in fabric (I think it's getting confused by the shell output in 'ps auxww'?).  Closing and re-opening the connection to run the shutdown fixes this.

All functional try-mac-slaveNN should be coming up now.