616350 - slave-side slave-alloc support for darwin build and test

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Similar to bug 616003, but for darwin.

Send out runslave.py with puppet, and convince the mac slaves to run it on startup instead of 'buildbot start'.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

OK, this is mostly written and ready to finish testing in staging.  I need to verify:

 - adding /tools/buildbot on talos slaves
 - proper startup on darwin{9,10} {test,build}
 - still works on linux

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m616350-tools-r1.patch

This removes all of the buildslave-startup stuff from tools; see the next patch for the reason.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m616350-puppet-manifests-r1.patch

Add a 'buildslave' module with a 'buildslave::startup' class.

For the linux systems, this does not change the on-client behavior (although I verified that the old behavior still functions).  It just moves the functionality into a single repository, and encapsulates it nicely under one directory (modules/buildslave).  Note that all of the files are included here - they're short text files with no passwords, so this won't hurt.

As for the Mac startup, this is a significant change from the old puppeteering.  First, it uses the same plist on every system, building it from a template.  This means that all runslave.py's will be run with a cwd of /Users/cltbld, which isn't a problem (runslave.py figures out the proper basedir), and that CVS_RSH is set for every slave - I don't expect that to be a problem, either.  The different PYTHONPATHs are handled with a puppet variable.

Second, there's some trickiness required when the plist changes.  Because puppet runs in parallel to the cltbld login process, if we change the plist, we don't know whether it's already been parsed by cltbld's launchd instance.  The simplest solution is used here - just reboot the machine if any plists change.  This occurs before the puppet.finished sentinel is written, so there's no danger of starting and then aborting a build.

I think there are better ways of doing this - particularly if we could hold off on the automatic login until puppet is finished.  But that's another bug (bug 620530).

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 498891 [details] [diff] [review]
m616350-puppet-manifests-r1.patch

You shouldn't need to add the "is_talos" variable, that info is already set-up by base nodes as "$slaveType", which can be "build", "test", "stage", or "master" (the latter two don't run slaves, so I don't think you care about them).

r=me with that changed.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m616350-puppet-manifests-r2.patch

* Remove $is_talos
* Add/update MOZILLA DEPLOYMENT NOTES comments
* remove unnecessary README.txt

As discussed - can I get a *stamp*?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

This is soaking in staging over the holiday weekend, along with bug 621195.  Things seem to be functioning normally so far.

I'll land this on Tuesday.  There's no downtime required, because at worst it means that slaves don't start up, pushing up wait times, but no oranges.  I'm as confident as I can be that slaves *will* start up.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m616350-puppet-manifests-r3.patch

Ben identified substitution of PATHs for PYTHONPATH in some of the plists.  This is verified to fix these problems in staging.  Now we have (representing no change from the old plists):

darwin9 test:
    <key>PATH</key>
    <string>/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin</string>
    <key>PYTHONPATH</key>
    <string>/Library/Python/2.5/site-packages</string>

darwin9 build:
    <key>PATH</key>
    <string>/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin</string>

darwin10 test:
    <key>PATH</key>
    <string>/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin</string>
    <key>PYTHONPATH</key>
    <string>/Library/Python/2.5/site-packages</string>

darwin10 build:
    <key>PATH</key>
    <string>/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin</string>
    <key>PYTHONPATH</key>
    <string>/tools/buildbot/lib/python2.6/site-packages:/tools/twisted/lib/python2.6/site-packages/:/tools/twisted-core/lib/python2.6/site-packages:/tools/zope-interface/lib/python2.6/site-packages</string>

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 499568 [details] [diff] [review]
m616350-puppet-manifests-r3.patch

e7ae28b6bc53, updated on all masters