429430 - Redesign mac refimage so no additional manual setup needed

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

To setup a new mac, we start by putting an image of an existing refimage mac, which gives us a basic set of OS+toolchain installs. However, we then need to follow these manual instructions:

http://wiki.mozilla.org/ReferencePlatforms
or
http://wiki.mozilla.org/BuildbotTestfarm

to manually install the rest of the toolchain softwarem not included in the
refimage VM. All this takes a long time, and can be tricky. As we setup more
and more machines, this overhead is becoming a problem.

Lets eliminate these manual post-install steps

1) For software that needs configuring, write a script, run as root, which can
be run on first boot, and which fills in details like machine/host name, smtp
configuration, etc.

2) For software that changes rapidly, whatever version is on the refimage is
likely to be out of date anyway by the time we get to clone new VMs. Instead
lets design the refimage to contain enough to get started, and include in
refimage a script that checks for updates on first launch, and refresh forward
to the latest available at that time. One way of doing this would be to pull
specific tag versions of buildbot from CVS, for example, but there are probably
other ways to do this. Or pull a tagged version of a text file, which contains
URLs for downloading using WGET.

3) After VM is created, with bang-up-to-date versions of software, it will have
to keep checking and refreshing forward, or else this new VM will drift out of
date. We need a periodic check to verify VM is still running the right versions
of toolchain, and refresh forward if not. This would allow us to know that all
slaves are always in sync with each other. Open question: how frequently should
we be rechecking & refreshing? Once a day seems reasonable start, but thats
just a swag.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Putting back in the pool after triage.

Comment 2

[[e-mail] (:bhearsum)]

A lot of the scripts being used for this are sitting in bug 429427, because they're shared with Linux.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: buildbot-tac plist launcher

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 410826 [details]
buildbot-tac plist launcher

RCS file: /mofo/puppet-files/darwin9/buildbot-tac.generator.com.plist,v
done
Checking in buildbot-tac.generator.com.plist;
/mofo/puppet-files/darwin9/buildbot-tac.generator.com.plist,v  <--  buildbot-tac.generator.com.plist
initial revision: 1.1
done

Comment 5

[Chris Cooper [:coop] (he/him)]

Assigning to Ben since he seems to be doing active work here now that bug 429427 is fixed.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: add mac ref mini to puppet

Note that as per bug 524108 we're using the staging keys, not production ones.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 411712 [details] [diff] [review]
add mac ref mini to puppet

changeset:   76:05a3ab223a11

Comment 8

[[e-mail] (:bhearsum)]

Attachment: ignore mac ref platform during tac generation

Comment 9

[[e-mail] (:bhearsum)]

Attachment: add try, talos support to buildbot-tac init script

This patch adds some more intelligent host detection to the buildbot-tac init script for the purpose of setting the right password for different types of machines.

No Windows support here because I don't expect we'll be using this script on Windows.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: better try, talos support in buildbot-tac.py

Like the init script patch, this patch adds better support for Talos and Try machine detection: setting the right master and port based on the hostname. Also fixes a typo bug (MissingOptionsException != MissingOptionsError).

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 411753 [details] [diff] [review]
ignore mac ref platform during tac generation

Checking in buildbot-tac;
/mofo/puppet-files/shared/buildbot-tac,v  <--  buildbot-tac
new revision: 1.2; previous revision: 1.1
done

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 411789 [details] [diff] [review]
better try, talos support in buildbot-tac.py

>diff --git a/buildbot-helpers/buildbot-tac.py b/buildbot-helpers/buildbot-tac.py
>+    elif 'talos' in slavename or 'qm-p' in slavename:
>         buildmaster_host = TALOS_BUILDMASTER
>-        if 'linux' in slavename:
>+        if 'qm-p' in slavename:
>+            d['port'] = 9011

9011 is talos try ? Matching on '-try' might be more obvious.

Comment 13

[[e-mail] (:bhearsum)]

(In reply to comment #12)
> (From update of attachment 411789 [details] [diff] [review])
> >diff --git a/buildbot-helpers/buildbot-tac.py b/buildbot-helpers/buildbot-tac.py
> >+    elif 'talos' in slavename or 'qm-p' in slavename:
> >         buildmaster_host = TALOS_BUILDMASTER
> >-        if 'linux' in slavename:
> >+        if 'qm-p' in slavename:
> >+            d['port'] = 9011
> 
> 9011 is talos try ?

Yup.

> Matching on '-try' might be more obvious.

I'll fix this on check-in.

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 411785 [details] [diff] [review]
add try, talos support to buildbot-tac init script

Checking in buildbot-tac;
/mofo/puppet-files/shared/buildbot-tac,v  <--  buildbot-tac
new revision: 1.3; previous revision: 1.2
done

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 411789 [details] [diff] [review]
better try, talos support in buildbot-tac.py

changeset:   413:b74e43b2a0c3

Comment 16

[[e-mail] (:bhearsum)]

Attachment: update build tools checkouts

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 411962 [details] [diff] [review]
update build tools checkouts

I had to check in a bustage fix os the real, buildtools_version is: 480fc8c00479

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 411962 [details] [diff] [review]
update build tools checkouts

changeset:   78:b5b502462d51
RCS file: /mofo/puppet-files/centos5/dist/build-tools-0b149c0ad18d.tgz,v
done
Checking in build-tools-0b149c0ad18d.tgz;
/mofo/puppet-files/centos5/dist/build-tools-0b149c0ad18d.tgz,v  <--  build-tools-0b149c0ad18d.tgz
initial revision: 1.1

done

Checking in build-tools-0b149c0ad18d.tgz;
/mofo/puppet-files/darwin9/devtools/build-tools-0b149c0ad18d.tgz,v  <--  build-tools-0b149c0ad18d.tgz
initial revision: 1.1
done

Comment 19

[[e-mail] (:bhearsum)]

Attachment: disable mac key management

I originally implemented this when we first brought Mac puppet online without thinking hard enough about the implications. Due to this breaking 0 -> staging (requires manual intervention, turning off Puppet), I think we should just turn it off for now.

Comment 20

[[e-mail] (:bhearsum)]

Comment on attachment 414315 [details] [diff] [review]
disable mac key management

changeset:   84:acd01e2a5725

Comment 21

[[e-mail] (:bhearsum)]

Updated https://wiki.mozilla.org/ReferencePlatforms/Mac-10.5#Manual_setup_after_cloning, we're all done here!