748628 - Switch Thunderbird builds to use OS X 10.7 build machines

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John Hopkins (:jhopkins)]

Like Firefox bug 720027, we need to start using the OS X 10.7 build machines.

Does this need to be done before or after the colo move?  Setting to blocking 698843 for now to give visibility.

Comment 1

[Chris Cooper [:coop] (he/him)]

I'm running a test build by hand on r5-mini-005 to see how far away we are. The only thing I've needed to change so far is the path the ccache in the mozconfig.

NOTE: this is not me signing up for this bug, but this should let us know whether more invasive changes are required.

Comment 2

[Chris Cooper [:coop] (he/him)]

Hit the following error during the mozmill install:

/usr/bin/python2.7 ../../../mozilla/_tests/mozmill/resources/installmozmill.py ../../../mozilla/_tests/mozmill/../mozmill-virtualenv
...
distutils.errors.DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.5" but "10.7" during configure

Full log is here: http://people.mozilla.org/~coop/logs/tb-lion-build.log.gz

Comment 3

[Mark Banner (:standard8)]

Based on http://hg.mozilla.org/comm-central/annotate/9b83555adeb2/mail/testsuite-targets.mk#l32

I think we should try prefixing the following line with MACOSX_DEPLOYMENT_TARGET=

http://hg.mozilla.org/comm-central/annotate/9e48cf409982/mail/test/mozmill/Makefile.in#l91

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #1) 
> NOTE: this is not me signing up for this bug, but this should let us know
> whether more invasive changes are required.

Look at me lying and taking this bug.

Comment 5

[Chris Cooper [:coop] (he/him)]

Using a new mozconfig provided by Mark: http://pastebin.mozilla.org/1599056

A new build is in progress. Hit the following syntax error right off the bat, but it doesn't seem to be slowing things down:

/builds/slave/coop/build/.mozconfig: line 98: syntax error: unexpected end of file

Comment 6

[Chris Cooper [:coop] (he/him)]

We get this now during the mozmill install:

distutils.errors.DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.6" but "10.7" during configure

Comment 7

[Chris Cooper [:coop] (he/him)]

Attachment: .mozconfig attempt by jhopkins

Here's a mozconfig that jhopkins tried today. It produced the following output, failing in nspr:

http://people.mozilla.org/~coop/logs/build-tb-10.7_v2.log.gz

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #7) 
> Here's a mozconfig that jhopkins tried today. It produced the following
> output, failing in nspr:

I had a similar problem with NSPR on lion when trying to build spidermonkey. NSPR is braindead about using 64bit on mac. Adding a global --enable-64bit to the mozconfig has gotten me past that and back to the mozmill error. Still plugging away at this.

Comment 9

[Mark Banner (:standard8)]

Comment on attachment 619172 [details]
.mozconfig attempt by jhopkins

Quoting the mozconfig:

ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.7.sdk

#CC=/usr/bin/gcc-4.2
#CXX=/usr/bin/g++-4.2
CC=clang
CXX=clang++


Should definitely be using the 10.6 sdk at the moment, that could be why it failed in NSS.

Also not sure why the switch to clang, as I don't think FF have done that yet, although in theory it should work.

Comment 10

[Mark Banner (:standard8)]

Adding Sid to the cc list in case he's got any ideas about the mozmill issue.

Comment 11

[Mark Banner (:standard8)]

Bug 750652 has the fix for the MozMill/virtualenv issue. With that I've successfully completed a 32 bit debug build on the 10.7 builder using a modified mozconfig.

I'm currently building with updated 64 bit and universal mozconfigs on 10.7 and will get a patch up for those in a bit if everything is ok.

Comment 12

[Mark Banner (:standard8)]

Ok, I've done successful builds of 32 & 64 bit debug and of universal on the 10.7 builder with the patch from bug 750652 and the mozconfig changes I've just posted in bug 750669.

I've not tested them, but I'm pretty confident they will work correctly.

We now need build config to swap comm-central, comm-aurora and try-comm-central to being 10.7 based. I suspect this is just porting attachment 612101 [details] [diff] [review] to the Thunderbird config.py?

Comment 13

[Mark Banner (:standard8)]

Attachment: Proposed fix

I've not really tested this but I believe this should do the necessary switch.

Comment 14

[Chris Cooper [:coop] (he/him)]

Comment on attachment 619908 [details] [diff] [review]
Proposed fix

Review of attachment 619908 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good at first pass. I'll set this up in staging this afternoon to make sure the builders get created as expected.

Comment 15

[Chris Cooper [:coop] (he/him)]

Comment on attachment 619908 [details] [diff] [review]
Proposed fix

Review of attachment 619908 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, modulo the import issue. r+ with that fix

::: mozilla/thunderbird_config.py
@@ +1068,5 @@
>              'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib64',
>          }
>  
> +# Bug 720027 / Bug 748628, do macosx64 builds on Lion slaves where appropriate
> +lion_branches = ['comm-central', 'try-comm-central',] + ACTIVE_PROJECT_BRANCHES[:]

Need to import ACTIVE_PROJECT_BRANCHES from project_branches like in config.py.

Comment 16

[Mark Banner (:standard8)]

Attachment: Proposed fix v2

Sorry, I've just realised, PROJECT_BRANCHES and ACTIVE_PROJECT_BRANCHES are really Firefox related items and configuration. There's no need for us to touch it here as config.py is doing the necessary changes, for this it would effectively be a no-op.

This patch just differs from the previous by removing the addition of ACTIVE_PROJECT_BRANCHES.

Comment 17

[Mark Banner (:standard8)]

Landed: http://hg.mozilla.org/build/buildbot-configs/rev/3546557799a1

Comment 18

[Rail Aliiev [:rail]]

(In reply to Mark Banner (:standard8) from comment #17)
> Landed: http://hg.mozilla.org/build/buildbot-configs/rev/3546557799a1

It doesn't pass preproduction test-masters check:

INFO  - starting to print log file '/builds/buildbot/preproduction/slave/test-masters/buildbot-configs/test-output/bm09-try1-d8rQTw-checkconfig.log'
INFO  - /builds/buildbot/preproduction/slave/test-masters/sandbox/lib/python2.6/site-packages/twisted/mail/smtp.py:10: DeprecationWarning: the MimeWriter module is deprecated; use the email package instead
INFO  -   import MimeWriter, tempfile, rfc822
INFO  - Traceback (most recent call last):
INFO  -   File "/builds/buildbot/preproduction/slave/test-masters/sandbox/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/scripts/runner.py", line 1039, in doCheckConfig
INFO  -     ConfigLoader(configFileName=configFileName)
INFO  -   File "/builds/buildbot/preproduction/slave/test-masters/sandbox/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
INFO  -     self.loadConfig(configFile, check_synchronously_only=True)
INFO  -   File "/builds/buildbot/preproduction/slave/test-masters/sandbox/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/master.py", line 784, in loadConfig
INFO  -     % (b['name'], n))
INFO  - ValueError: builder OS X 10.7 32-bit try-comm-central build uses undefined slave bld-lion-r5-041
INFO  - finished printing log file '/builds/buildbot/preproduction/slave/test-masters/buildbot-configs/test-output/bm09-try1-d8rQTw-checkconfig.log'
ERROR - TEST-FAIL bm09-try1 failed to run checkconfig

Comment 19

[Mark Banner (:standard8)]

Attachment: Add macosx64-lion to preproduction and staging configs for Thunderbird

This adds the macosx64-lion lines to the thunderbird configs for preproduction and staging. It appears they were already added in production config.

Sorry for missing that, I should have noticed it in attachment 612101 [details] [diff] [review].

Comment 20

[Rail Aliiev [:rail]]

Comment on attachment 620272 [details] [diff] [review]
Add macosx64-lion to preproduction and staging configs for Thunderbird

It still fails:

INFO  - created  "bm09-try1" master, running checkconfig
INFO  - starting to print log file '/tmp/bm09-try1-UZu58s-checkconfig.log'
INFO  - Traceback (most recent call last):
INFO  -   File "/home/rail/work/mozilla/build/buildbot/master/buildbot/scripts/runner.py", line 1039, in doCheckConfig
INFO  -     ConfigLoader(configFileName=configFileName)
INFO  -   File "/home/rail/work/mozilla/build/buildbot/master/buildbot/scripts/checkconfig.py", line 31, in __init__
INFO  -     self.loadConfig(configFile, check_synchronously_only=True)
INFO  -   File "/home/rail/work/mozilla/build/buildbot/master/buildbot/master.py", line 784, in loadConfig
INFO  -     % (b['name'], n))
INFO  - ValueError: builder OS X 10.7 32-bit try-comm-central build uses undefined slave bld-lion-r5-041
INFO  - finished printing log file '/tmp/bm09-try1-UZu58s-checkconfig.log'
ERROR - TEST-FAIL bm09-try1 failed to run checkconfig


You can test the configs yourself. I have a script for that:

===========================================
export PATH=`pwd`/buildbot/master/bin:$PATH
export PYTHONPATH=`pwd`:`pwd`/buildbot/master:`pwd`/tools/lib/python:`pwd`/../twisted/Twisted-10.1.0
export TEMP=/tmp
export PYTHONDONTWRITEBYTECODE=1
cd buildbot-configs && python setup-master.py --test --masters-json ../tools/buildfarm/maintenance/production-masters.json --error-logs
===========================================
with tools, buildbot-configs, builbotcustom and buildbot (from hg.m.o) directories (adjust your path to twisted if it's not system wide installed).

Comment 21

[Mark Banner (:standard8)]

Attachment: Add macosx64-lion to preproduction and staging configs for Thunderbird v2

This one is passing setup_master.py for try server. I'd somehow missed the differentiation between the normal SLAVES and the TRY_SLAVES for the lion builders.

Comment 22

[Rail Aliiev [:rail]]

Comment on attachment 620306 [details] [diff] [review]
Add macosx64-lion to preproduction and staging configs for Thunderbird v2

lgtm

Comment 23

[Mark Banner (:standard8)]

Checked in: http://hg.mozilla.org/build/buildbot-configs/rev/234a2c7fd454

Comment 24

[Mark Banner (:standard8)]

Attachment: Fix some more issues

So I know know what I did wrong, I assumed that attachment 612101 [details] [diff] [review] was what actually landed, which was a bad assumption, this is what landed:

http://hg.mozilla.org/build/buildbot-configs/rev/d34646b57ba1

This fixes up the remaining differences and also ports http://hg.mozilla.org/build/buildbot-configs/rev/e06e7c886393 just in case we ever need it.

Comment 25

[Mark Banner (:standard8)]

Checked in: http://hg.mozilla.org/build/buildbot-configs/rev/f7ebe0f989d8

Comment 26

[Mark Banner (:standard8)]

Attachment: Add the 'TB ' prefix onto the 10.7 builders

We missed this earlier, but we should have the same prefix as we've got on our other builders.

Comment 27

[Mark Banner (:standard8)]

Comment on attachment 620629 [details] [diff] [review]
Add the 'TB ' prefix onto the 10.7 builders

Checked in:

http://hg.mozilla.org/build/buildbot-configs/rev/d81f58768d7a

Comment 28

[Mark Banner (:standard8)]

10.7 builders are now running. The graph server switch is the remaining change, but that's also being tracked against the comm-central switch now, so this bug is good.