720027 - Do all active development branch builds on Rev5 machines running Mac OS X 10.7 Lion

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

We want to replace our Macmini2,1 (rev2) build machines with newer, faster hardware.  This hardware only runs Mac OS X 10.7, so we need a new builder image to run our builds on Mac OS X 10.7.

Because of the timelines we are working with, the plan is currently to use our Puppet 0.24.8 based puppet infrastructure.

Open questions:
1) will we build ESR on rev2/10.6 hardware?
2) will we continue to support 10.5 in Firefox > 13?
  -if not, we should continue to build FF < 13 on rev2 hardware, once 12 migrates to aurora, there shouldn't be much capacity needed for rev2 hardware.
3) Which version of Xcode should we install?
  -4.1 works with 10.5 compatibility hacks
  -4.2 has much better clang, non-issue if we use open-source clang

Comment 1

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I think the only question I have something to add is 3.

We should probably stay with xcode 4.1 for now. I am probably the most excited about switching to clang, but lets do a move at a time. Vanilla gcc-4.2 is gone from xcode 4.2.

We are testing a build done with xcode 4.1 on a 10.7 host in
https://bugzilla.mozilla.org/show_bug.cgi?id=720377

I am not sure when ESR will branch, but I am trying to get the necessary patches ported to aurora in https://bugzilla.mozilla.org/show_bug.cgi?id=719499.

Comment 2

[[e-mail] (:bhearsum)]

The ESR is branching from Firefox 10, which is in Beta right now and ships next week.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #1)
> I think the only question I have something to add is 3.
> 
> We should probably stay with xcode 4.1 for now. I am probably the most
> excited about switching to clang, but lets do a move at a time. Vanilla
> gcc-4.2 is gone from xcode 4.2.

Rafael, please confirm that these statements are true and accurate:

1) xcode 4.1 is a requirement for building 10.5 compatible binaries on 10.7 and does not have an apple version of clang that works for us
2) xcode 4.2 is a requirement for building with a working apple version of clang and is not compatible with building 10.5 compatible binaries on 10.7
3) open source clang releases can use xcode 4.1 or 4.2 as functionally equivalent

The above reflects my understanding, so I'd like to make sure I'm on the same page.


(In reply to Ben Hearsum [:bhearsum] from comment #2)
> The ESR is branching from Firefox 10, which is in Beta right now and ships
> next week.

If that's the case, I think the safest option for ESR is to maintain Rev2 machines in a segregated pool.  Changing the OS and SDK that builds use one week before shipping seems overly risky, as does changing them midway through the releases that are supposed to be stable.

Comment 4

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

> 1) xcode 4.1 is a requirement for building 10.5 compatible binaries on 10.7
> and does not have an apple version of clang that works for us

Correct. I don't remember what was missing from the clang in 4.1, but I do remember reports of the build failing.

> 2) xcode 4.2 is a requirement for building with a working apple version of
> clang and is not compatible with building 10.5 compatible binaries on 10.7

Not sure it is not compatible with building 10.5 binaries. I think it should be, but I have not tested that.

> 3) open source clang releases can use xcode 4.1 or 4.2 as functionally
> equivalent

Yes, we would be using only the SDK included in xcode.

> The above reflects my understanding, so I'd like to make sure I'm on the
> same page.
>

Fairly close at least.

Comment 5

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Ah, and remember with 4.2 we would have to use clang (the builtin one or the open source one).

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

This bug should be used as bug to track the overall project.  Assigning myself to get it out of bug queries

Comment 7

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Adding 721603 since spinning the loop did fix the problem found in 720377.

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: move changes back to plain macosx mozconfigs

I'd like to have this patch ready to land when the buildbot-configs changes land so that the macosx directories are the definitive configuration source.

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbot-configs v2

FF14 is the first release that will be built at any point in its life on new mac builders.
Basic structure of the patch is:
1) convert old 'macosx' to 'macosx-legacy' everywhere.  This allows us to reuse 'macosx' going forward and is part of a long-overdue mac builder name cleanup
2) leave macosx64 as is but only use it on FF13 and older
3) create macosx32-debug for 32bit debug tests
4) use 'macosx' for 32/64 universal builds and 64bit debug
5) make mozilla-central, try, projects and project branches build on 10.7 machines

I've verified that the correct tests are triggered to match the coverage we currently have.  For mozilla-tests/config.py, a new set of global platform list (PRE14_*) is created to represent platforms needed on branches running firefox before version 14.

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbotcustom v2

l10nverify is still done on darwin10 slaves, should we move this job to macosx as a follow up?  We could have the SLAVES key that is used specified in releaseConfig or we could just always do l10n verify on the new 10.7 slaves.

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: tools v1

this is needed to have the macosx test masters create the macosx32 builders.

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to John Ford [:jhford] from comment #11)
> Created attachment 611055 [details] [diff] [review]
> tools v1
> 
> this is needed to have the macosx test masters create the macosx32 builders.

Rail, I am not sure if there is anything else in tools that needs to be updated, mind verifying that there aren't other things in tools that use the buildbot platform names?

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: update trychooser website

This changes the try chooser to reflect the rename of the main mac build platform.  I've also added b2g, which was landed a little while ago

Comment 14

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 611058 [details] [diff] [review]
update trychooser website

thanks for keeping this up to date

Comment 15

[Rail Aliiev [:rail]]

(In reply to John Ford [:jhford] from comment #10)
> Created attachment 611038 [details] [diff] [review]
> buildbotcustom v2
> 
> l10nverify is still done on darwin10 slaves, should we move this job to
> macosx as a follow up?  We could have the SLAVES key that is used specified
> in releaseConfig or we could just always do l10n verify on the new 10.7
> slaves.

AFAIK, these builders can be split by platform. 

We can't run all of them on linux slaves because we mount dmg using hdiutil, which is avalabale on mac builders. We can run linux and windows builders on linux slaves (requires 7zip to be installed on linux slaves).

There is nothing 10.6-specific in the scripts, so, I think, it's safe to move them to 10.7. Just make sure that 10.7 build slaves have 7zip installed.

Comment 16

[Rail Aliiev [:rail]]

(In reply to John Ford [:jhford] from comment #12)
> (In reply to John Ford [:jhford] from comment #11)
> > Created attachment 611055 [details] [diff] [review]
> > tools v1
> > 
> > this is needed to have the macosx test masters create the macosx32 builders.
> 
> Rail, I am not sure if there is anything else in tools that needs to be
> updated, mind verifying that there aren't other things in tools that use the
> buildbot platform names?

I don't mind to ran a couple of staging releases, once we've done with reviews.

BTW, have you ever thought about leaving platform names as is and changing slaves lists so you can move them across branches?

Something like

SLAVES_10_6 = [...]
SLAVES_10_7 = [...]
...
branches['mozillla-central']['macosx64']['slaves'] = SLAVES_10_7
...
branches['mozillla-beta']['macosx64']['slaves'] = SLAVES_10_6

For me it looks like an easier way to go, but I'm not sure if there are some other stoppers, which require separate platform names.

I know, that changing platform names is soooo painful, so I'd be glad if we can discuss other options as well.

Comment 17

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Rail Aliiev [:rail] from comment #15)
> There is nothing 10.6-specific in the scripts, so, I think, it's safe to
> move them to 10.7. Just make sure that 10.7 build slaves have 7zip installed.

7z, 7za and 7zr are all installed and on the default path.

Comment 18

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Rail Aliiev [:rail] from comment #16)
> I don't mind to ran a couple of staging releases, once we've done with
> reviews.

That'd be great!

Comment 19

[Chris Cooper [:coop] (he/him)]

Comment on attachment 611038 [details] [diff] [review]
buildbotcustom v2

Review of attachment 611038 [details] [diff] [review]:
-----------------------------------------------------------------

My overwhelming concern here is continuity for existing platforms in things like, e.g. graphserver. Is messing with what qualifies as "macosx" going to screw up our reporting?

I would be much more confident in these changes if the machines were already running on dev-master01. Is there a reason you haven't set the rev5s machines up to run there? Couldn't QA review a build from anywhere? If this patch works in staging, I will gladly flip my review bit to +.

::: process/factory.py
@@ +1404,4 @@
>            warnOnFailure=True,
>            haltOnFailure=True
>            ))
> +        if self.platform in ('macosx', 'macosx64', 'linux', 'linux64', 'macosx-legacy'):

"macosx32" meant to be in this list too? Easier to use self.platform.startswith(('linux','macosx')) in that case?

::: process/release.py
@@ +914,5 @@
>  
> +            if 'macosx-legacy' in branchConfig['platforms']:
> +                slaves = branchConfig['platforms']['macosx-legacy']['slaves']
> +            else:
> +                # Not sure if this would work on 10.7 build slaves

Should we check this first? Would dump-masters in staging tell us?

Comment 20

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Chris Cooper [:coop] from comment #19)
> Comment on attachment 611038 [details] [diff] [review]
> buildbotcustom v2
> 
> Review of attachment 611038 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> My overwhelming concern here is continuity for existing platforms in things
> like, e.g. graphserver. Is messing with what qualifies as "macosx" going to
> screw up our reporting?

I too share that concern, but everything I've checked so far seems to be using builder names instead of config.py dictionary key values.

Graphserver is reporting correctly using builder names, like "OS_X_10.7_32-bit_mozilla-central_leak_test", "OS_X_10.7_64-bit_mozilla-inbound_leak_test" and "OS_X_10.7_mozilla-inbound".  I am not sure where else we're going to have this problem.  The most likely reporting issues caused by this would be showing lion as leopard.  

> I would be much more confident in these changes if the machines were already
> running on dev-master01. Is there a reason you haven't set the rev5s
> machines up to run there? Couldn't QA review a build from anywhere? If this
> patch works in staging, I will gladly flip my review bit to +.

These builds have been running in staging for >1 month with the r5-mini machines and the entire time I've had the 80 production machines.  There have been no configuration changes made between the r5-mini* machines and the bld-lion-r5-* machines.  Updates have been working very well for me.  

http://dev-master01.build.scl1.mozilla.com:8050/one_line_per_build

Earlier in the project, Anthony from QA took a build from staging and did the non-update portions of the beta tests.  We'll definitely make another build available before we deploy this to production, I'm holding off on having r+'d patches in case we need to make any large changes.  The portion of QA testing that needs to be done post-landing is around the diverted updates.  Nightly updates will be diverted when this lands so that we have a chance to verify that updates are working as expected.  Even though I've been updating perfectly in staging, I want to make sure that differences between prod and staging are tested for.

Because of a change to dev-stage's tools checkout, unit tests have been trying to download from production urls leading to failures.  Older tests work and the failing tests do show which tests are being triggered.

> ::: process/factory.py
> @@ +1404,4 @@
> >            warnOnFailure=True,
> >            haltOnFailure=True
> >            ))
> > +        if self.platform in ('macosx', 'macosx64', 'linux', 'linux64', 'macosx-legacy'):
> 
> "macosx32" meant to be in this list too? Easier to use
> self.platform.startswith(('linux','macosx')) in that case?

Good point, I'll do that.

> ::: process/release.py
> @@ +914,5 @@
> >  
> > +            if 'macosx-legacy' in branchConfig['platforms']:
> > +                slaves = branchConfig['platforms']['macosx-legacy']['slaves']
> > +            else:
> > +                # Not sure if this would work on 10.7 build slaves
> 
> Should we check this first? Would dump-masters in staging tell us?

No, dump-masters wouldn't tell us, we need to test the scripts on an actual slave.  The comment refers to us needing to pick a pool of slaves to run l10n verify on.  Since we aren't going to have many macosx-legacy or macosx64 slaves, it makes sense to run all of the l10n verify jobs on the new pool of machines.  With Rail's input in comment 15, it sounds like running l10n verify on the rev5 machines is a fine outcome.

Comment 21

[Armen [:armenzg]]

Comment on attachment 611038 [details] [diff] [review]
buildbotcustom v2

Review of attachment 611038 [details] [diff] [review]:
-----------------------------------------------------------------

::: misc.py
@@ +1382,4 @@
>                          clobberTime=clobberTime,
>                          signingServers=secrets.get(pf.get('nightly_signing_servers')),
>                          baseMirrorUrls=config.get('base_mirror_urls'),
> +                        extraConfigureArgs=config.get('l10n_extra_configure_args', []) + pf.get('l10n_extra_configure_args', []),

I am confused. what is this for?

@@ +3228,4 @@
>                                                         pgo_builders, merge_tests))
>  
>                          for suites_name, suites in branch_config['platforms'][platform][slave_platform][unittest_suites]:
> +                            print '%s - %s - %s - %s ' % (branch, platform, test_type, suites_name)

Do you need this print statement in here?

Comment 22

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #21)
> Comment on attachment 611038 [details] [diff] [review]
> buildbotcustom v2
> 
> Review of attachment 611038 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: misc.py
> @@ +1382,4 @@
> >                          clobberTime=clobberTime,
> >                          signingServers=secrets.get(pf.get('nightly_signing_servers')),
> >                          baseMirrorUrls=config.get('base_mirror_urls'),
> > +                        extraConfigureArgs=config.get('l10n_extra_configure_args', []) + pf.get('l10n_extra_configure_args', []),
> 
> I am confused. what is this for?

This is needed to set the --target for configure to x86_64.  Without being able to set this configure option, the builds fail because of a mismatch between the auto-detected targets

> @@ +3228,4 @@
> >                                                         pgo_builders, merge_tests))
> >  
> >                          for suites_name, suites in branch_config['platforms'][platform][slave_platform][unittest_suites]:
> > +                            print '%s - %s - %s - %s ' % (branch, platform, test_type, suites_name)
> 
> Do you need this print statement in here?

Nope, I thought I removed all of my prints, but I must have missed that one

Comment 23

[Armen [:armenzg]]

Comment on attachment 611036 [details] [diff] [review]
buildbot-configs v2

Review of attachment 611036 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +1265,5 @@
> +# platforms to be concerned about.  Once Firefox 14 is on Aurora, we can add mozilla-aurora
> +# to the branch tuple above and delete this section
> +for branch in ('mozilla-aurora', ):
> +    for pf in PLATFORMS:
> +        if pf in ('macosx', 'macosx32') or 'android' in pf:

Could you please file a bug to follow up on this hack? or make it more obvious so it is not missed on uplift day?

@@ +1351,5 @@
> +    else:
> +        if p.has_key('macosx-legacy'):
> +            del p['macosx-legacy']
> +        if p.has_key('macosx64'):
> +            del p['macosx64']

We probably need a disclaimer for this hack as FF14 goes into these branches.

::: mozilla/config.py
@@ +1120,5 @@
> +            'macosx64': {},
> +            'android': {},
> +            'linux-debug': {},
> +            'android-debug': {},
> +        }

I guess we need to follow up on these as FF14 goes through them.

I think we should be adding "XXX" in them and a note, no?

@@ +1727,5 @@
>      BRANCHES['try']['platforms'][platform]['stage_product'] = 'firefox'
>  
> +for i in ('macosx64', 'macosx64-debug', 'macosx-legacy', 'macosx-legacy-debug'):
> +    if BRANCHES['try']['platforms'].has_key(i):
> +        del BRANCHES['try']['platforms'][i]

Would this be a permanent change?

::: mozilla/project_branches.py
@@ +20,4 @@
>              'paint': 0,
>          },
>          'add_test_suites': [
> +            # TODO: do I need something for lion builds here?

You are good in here.

::: mozilla/staging_config.py
@@ +1,3 @@
>  MAC_SNOW_MINIS = ['moz2-darwin10-slave%02i' % x for x in range(1,30) + range(40,57)]
>  MAC_MINIS      = ['moz2-darwin9-slave%02i' % x for x in range(1,73) if x not in (4,5,20,40,59)]
> +MAC_LION_MINIS = ['r5-mini-%03i' % x for x in range(1,7)] + ['bld-lion-r5-%03d' % x for x in range(1,81)]

Should we just rename the "r5-mini" to be "bld-lion-r5"?

@@ +39,5 @@
>  
>  
>  GLOBAL_VARS = {
>      'staging': True,
> +    'config_repo_path': 'users/jford_mozilla.com/buildbot-configs',

Do you want this change to be permanent? (Same with next 2 changes)

Comment 24

[Armen [:armenzg]]

Comment on attachment 611038 [details] [diff] [review]
buildbotcustom v2

Trying to put back coop's r-.

Comment 25

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #23)
> Could you please file a bug to follow up on this hack? or make it more
> obvious so it is not missed on uplift day?

I'll file that bug once we decide this is the approach we'll take

> @@ +1351,5 @@
> > +    else:
> > +        if p.has_key('macosx-legacy'):
> > +            del p['macosx-legacy']
> > +        if p.has_key('macosx64'):
> > +            del p['macosx64']
> 
> We probably need a disclaimer for this hack as FF14 goes into these branches.

I'll add more documentation in the file about when things need to happen.  Like above, filing a bug is probably worthwhile

> I guess we need to follow up on these as FF14 goes through them.
> 
> I think we should be adding "XXX" in them and a note, no?

I'll add those
 
> @@ +1727,5 @@
> >      BRANCHES['try']['platforms'][platform]['stage_product'] = 'firefox'
> >  
> > +for i in ('macosx64', 'macosx64-debug', 'macosx-legacy', 'macosx-legacy-debug'):
> > +    if BRANCHES['try']['platforms'].has_key(i):
> > +        del BRANCHES['try']['platforms'][i]
> 
> Would this be a permanent change?

Yes

> ::: mozilla/project_branches.py
> @@ +20,4 @@
> >              'paint': 0,
> >          },
> >          'add_test_suites': [
> > +            # TODO: do I need something for lion builds here?
> 
> You are good in here.

Great, i'll remove the TODO

> Should we just rename the "r5-mini" to be "bld-lion-r5"?

Nope, those are the 6 staging slaves.  We can file a bug to have their hostnames converted to bld-lion-r5-XXX names at some point.

> @@ +39,5 @@
> >  
> >  
> >  GLOBAL_VARS = {
> >      'staging': True,
> > +    'config_repo_path': 'users/jford_mozilla.com/buildbot-configs',
> 
> Do you want this change to be permanent? (Same with next 2 changes)

Nope, i thought I reverted all of my staging changes, but I think my |hg meld| call wrote to a tmp file instead of the real files.  I've since figured out a way to avoid this problem.

Comment 26

[[e-mail] (:bhearsum)]

There's some stuff in tools that needs adjustment if "macosx64" is getting renamed to "macosx" again. I think these are the only things, but it's hard to be sure without a staging release:
- http://mxr.mozilla.org/build-central/source/tools/release/update-verify-bump.pl#242
- http://mxr.mozilla.org/build-central/source/tools/lib/python/release/platforms.py#23

There may also be some things in buildbotcustom.

Comment 27

[Chris Cooper [:coop] (he/him)]

Comment on attachment 611036 [details] [diff] [review]
buildbot-configs v2

Review of attachment 611036 [details] [diff] [review]:
-----------------------------------------------------------------

While there's nothing strictly wrong with the patch, I question the choice to link the renaming of platforms to this time critical deployment of rev5 machines. 

If you're going to go through with the renaming, I'm going to want to see the whole shebang in staging - nightlies, releases, posting to graphs-stage - before we can go forward. We can't airlift this onto m-c and expect developer patience while we sort out issues like we had to with the previous platform change (PGO).

::: mozilla/config.py
@@ +1727,5 @@
>      BRANCHES['try']['platforms'][platform]['stage_product'] = 'firefox'
>  
> +for i in ('macosx64', 'macosx64-debug', 'macosx-legacy', 'macosx-legacy-debug'):
> +    if BRANCHES['try']['platforms'].has_key(i):
> +        del BRANCHES['try']['platforms'][i]

If we go with jhford's proposed naming changes, only macosx and macosx32 will exist, so yes.

Comment 28

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Chris Cooper [:coop] from comment #27)
> While there's nothing strictly wrong with the patch, I question the choice
> to link the renaming of platforms to this time critical deployment of rev5
> machines. 

I think that it's worth noting that these patches were written about 1.5-2 months ago when there was less of a time pressure.  The project was put on hold for a month because merging become a burden and b2g needed to be done.

I still think this is the right approach long term, but I am going to write up a patch tonight that uses Rail's suggestion from comment 16.

Comment 29

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: alternate patch -- buildbot-configs v1

This is the alternate approach.  I am not sure if I'll need to fix the l10n bits that were failing on 10.7 before or whether they'll work as is.  If I don't need to worry about fixing the l10n issue, there wouldn't be a buildbotcustom component to this patch.

Comment 30

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: alternate patch -- buildbot-configs v2

Tested this.  Nightly works, I got an update.  by changing the lion_branches list, we can enable this selectively by branch.  Aside from changing the builder name to reflect the actual platform, these builds will report and trigger exactly as our current OS X 10.6 builds do.

The issue I was having before regarding L10N using an incorrect autoconf target does not show up with this patch.

This is definately the simpler patch to take.  The actual builds produced are identical to the testing I've done up until now, the difference between this patch and the other is triggering and reporting.

Comment 31

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: alternate patch -- buildbotcustom v2

This patch is needed to be able to tell buildbot that sometimes we want to look at a different SLAVES key for the list of l10n slaves.  Without this, l10n repacks for 10.7 branches would happen on 10.6 machines.

Comment 32

[Rail Aliiev [:rail]]

Can't stop myself from posting this! :D
   _
 ( ((
  \ =\
 __\_ `-\ 
(____))(  \---- AWESOME JOB!
(____)) _  
(____))
(____))____/----

BTW, LION looks like l10n :)

Comment 33

[John Ford [:jhford] CET/CEST Berlin Time]

the alternate patches are using incorrect builder names for debug builders.  The following interdiff fixes that:

diff -u b/mozilla/config.py b/mozilla/config.py
--- b/mozilla/config.py
+++ b/mozilla/config.py
@@ -1702,8 +1702,8 @@
 # This is a mapping of platform key to lion specific base_name formatters
 lion_names = {
     'macosx64': 'OS X 10.7 %(branch)s',
-    'macosx64-debug': 'OS X 10.7 64-bit %(branch)s',
-    'macosx-debug': 'OS X 10.7 32-bit %(branch)s',
+    'macosx64-debug': 'OS X 10.7 64-bit %(branch)s leak test',
+    'macosx-debug': 'OS X 10.7 32-bit %(branch)s leak test',
 }
 for b in BRANCHES.keys():
     if b in lion_branches:

Comment 34

[Chris Cooper [:coop] (he/him)]

Comment on attachment 612101 [details] [diff] [review]
alternate patch -- buildbot-configs v2

Review of attachment 612101 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/production_config.py
@@ +1,1 @@
> +MAC_LION_MINIS = ['bld-lion-%03d' % x for x in range(41,81)]

Is there a reason why we don't include the r5-mini-* slaves here just for completeness?

Comment 35

[Chris Cooper [:coop] (he/him)]

(In reply to John Ford [:jhford] from comment #30) 
> This is definately the simpler patch to take.  The actual builds produced
> are identical to the testing I've done up until now, the difference between
> this patch and the other is triggering and reporting.

FWIW, I do want to see the cleanup work done with the Mac platforms, I'm just adverse to doing it _now_. Once mozilla-release is building on lion, I'd be much happier to review those patches.

Comment 36

[Shane Bundy]

https://hg.mozilla.org/mozilla-central/rev/b687cabf75a1

Comment 37

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: update logs from 10.5,6,7

With updates diverted, I was able to test that each of 10.5, 10.6 and 10.7 were able to update themselves from a build done on 10.6 to a build done on 10.7 and run the browser after the fact.  As you'll see, I used to use the arch command to run Firefox on 10.5.  This is a known issue and expected.  I verified that double clicking the resulting app bundle started firefox correctly.

As far as I know, this means we're clear to enable lion builders on all of mozilla-central, try and project branches.

Comment 38

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: finish the deployment

This patch finishes up the deployment by setting all development branches (i.e. all but m-a, m-b, m-r, m-192, esr10) to build Mac on 10.7.  This also reverts the update diversion from bug 743789

Comment 40

[John Ford [:jhford] CET/CEST Berlin Time]

This was deployed today.  New issues should be filed in new bugs.

Comment 41

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/11c52312cbce
https://hg.mozilla.org/mozilla-central/rev/71e7824b9fe6

Comment 42

[Steven Michaud [:smichaud] (Retired)]

Bug 753248, which was triggered by this bug (by starting to do mozilla-central and aurora nightly builds on OS X 10.7), is a topcrasher.  So we must not start doing any release builds on OS X 10.7 until we've found a fix or workaround for bug 753248.