773120 - turn off Mac OS X 10.5 test machines for Firefox/Thunderbird 17

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P1)

Description

[Josh Aas]

We are not planning to support Mac OS X 10.5 with Firefox 17. The builds will fail to run on anything less than Mac OS X 10.6. We will need to turn off any Mac OS X 10.5 test machines to avoid bustage.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

How soon are you planning to make build changes dropping 10.5 ? Should we make this change as part of the general merge mechanics (bug 764909 and deps) ?

Comment 2

[Josh Aas]

I am hoping to make the change as soon as Firefox 17 development begins, but I can delay a bit if there is a problem. At a minimum we just need to not show the results from any 10.5 machines - the critical thing here is to avoid the appearance of a busted tree. When the boxes are actually moved or taken offline is less important to me.

Comment 3

[Armen [:armenzg]]

/me sharpens the axe

It sounds that we want to do it on bug 764909.

Comment 4

[Phil Ringnalda (:philor)]

FWIW, "not showing" is dead simple and requires no coordination with anyone other than one person cc'ed on bug sheriffpass, which surely includes you. You open tbpl's "Tree Info" menu, pick "Open tree admin panel" and hide them.

However, whether it's this bug or another one, you better be gated on "Don't update 10.5 users of trunk builds, on any m-c related channel."

Comment 5

[Josh Aas]

(In reply to Phil Ringnalda (:philor) from comment #4)

> However, whether it's this bug or another one, you better be gated on "Don't
> update 10.5 users of trunk builds, on any m-c related channel."

There is already a bug with a patch for that.

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Josh Aas (Mozilla Corporation) from comment #5)
> (In reply to Phil Ringnalda (:philor) from comment #4)
> 
> > However, whether it's this bug or another one, you better be gated on "Don't
> > update 10.5 users of trunk builds, on any m-c related channel."
> 
> There is already a bug with a patch for that.

Looks like we can go ahead with the hiding now that Firefox 17 is the nightly branch.  What's the bug for the updates?

Comment 7

[Armen [:armenzg]]

(In reply to Lukas Blakk [:lsblakk] from comment #6)
> (In reply to Josh Aas (Mozilla Corporation) from comment #5)
> > (In reply to Phil Ringnalda (:philor) from comment #4)
> > 
> > > However, whether it's this bug or another one, you better be gated on "Don't
> > > update 10.5 users of trunk builds, on any m-c related channel."
> > 
> > There is already a bug with a patch for that.
> 
> Looks like we can go ahead with the hiding now that Firefox 17 is the
> nightly branch.  What's the bug for the updates?

bug 772735 is for the updates.

I hid the 10.5.8 tests on mozilla-central. I will attach a patch soon.

Comment 8

[Ed Morley [:emorley]]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #7)
> I hid the 10.5.8 tests on mozilla-central. I will attach a patch soon.

They also need hiding on {inbound,fx-team,ux,ionmonkey,all twigs,.....}, I'll do that now.

Are we not switching off the builds as well?

Comment 9

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #8)
> They also need hiding on {inbound,fx-team,ux,ionmonkey,all twigs,.....},
> I'll do that now.

Hidden on:
(mozilla-central)
mozilla-inbound
fx-team
ionmonkey
accessibility
build-system
jaegermonkey
profiling
services-central
ux
alder
ash
birch
cedar
holly
larch
maple
oak
pine
electrolysis

What are we doing about Try? Keeping a few 10.5 machines so if people explicitly request 10.5 using trychooser's -p, they can test aurora/beta pushes?

Also, the Jetpack tree has 10.5 m-c runs, no idea where they have to be switched off / if they'll be covered in the patch in this bug, so left showing for now: https://tbpl.mozilla.org/?tree=Jetpack

Comment 10

[Ed Morley [:emorley]]

Attachment: Disable OSX 10.5 on trunk trees

Used this as an excuse to get a bit more familiar with the buildbot configs, having not touched them much before. 

Was this the right approach, to disable on trunk but keep the 10.5 builds going on {try,aurora,beta,release,esr}?

Might be complete rubbish, but thought worth a try, even if it only ends up being a WIP to give whomever does it properly a head start :-)

Comment 11

[Armen [:armenzg]]

Attachment: disable 32-bit debug builds (64-bit will still run)

Comment 12

[Chris Cooper [:coop] (he/him)]

Comment on attachment 645007 [details] [diff] [review]
disable 32-bit debug builds (64-bit will still run)

Review of attachment 645007 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ -1214,5 @@
>  BRANCHES['mozilla-release']['repo_path'] = 'releases/mozilla-release'
>  BRANCHES['mozilla-release']['update_channel'] = 'release'
>  BRANCHES['mozilla-release']['l10n_repo_path'] = 'releases/l10n/mozilla-release'
>  BRANCHES['mozilla-release']['enable_weekly_bundle'] = True
>  BRANCHES['mozilla-release']['start_hour'] = [3]

Does lion_branches need to be pruned in config.py also? https://hg.mozilla.org/build/buildbot-configs/file/68c191f31d39/mozilla/config.py#l1654

If the macosx-debug key is not going to exist any more, we might as well clean up the other occurrences.

Comment 13

[Chris Cooper [:coop] (he/him)]

Comment on attachment 645007 [details] [diff] [review]
disable 32-bit debug builds (64-bit will still run)

Review of attachment 645007 [details] [diff] [review]:
-----------------------------------------------------------------

r+ provided dump_masters output looks like you expect.

Comment 14

[Armen [:armenzg]]

Comment on attachment 643951 [details] [diff] [review]
Disable OSX 10.5 on trunk trees

Hi Ed,
I'm working on this patch (which looks like yours) and I will attach it later today.
FTR this is what I do to work on a patch like this:
* I create a Mac master
./setup-master.py dir_to_work_on bm04-tests1-macosx (if you run with -l you will see that there are 3-4 macosx masters)
* I get a list of the builders that the master has
~/repos/braindump/buildbot-related/builder_list.py dir_to_work_on/master.cfg > old_list
* I do my hacks and I see what the new list is
~/repos/braindump/buildbot-related/builder_list.py dir_to_work_on/master.cfg > new_list

After that I can do diffs between the two.
[1]
[2] http://hg.mozilla.org/build/braindump/filelog/a5011b9d9433/buildbot-related/builder_list.py

Comment 15

[Armen [:armenzg]]

Comment on attachment 645007 [details] [diff] [review]
disable 32-bit debug builds (64-bit will still run)

http://hg.mozilla.org/build/buildbot-configs/rev/f39a4b749d23

Comment 16

[Ed Morley [:emorley]]

Comment on attachment 643951 [details] [diff] [review]
Disable OSX 10.5 on trunk trees

Thank you; I'll remember that for another time, since up until now I had no idea how to test the changes :-)

Comment 17

[Ed Morley [:emorley]]

Please may we have a reconfig for this sometime soon - the burning tests are still causing confusion periodically on various IRC channels.

Comment 18

[Chris Cooper [:coop] (he/him)]

(In reply to Ed Morley [:edmorley] from comment #17)
> Please may we have a reconfig for this sometime soon - the burning tests are
> still causing confusion periodically on various IRC channels.

Can you point me at results that you think should have disappeared? There have been several reconfigs since this landed, so it should already be in production.

Comment 19

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&noignore=1&jobname=10.5
https://tbpl.mozilla.org/?noignore=1&jobname=10.5

Thank you :-)

Comment 21

[Mark Banner (:standard8)]

(In reply to Ed Morley [:edmorley] from comment #19)
> https://tbpl.mozilla.org/?tree=Mozilla-Inbound&noignore=1&jobname=10.5
> https://tbpl.mozilla.org/?noignore=1&jobname=10.5

mozilla-tests/config.py wasn't touched, so I suspect that's the issue.

Comment 22

[Armen [:armenzg]]

I'm still working on the patch.

Comment 23

[Armen [:armenzg]]

Attachment: disable tests only for FF17 based branches except try

I will work now on trying to make it work for Thunderbird or let someone pick it up.

Comment 24

[[e-mail] (:bhearsum)]

Comment on attachment 648135 [details] [diff] [review]
disable tests only for FF17 based branches except try

Review of attachment 648135 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +1164,5 @@
>  BRANCHES['try']['platforms']['android']['enable_debug_unittests'] = True
>  BRANCHES['try']['pgo_strategy'] = 'try'
>  
> +# MERGE day - disable leopard tests for FF17 onwards
> +for branch in ('mozilla-aurora', 'mozilla-beta', 'mozilla-release', 'mozilla-esr10', 'try',):

We need to stop doing this sort of thing. It makes these configs even more difficult to reason about. Instead of this, leopard should be removed from the default list of platforms (https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla-tests/config.py#L75), and added *explicitly* to these branches around here: https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla-tests/config.py#L1098. It will be more verbose, but easier to understand.

Comment 25

[Armen [:armenzg]]

jhokpins, the patch does the job but bhearsum does not like the for loop and prefers to be more verbose to be more easily readable.

This patch does not disable Thunderbird tests but it should similar to this.

Comment 26

[John Hopkins (:jhopkins)]

Attachment: proposed buildbot-configs patch

Comment 27

[John Hopkins (:jhopkins)]

Attachment: diff old/new dump_master.py output

Comment 28

[John Hopkins (:jhopkins)]

Comment on attachment 648789 [details] [diff] [review]
proposed buildbot-configs patch

Landed in https://hg.mozilla.org/build/buildbot-configs/rev/225fdc17d656

Comment 29

[Ed Morley [:emorley]]

Has a reconfig occurred since this landed?

Not wanting to hassle people if it hasn't (no immediate rush); but slightly hesitant to wait 6 days again before asking (like in comment 17), in case there is a chance we've had one and this didn't quite stop the builds again :-)

(Had another instance of confusion on IRC this morning over the 10.5 builds again, which is what prompted me to take another look at &noignore=1).

Comment 30

[John Hopkins (:jhopkins)]

(In reply to Ed Morley [:edmorley] from comment #29)
> Has a reconfig occurred since this landed?

I think it's safe to say that if the patch wasn't landed on the production branch directly or hasn't been merged from the default branch to production, then it is still pending a reconfig (which is the case right now).

We usually do a handful of reconfigs per week so this change should take effect reasonably soon.

Comment 31

[Mark Banner (:standard8)]

John, do we need changes to mozilla/thunderbird_config.py as well to disable the OS X debug builds?

Comment 32

[Chris AtLee [:catlee]]

deployed via reconfig today

Comment 33

[Chris AtLee [:catlee]]

apparently we're still running 10.5 talos tests on m-c and m-i

Comment 34

[Mark Banner (:standard8)]

On Thunderbird, it seems like we're still running:

TB Rev3 MacOSX Leopard 10.5.8 comm-central opt test xpcshell
TB Rev3 MacOSX Leopard 10.5.8 comm-central opt test mozmill

Comment 35

[Mark Banner (:standard8)]

(In reply to [:armenzg] - gone from Aug. 3rd to Aug. 27th from comment #11)
> Created attachment 645007 [details] [diff] [review]
> disable 32-bit debug builds (64-bit will still run)

Errm, is it intentional that we're disabling the 32 bit builds/tests? What will cover the 32 bit side of the build for test failures?

I can understand shifting them to 10.6 or 10.7, but I'm not sure if completely loosing that half of the testing is intentional or not.

Comment 36

[[e-mail] (:bhearsum)]

(In reply to Mark Banner (:standard8) from comment #35)
> (In reply to [:armenzg] - gone from Aug. 3rd to Aug. 27th from comment #11)
> > Created attachment 645007 [details] [diff] [review]
> > disable 32-bit debug builds (64-bit will still run)
> 
> Errm, is it intentional that we're disabling the 32 bit builds/tests? What
> will cover the 32 bit side of the build for test failures?
> 
> I can understand shifting them to 10.6 or 10.7, but I'm not sure if
> completely loosing that half of the testing is intentional or not.

I have a vague memory of this getting discussed...I can't remember where or with whom though =\.

Comment 37

[Ed Morley [:emorley]]

(In reply to Ben Hearsum [:bhearsum] from comment #36)
> > Errm, is it intentional that we're disabling the 32 bit builds/tests? What
> > will cover the 32 bit side of the build for test failures?
> > 
> > I can understand shifting them to 10.6 or 10.7, but I'm not sure if
> > completely loosing that half of the testing is intentional or not.
> 
> I have a vague memory of this getting discussed...I can't remember where or
> with whom though =\.

Perhaps on the newsgroups?
https://groups.google.com/d/msg/mozilla.dev.platform/aT7hy7YDdqA/-IrOmUxNKvYJ

Also, we're still running tests on inbound/m-c/...

Comment 38

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #37)
> Also, we're still running tests on inbound/m-c/...

To clarify:
Unrelated to comment 36, we're still running busted 10.5 tests on several trees - this bug's patch doesn't appear to have worked.

Comment 39

[John Hopkins (:jhopkins)]

I am having a look now

Comment 40

[John Hopkins (:jhopkins)]

I believe what needs to be done is to remove 'leopard' from the mozilla-inbound/central *_tests entries, such as:

------
'chromer_tests': (1,
                   True,
                   {},
                   ['fedora',
                    'fedora64',
                    'xp',
                    'win7',
                    'leopard',
                    'snowleopard',
                    'lion',
                    'mountainlion']),
 'cold_tests': (0,
                True,
                {'talosAddOns': ['profiles/dirtyDBs.zip',
                                 'profiles/dirtyMaxDBs.zip']},
                ['leopard',
                 'snowleopard',
                 'lion',
                 'mountainlion',
                 'fedora',
                 'fedora64']),
------

I also see a ['platforms']['macosx64']['leopard'] dictionary that should be removed similarly.

Comment 42

[John Hopkins (:jhopkins)]

Attachment: one approach. work in progress

this is proving to be much more difficult than i imagined.  i'm setting this bug aside for now.

Comment 43

[John Hopkins (:jhopkins)]

Attachment: remove builders directly once all preprocessing is done

Comment 44

[John Hopkins (:jhopkins)]

Attachment: list of builders deleted or kept

This goes along with patch 653099 - it's the output generated during checkconfig.  Does this match the list of things we want to delete?

Comment 45

[[e-mail] (:bhearsum)]

Comment on attachment 653099 [details] [diff] [review]
remove builders directly once all preprocessing is done

Review of attachment 653099 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/tests_master.cfg
@@ +177,5 @@
> +            print("deleting builddir=%s branch=%s platform=%s" % (b['builddir'], b['properties']['branch'], b['properties']['platform']))
> +            c['builders'] = filter(lambda i: i != b, c['builders'])
> +        else:
> +            print("keeping  builddir=%s branch=%s platform=%s" % (b['builddir'], b['properties']['branch'], b['properties']['platform']))
> +

Ugh, does this post-facto logic need to spread to config.py? Why can't this be done there instead?

Comment 46

[John Hopkins (:jhopkins)]

(In reply to Ben Hearsum [:bhearsum] from comment #45)
> Ugh, does this post-facto logic need to spread to config.py? Why can't this
> be done there instead?

I wasn't able to figure out a clean way to do this in config.py (I did invest quite a bit of time trying).  It was easiest to let the existing logic run and then remove the desired builders.

Comment 47

[Ed Morley [:emorley]]

Comment on attachment 653101 [details] [diff] [review]
list of builders deleted or kept

I would have said it looks fine, since there are deleted leopard entries, except some of those entries list platform as macosx64, whereas I thought it should be macosx for 10.5? 

I'm clearly missing something crucial/obvious here, so I don't think I'm really qualified enough to say whether it's working or not, sorry!

Comment 48

[Phil Ringnalda (:philor)]

Comment on attachment 653101 [details] [diff] [review]
list of builders deleted or kept

>deleting builddir=jetpack-mozilla-aurora-leopard-debug branch=addon-sdk platform=leopard
>deleting builddir=jetpack-mozilla-aurora-leopard-opt branch=addon-sdk platform=leopard
>deleting builddir=jetpack-mozilla-beta-leopard-debug branch=addon-sdk platform=leopard
>deleting builddir=jetpack-mozilla-beta-leopard-opt branch=addon-sdk platform=leopard
>deleting builddir=jetpack-mozilla-central-leopard-debug branch=addon-sdk platform=leopard
>deleting builddir=jetpack-mozilla-central-leopard-opt branch=addon-sdk platform=leopard
>deleting builddir=jetpack-mozilla-release-leopard-debug branch=addon-sdk platform=leopard
>deleting builddir=jetpack-mozilla-release-leopard-opt branch=addon-sdk platform=leopard

That ain't right (the -aurora one should stick around for a week, the -beta one for seven weeks, and the -release one for thirteen weeks), but... whatever. They've never had a 10.5-only bustage, and if the only choices available are to keep running thousands of jobs that don't start, or take away a few weeks of coverage from them, better to stop the wasted jobs.

Comment 50

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

1) we will stop doing 10.5 testing in FF17 but will keep smaller 10.5 test pool to support 10.5 testing in FF14/15/16/ESR. 


2) There is some concern that turning off 10.5 testing would have the side effect of also turning off *all* our 32bit testing on OSX. Filed bug#784465 to track need for 32bit testing on 10.6.

Comment 51

[[e-mail] (:bhearsum)]

Attachment: remove macosx64 from BRANCHES *and* PLATFORMS

John and I video chatted about this the other day and I think this should do what we need....the problem with the original patch is that it doesn't remove 'leopard' from the PLATFORMS dictionary. Confusingly, *it* gets passed to generateTalosBranchObjects instead of being copied into BRANCHES (https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla-tests/tests_master.cfg#L96)

I'll post a diff that shows the before and after builder list, too.

Comment 52

[[e-mail] (:bhearsum)]

Attachment: removed builders w/ my patch

Here's another check w/ grep on the "after" list of builders:
grep Leopard b.txt | grep -v Snow | grep -v aurora | grep -v beta | grep -v esr10 | grep -v release

Comment 53

[Rail Aliiev [:rail]]

Comment on attachment 654624 [details]
removed builders w/ my patch

Ship it!

Comment 54

[John Hopkins (:jhopkins)]

Comment on attachment 654624 [details]
removed builders w/ my patch

Still a bunch of "release-" builders being removed.  These should be preserved and phased out over time.

Comment 55

[John Hopkins (:jhopkins)]

Comment on attachment 654623 [details] [diff] [review]
remove macosx64 from BRANCHES *and* PLATFORMS

See previous comment.

Comment 56

[[e-mail] (:bhearsum)]

Attachment: 2nd try

Try #2. Turns out we can't touch PLATFORMS because it gets passed *directly* to generateTalosBranchObjects, not interpolated into BRANCHES. This patch should remove all Leopard builders except:
* Those on aurora, beta, release, esr10
* The jetpack jobs from PROJECTS

We can't remove the jetpack ones because PROJECTS again gets passed directly to generateTalosBranchObjects and has no facility for branch-specific options. I think we should just leave them on and let them fail. I'll attach another patch that shows the removed builders, but here's my grepping on the full dump of them:
grep "[Ll]eopard" ~/tmp/b.txt | grep -v Snow  | grep -v aurora | grep -v beta | grep -v release | grep -v esr
jetpack-mozilla-central-leopard-opt
jetpack-mozilla-central-leopard-debug
jetpack-mozilla-central-snowleopard-opt
jetpack-mozilla-central-snowleopard-debug

Comment 57

[[e-mail] (:bhearsum)]

Attachment: removed builders

Comment 58

[[e-mail] (:bhearsum)]

Just remembered: we should be careful when landing this, removing builders has the potential to hang any slaves currently running those builds. Might need to graceful shutdown the masters to do it correctly.

Comment 59

[Rail Aliiev [:rail]]

Comment on attachment 654801 [details] [diff] [review]
2nd try

lgtm

Comment 60

[[e-mail] (:bhearsum)]

Attachment: attach the right patch

I forgot to rediff before I attached this. The builder list was generated w/ this version of the patch.

Comment 61

[[e-mail] (:bhearsum)]

Comment on attachment 655012 [details] [diff] [review]
attach the right patch

I'll probably land this in production later today.

Comment 62

[[e-mail] (:bhearsum)]

Not an acute issue -> not a buildduty concern.

Comment 63

[[e-mail] (:bhearsum)]

This bug doesn't depend on us having 32-bit tests on 10.6

Comment 64

[[e-mail] (:bhearsum)]

This is in production now, we won't be running any Leopard tests for pushes from now on.

Comment 65

[Mark Banner (:standard8)]

Assuming I'm not mistaking anything, this isn't fully fixed yet - We are still running "TB OS X 10.7 32-bit comm-central leak test build".

AFAICT the config.py changes in attachment 645007 [details] [diff] [review] haven't been ported to thunderbird_config.py.

Comment 66

[[e-mail] (:bhearsum)]

(In reply to Mark Banner (:standard8) from comment #65)
> Assuming I'm not mistaking anything, this isn't fully fixed yet - We are
> still running "TB OS X 10.7 32-bit comm-central leak test build".
> 
> AFAICT the config.py changes in attachment 645007 [details] [diff] [review]
> haven't been ported to thunderbird_config.py.

Armen, can you please port that patch to thunderbird_config.py?

Comment 67

[Armen [:armenzg]]

Got poked by bhearsum. I will focus on it.

Comment 68

[Armen [:armenzg]]

Attachment: disable 32-bit macosx debug builds (except few branches)

These are the builders that get removed:
Armens-MacBook-Air:build armenzg$ diff old new 
121d120
< TB OS X 10.7 32-bit comm-aurora leak test build NightlyBuildFactory
153d151
< TB OS X 10.7 32-bit comm-central leak test build NightlyBuildFactory

Comment 69

[Armen [:armenzg]]

Attachment: disable 32-bit macosx debug builds (except few branches)

Back porting to Firefox today's simpler config changes.

The only builders being changed are:
Armens-MacBook-Air:build armenzg$ diff old new 
121d120
< TB OS X 10.7 32-bit comm-aurora leak test build NightlyBuildFactory
153d151
< TB OS X 10.7 32-bit comm-central leak test build NightlyBuildFactory

Comment 70

[Armen [:armenzg]]

Comment on attachment 656200 [details] [diff] [review]
disable 32-bit macosx debug builds (except few branches)

http://hg.mozilla.org/build/buildbot-configs/rev/cbac6ee52d5f

Comment 71

[[e-mail] (:bhearsum)]

Made it to production (again) today. I think we're all done here (again).