Closed Bug 500676 Opened 15 years ago Closed 15 years ago

Transition Mac builds to 10.5

Categories

(Mozilla Messaging Graveyard :: Release Engineering, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: standard8, Assigned: gozer)

References

Details

(Whiteboard: [waiting on aus transition?])

Attachments

(5 files, 4 obsolete files)

I think we've already decided that to move to Mac OS X 10.5 for our builders is a good idea. We've already have unit tests and shark builds running on 10.5 with no issues. I've been using one of the try servers on and off for a few days and I've not had any problems with it, although I am on 10.5 not 10.4.

It would be good to get this done before b3 so we have plenty of time to check for issues.

If we can, I'd like to keep a few 10.4 builders, possibly running via the make package/make package-tests facility and downloading the 10.5 builds. Though I'll file a separate bug for transitioning those when I get chance (and mark up dependencies).
One easy first step is to move the trunk bloat builders to leopard.
Attachment #385398 - Flags: review?(bugzilla)
And the same for regular comm-central trunk dep/nightly builders.
Attachment #385399 - Flags: review?(bugzilla)
Might as well run them in parallel for a while.
Attachment #385398 - Attachment is obsolete: true
Attachment #385399 - Attachment is obsolete: true
Attachment #385403 - Flags: review?(bugzilla)
Attachment #385398 - Flags: review?(bugzilla)
Attachment #385399 - Flags: review?(bugzilla)
Missed a small detail, for the interim, better make sure only 10.4 builders keep uploading to ftp.m.o, as the generated package names will be the same.

Once they are nice and stable, we should switch to letting the 10.5 builders upload to stage
Attachment #385403 - Attachment is obsolete: true
Attachment #385405 - Flags: review?(bugzilla)
Attachment #385403 - Flags: review?(bugzilla)
Attachment #385405 - Flags: review?(bugzilla) → review+
Comment on attachment 385405 [details] [diff] [review]
[checked in] Add 10.5 comm-central bloat/dep/nightly builders in parallel v2

Hopefully the bloat threshold should be fine. I can't see any problems with this.
Comment on attachment 385405 [details] [diff] [review]
[checked in] Add 10.5 comm-central bloat/dep/nightly builders in parallel v2

http://hg.mozilla.org/build/buildbot-configs/rev/21349848cb4a
Attachment #385405 - Attachment filename: comm-central-10.5.diff → [checked in] comm-central-10.5.diff
Attachment #385405 - Attachment is obsolete: true
Attachment #385405 - Attachment description: Add 10.5 comm-central bloat/dep/nightly builders in parallel v2 → [checked in] Add 10.5 comm-central bloat/dep/nightly builders in parallel v2
Attachment #385405 - Attachment filename: [checked in] comm-central-10.5.diff → comm-central-10.5.diff
Missed the required mozconfigs.

comparing with ssh://hg.mozilla.org/build/buildbot-configs
searching for changes
changeset:   1236:3cba119333d6
tag:         tip
user:        Philippe M. Chiasson <gozer@mozillamessaging.com>
date:        Fri Jun 26 21:59:18 2009 -0400
summary:     Bug 500676. Copy missing OS X 10.5 mozconfigs
I'm guessing 10.5 leaks more than 10.4. Can I get a copy of sdleak.tree and sdleak.log for reasonably equivalent 10.5 and 10.4 builds please?

For now we can probably tweak the limit to 2300000.
Increasing the limit to 230,000

changeset:   1238:373ab65f2533
tag:         tip
user:        Philippe M. Chiasson <gozer@mozillamessaging.com>
date:        Sat Jun 27 22:57:43 2009 -0400
summary:     Bug 500676. Increase bloat leak treshold for OS X 10.5
(In reply to comment #8)
> For now we can probably tweak the limit to 2300000.

(In reply to comment #9)
> Increasing the limit to 230,000
> 
> summary:     Bug 500676. Increase bloat leak treshold for OS X 10.5

Old value was 1,400,000.
Did you miss a '0', or did you actually want to _decrease_ the threshold?
(In reply to comment #10)
> (In reply to comment #8)
> > For now we can probably tweak the limit to 2300000.
> 
> (In reply to comment #9)
> > Increasing the limit to 230,000
> > 
> > summary:     Bug 500676. Increase bloat leak treshold for OS X 10.5
> 
> Old value was 1,400,000.
> Did you miss a '0', or did you actually want to _decrease_ the threshold?

I think gozer misread my value - we want it increasing to 2,300,000.
I've checked in a change to increase the config to 2,300,000 (which we really want):

changeset:   1239:55669053980b
tag:         tip
user:        Mark Banner <bugzilla@standard8.plus.com>
date:        Mon Jun 29 09:24:41 2009 +0100
summary:     Part of bug 500676. Really increase the leak threshold this time. r=me.

AFAIK buildbot will require a manual resync to pick these changes up, so don't expect them straight away as I can't do that.
(In reply to comment #12)
> I've checked in a change to increase the config to 2,300,000 (which we really
> want):
> 
> changeset:   1239:55669053980b
> tag:         tip
> user:        Mark Banner <bugzilla@standard8.plus.com>
> date:        Mon Jun 29 09:24:41 2009 +0100
> summary:     Part of bug 500676. Really increase the leak threshold this time.
> r=me.
> 
> AFAIK buildbot will require a manual resync to pick these changes up, so don't
> expect them straight away as I can't do that.

Buildbot reconfigured.
(In reply to comment #8)
> I'm guessing 10.5 leaks more than 10.4. Can I get a copy of sdleak.tree and
> sdleak.log for reasonably equivalent 10.5 and 10.4 builds please?

I'll be attaching them separately.

I've grabbed them from the exact same revisions:

momo-vm-osx-leopard-02
rev:76a9c718b6fa
m-c:f63f28455fdf
Mail RLk:0B
Mail Lk:2.12MB
Mail MH:21.1MB
Mail A:735414

momo-vm-osx-tiger-02
rev:76a9c718b6fa
m-c:f63f28455fdf
Mail RLk:0B
Mail Lk:946KB
Mail MH:20.2MB
Mail A:662726
Let the 10.5 builds upload to stage, under 'comm-central-trunk-10.5-test' for now

comparing with ssh://hg.mozilla.org/build/buildbot-configs
searching for changes
changeset:   1241:28788f8ce818
tag:         tip
user:        Philippe M. Chiasson <gozer@mozillamessaging.com>
date:        Mon Jun 29 13:50:37 2009 -0400
summary:     Bug 500676. Let the experimental 10.5 trunk dep/nightly builds upload to stage
I've taken a look at the bloat builds. The console output for 10.4 versus 10.5 is basically the same apart from a few re-ordered lines.

The difference in the bloat logs seems to be operating system originated leaks. So whether we're not cleaning things up as well or 10.5 leaks more, I think it is mainly core issues that we can't do anything about here.
So is this merely an artifact of which OS version the bloat tests are running on?  If so, that's not a big deal.  If, on the other hand, the generated code has different bloat characteristics, that makes the argument for moving forward less compelling.
(In reply to comment #22)
> So is this merely an artifact of which OS version the bloat tests are running
> on?  If so, that's not a big deal.  If, on the other hand, the generated code
> has different bloat characteristics, that makes the argument for moving forward
> less compelling.

It is hard to tell without doing more work. However, here's a few comparisons that may help:

Thunderbird trunk builds:

Trace Malloc Leaks: 944 kB (10.4) versus 2.13MB (10.5)
Maximum Heap: 20.3MB (10.4) versus 20.9MB (10.5)
Allocs: 670000 (10.4) versus 720000 (10.5)

Firefox 3.0 builds (I believe these are 10.4) versus Firefox 3.5 builds (10.5):

Trace Malloc Leaks: 906kB (FF 3.0) versus 2.09MB (FF 3.5)
Maximum Heap: 20.9 MB (FF 3.0) versus 23.5MB (FF 3.5)
Allocs: 490K (FF 3.0) versus 830000 (FF 3.5)

Whilst I realise that there have been a lot of changes in FF 3.5, to end up with similar ratios is certainly interesting and probably something we can't do much about.

gozer also tells me that the 10.4 builders are taking up over 40% CPU on the host machine, where the 10.5 builders are only taking up 15-20%, so in the long term it would be better for us to do the switch as we can run more builders (which we need).
(In reply to comment #23)
> (In reply to comment #22)
> [...]
> gozer also tells me that the 10.4 builders are taking up over 40% CPU on the
> host machine, where the 10.5 builders are only taking up 15-20%, so in the long
> term it would be better for us to do the switch as we can run more builders
> (which we need).

Which from my pov is the main reason I'd like to see us move over to 10.5 builders. They are more stable, faster and actually supported.
I've posted to m.d.platform regarding the differences to see if we can get some ideas there.

Later I'd like to see if we can arrange to get a build with known results copied over to a different builder and run to see what figures we end up with. I'll chat to gozer about how to do this on irc.
Assignee: gozer → bugzilla
Assignee: bugzilla → gozer
Depends on: 487629
I've got some preliminary results from copying over a 10.5 bloat build over to a 10.4 builder. For reference, it's this build: <http://build.mozillamessaging.com/buildbot/production/builders/MacOSX%2010.5%20comm-central%20bloat/builds/142>


Originally, on the 10.5 builder, the leak stats were reported as:
Leaks: 2230141 bytes, 24152 allocations
Maximum Heap Size: 22114783 bytes
159038414 bytes were allocated in 727932 allocations.
Logged 169 free (or realloc) calls for which we missed the original malloc.

After re-running the bloat tests on 10.4, the stats looked like:
Leaks: 905910 bytes, 11630 allocations
Maximum Heap Size: 15221486 bytes
110315586 bytes were allocated in 426731 allocations.
Logged 130 free (or realloc) calls for which we missed the original malloc.

For comparaison, an ordinary 10.4 build from about the same time:

Leaks: 966532 bytes, 12165 allocations
Maximum Heap Size: 20907414 bytes
144492474 bytes were allocated in 663645 allocations.
Logged 358 free (or realloc) calls for which we missed the original malloc.
(In reply to comment #26)
> After re-running the bloat tests on 10.4, the stats looked like:
> Leaks: 905910 bytes, 11630 allocations
> Maximum Heap Size: 15221486 bytes
> 110315586 bytes were allocated in 426731 allocations.
> Logged 130 free (or realloc) calls for which we missed the original malloc.
> 
> For comparaison, an ordinary 10.4 build from about the same time:
> 
> Leaks: 966532 bytes, 12165 allocations
> Maximum Heap Size: 20907414 bytes
> 144492474 bytes were allocated in 663645 allocations.
> Logged 358 free (or realloc) calls for which we missed the original malloc.

That's quite interesting, and shows reductions all round on the numbers side. This makes it look like the 10.4 to 10.5 transition is to do with the system libraries, and the 10.5 build running on 10.4 benefits (probably) from a newer (more efficient?) compiler.

I think it would be worth a test to take a 10.4 build and run it on 10.5 - just to confirm the differences go that way as well.
And numbers from a 10.4 build running on a 10.5 box:

Leaks: 2219038 bytes, 24019 allocations
Maximum Heap Size: 23091698 bytes
160309694 bytes were allocated in 764193 allocations.
Logged 206 free (or realloc) calls for which we missed the original malloc.
Cool.  Looks like the gains are compile time (which this change will help), and the losses are all runtime, which this change won't effect.  Let's do it!
changeset:   1288:e7d359a735eb
user:        Philippe M. Chiasson <gozer@mozillamessaging.com>
date:        Thu Jul 09 15:14:16 2009 -0400
summary:     Bug 500676. Move TB bloat builders to OS X 10.5 only

changeset:   1287:aab3c3fdcebc
user:        Philippe M. Chiasson <gozer@mozillamessaging.com>
date:        Thu Jul 09 15:09:10 2009 -0400
summary:     Bug 500676. Move TB unittest builders to OS X 10.5 only.

Buildbot master reconfigured.
changeset:   1293:2d3a0dd84553
user:        Philippe M. Chiasson <gozer@mozillamessaging.com>
date:        Thu Jul 09 21:48:42 2009 -0400
summary:     Bug 500676. Move TB comm-central dep/nightly builders to OS X 10.5 only.

searching for changes
changeset:   1295:9605aa5f745c
tag:         tip
user:        Philippe M. Chiasson <gozer@mozillamessaging.com>
date:        Thu Jul 09 22:06:08 2009 -0400
summary:     Bug 500676. Forget we have 10.4 builders

Buildbot master reconfigured.
The 1.9.1 bloat builders had the old 10.4 threshold, I've increased it and added a bit over the trunk level because 1.9.1 leaks a bit more.

changeset:   1296:fb347a62a71a
tag:         tip
user:        Mark Banner <bugzilla@standard8.plus.com>
date:        Fri Jul 10 10:40:13 2009 +0100
files:       thunderbird/config.py
description:
Adjust leak threshold for the comm-1.9.1 10.5 builders (part of bug 500676).
Buildbot master reconfigured.
2 new OS X 10.5 builders (vm-osx-leopard-03/04) have been added to the build pool and are green.
Just to summarise:

We've switched all builders to 10.5 except for the dep/nightly builder on 1.9.1 which is still on 10.4. Trunk nightly builds are on 10.5.

AFAIK The builders have been stable since we landed the last config patch.
Whiteboard: [waiting on aus transition?]
(In reply to comment #35)
> Just to summarise:
> 
> We've switched all builders to 10.5 except for the dep/nightly builder on 1.9.1
> which is still on 10.4. Trunk nightly builds are on 10.5.

They are still on 10.4 because of bug 487629 or bug 501201. So if we are willing/ready to close bug 487629 soon, this last builder can switch over to 10.5 too.

> AFAIK The builders have been stable since we landed the last config patch.

I agree, pretty green and stable.
comm-1.9.1 dep/nightly builders have been moved to 10.5, and have been green.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: