674647 - Switch the OS X debug builds to 10.6

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P2)

Description

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: mozconfig patch

We should build the debug build in the same way we build the release ones: a 32 bit binary that supports 10.5 combined with a 64 bit one that supports 10.6.

The first step is to move the builds to a 10.6 machine. The second one is to change the mozconfig files.

The attached patch changes the mozconfigs such that the diff from
macosx64/mozilla-2.0/nightly/mozconfig to 
macosx64/mozilla-2.0/debug/mozconfig is

+ac_add_options --disable-optimize
+ac_add_options --enable-debug
+ac_add_options --enable-trace-malloc

And from

macosx64/mozilla-2.0/nightly/mozconfig to
mozilla2/macosx64/try/nightly/mozconfig is

+ac_add_options --disable-auto-deps
 
 export CFLAGS="-gdwarf-2"
 export CXXFLAGS="-gdwarf-2"
@@ -19,3 +20,9 @@
 
 # Enable parallel compiling
 mk_add_options MOZ_MAKE_FLAGS="-j4"
+if [ -f $topsrcdir/mozconfig-extra ] ; then
+  . $topsrcdir/mozconfig-extra
+fi
+if [ -f $topsrcdir/mozconfig-extra-macosx64 ] ; then
+  . $topsrcdir/mozconfig-extra-macosx64
+fi

We should probably also delete the macosx directory completely.

I am not sure why accessibility was enabled in the debug build but not the nightly. We should probably have it enabled in none or both.

Comment 1

[Armen [:armenzg]]

(In reply to comment #0)
> We should build the debug build in the same way we build the release ones: a
> 32 bit binary that supports 10.5 combined with a 64 bit one that supports
> 10.6.
Any specific technical reason to do so? (since we don't ship them to users)
or is this to make things consistent? or to make things for testing community?
What I am trying to get to is: What is the ultimate reason to do so?

IIUC if we were to make our debug builds to be universal it would make our debug builds to take double the time to be pushed out the door and end to end results for debug builds on mac be much longer.
If that is OK with developers then is fine by us (I think).

> I am not sure why accessibility was enabled in the debug build but not the
> nightly. We should probably have it enabled in none or both.

accessibility is not ready to be shipped to our users and that is why we chose to only enable it debug builds.
Enabling it by default on nightly builds would make the builds slower and we don't want that until ready.
We can talk with davidb if we want to figure out what the right way of doing this.

Comment 2

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

> Any specific technical reason to do so? (since we don't ship them to users)
> or is this to make things consistent? or to make things for testing
> community?
> What I am trying to get to is: What is the ultimate reason to do so?

Having debugs builds is so that we get more testing (via more asserts) and we can debug issues faster. Having them be different from the bits we ship makes them less useful. For example, we don't have a 64 bit debug.

This is also a step in drop the dependency in the 10.5 sdk.
 
> IIUC if we were to make our debug builds to be universal it would make our
> debug builds to take double the time to be pushed out the door and end to
> end results for debug builds on mac be much longer.
> If that is OK with developers then is fine by us (I think).

Fair point. We could have a 64 bit debug and a 32 bit debug which can be built and test in parallel.

> > I am not sure why accessibility was enabled in the debug build but not the
> > nightly. We should probably have it enabled in none or both.
> 
> accessibility is not ready to be shipped to our users and that is why we
> chose to only enable it debug builds.
> Enabling it by default on nightly builds would make the builds slower and we
> don't want that until ready.
> We can talk with davidb if we want to figure out what the right way of doing
> this.

I remember some discussion about disabling it at runtime. Davidb, has that be implemented? Would that be hard to implement if not?

Comment 3

[Armen [:armenzg]]

(In reply to comment #2)
> > Any specific technical reason to do so? (since we don't ship them to users)
> > or is this to make things consistent? or to make things for testing
> > community?
> > What I am trying to get to is: What is the ultimate reason to do so?
> 
> Having debugs builds is so that we get more testing (via more asserts) and
> we can debug issues faster. Having them be different from the bits we ship
> makes them less useful. For example, we don't have a 64 bit debug.

If I am not mistaken we do have 64-bit debug builds:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx64-debug/1311792769/
besides the 32-bit debug builds:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1311794897/

Is this right?

> 
> This is also a step in drop the dependency in the 10.5 sdk.
>  
You are going to need sign off on that.
Probably the platform Tuesday meeting is the right place for that.

> > IIUC if we were to make our debug builds to be universal it would make our
> > debug builds to take double the time to be pushed out the door and end to
> > end results for debug builds on mac be much longer.
> > If that is OK with developers then is fine by us (I think).
> 
> Fair point. We could have a 64 bit debug and a 32 bit debug which can be
> built and test in parallel.
> 
We wouldn't want to be generating:
* 10.5 debug (32-bit as of now)
* 10.6 universal debug (being debated)
* 10.6 debug (64-bit as of now)

I would prefer to be doing this:
* 10.6 universal debug (double the compilation time)

> > > I am not sure why accessibility was enabled in the debug build but not the
> > > nightly. We should probably have it enabled in none or both.
> > 
> > accessibility is not ready to be shipped to our users and that is why we
> > chose to only enable it debug builds.
> > Enabling it by default on nightly builds would make the builds slower and we
> > don't want that until ready.
> > We can talk with davidb if we want to figure out what the right way of doing
> > this.
> 
> I remember some discussion about disabling it at runtime. Davidb, has that
> be implemented? Would that be hard to implement if not?

He filed bug 661740 for that.

Comment 4

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

 > If I am not mistaken we do have 64-bit debug builds:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> central-macosx64-debug/1311792769/
> besides the 32-bit debug builds:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> central-macosx-debug/1311794897/
> 
> Is this right?

Sorry, yes. So the situation with 10.6 is already better. The main point of the bug is to get us out of building on 10.5, so I can split if you want.


> > This is also a step in drop the dependency in the 10.5 sdk.
> >  
> You are going to need sign off on that.
> Probably the platform Tuesday meeting is the right place for that.

Cool. Do you have a link? Not that I am *not* proposing dropping support for 10.5, just stop building with the 10.5 sdk.

> > Fair point. We could have a 64 bit debug and a 32 bit debug which can be
> > built and test in parallel.
> > 
> We wouldn't want to be generating:
> * 10.5 debug (32-bit as of now)
> * 10.6 universal debug (being debated)
> * 10.6 debug (64-bit as of now)
> 
> I would prefer to be doing this:
> * 10.6 universal debug (double the compilation time)

And then run tests with -arch i386 and -arch x86_64? I would consider that the prefect solution if we are OK with the build time.

> > I remember some discussion about disabling it at runtime. Davidb, has that
> > be implemented? Would that be hard to implement if not?
> 
> He filed bug 661740 for that.

Thanks

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

BTW, adding --enable-update-packaging and --enable-update-channel=nightly will have the effect of updating debug builds to opt ones. At the moment we don't support updates for debug builds, nor produce 'nightlies' for them. I suggest leaving those lines out of the mozconfig.

Comment 6

[Armen [:armenzg]]

(In reply to comment #4)
>  > If I am not mistaken we do have 64-bit debug builds:
> > http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> > central-macosx64-debug/1311792769/
> > besides the 32-bit debug builds:
> > http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> > central-macosx-debug/1311794897/
> > 
> > Is this right?
> 
> Sorry, yes. So the situation with 10.6 is already better. The main point of
> the bug is to get us out of building on 10.5, so I can split if you want.
> 
What is the part you would like to separate?
I thought we just want to focus on switching to stop using the 10.5 sdk on the debug builds which implies building the debug builds on the 10.6 machines as a universal *debug* build.

Am I following correctly?

> 
> > > This is also a step in drop the dependency in the 10.5 sdk.
> > >  
> > You are going to need sign off on that.
> > Probably the platform Tuesday meeting is the right place for that.
> 
> Cool. Do you have a link? Not that I am *not* proposing dropping support for
> 10.5, just stop building with the 10.5 sdk.
> 
https://wiki.mozilla.org/Platform#Meetings
jp can let you know more on how to bring it up. I think round table should do.
He sends a notice to dev.planning the day before with the link to the meeting hours.

> > > Fair point. We could have a 64 bit debug and a 32 bit debug which can be
> > > built and test in parallel.
> > > 
> > We wouldn't want to be generating:
> > * 10.5 debug (32-bit as of now)
> > * 10.6 universal debug (being debated)
> > * 10.6 debug (64-bit as of now)
> > 
> > I would prefer to be doing this:
> > * 10.6 universal debug (double the compilation time)
> 
> And then run tests with -arch i386 and -arch x86_64? I would consider that
> the prefect solution if we are OK with the build time.
> 
Can you elaborate more? I am afraid I don't know much about those two targets (perhaps is what we already use on the opt 10.6 testing scenario -- see below).

Perhaps explaining how the opt side works might help here (not sure if we use such flags for that scenario):
* we build 10.6 opt universal on 10.6
** we test that build on the 10.5 machines
** we test that build on the 10.6 machines

On the other side the debug tests work like this:
* we build 10.5 debug build on 10.5
** we test that build on the 10.5 machines
* we build 10.6 debug build on 10.6
** we test that build on the 10.6 machines

What you seem to want is to change it to:
* we build 10.6 *debug* universal on 10.6 (no 10.5 sdk anymore)
** we test that build on the 10.5 machines
** we test that build on the 10.6 machines

Does it make sense? Does it match what you want?

WRT to the build time increase it would be part of mentioning it on the meeting.

Comment 7

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

> What is the part you would like to separate?
> I thought we just want to focus on switching to stop using the 10.5 sdk on
> the debug builds which implies building the debug builds on the 10.6
> machines as a universal *debug* build.
> 
> Am I following correctly?

Yes, but I mixed up the somewhat unrelated issue of having the debug builds look a bit more like the regular ones.

> Can you elaborate more? I am afraid I don't know much about those two
> targets (perhaps is what we already use on the opt 10.6 testing scenario --
> see below).
> 
> Perhaps explaining how the opt side works might help here (not sure if we
> use such flags for that scenario):
> * we build 10.6 opt universal on 10.6
> ** we test that build on the 10.5 machines
> ** we test that build on the 10.6 machines

This part is "perfect" :-) So mozilla2/macosx/try/nightly/mozconfig doesn't get used at all?

> On the other side the debug tests work like this:
> * we build 10.5 debug build on 10.5
> ** we test that build on the 10.5 machines
> * we build 10.6 debug build on 10.6
> ** we test that build on the 10.6 machines
> 
> What you seem to want is to change it to:
> * we build 10.6 *debug* universal on 10.6 (no 10.5 sdk anymore)
> ** we test that build on the 10.5 machines
> ** we test that build on the 10.6 machines
> 
> Does it make sense? Does it match what you want?

Yes. Doing two independent builds on 10.6 would also match.

> WRT to the build time increase it would be part of mentioning it on the
> meeting.


What I would propose is:

* I will drop the patch and we keep this bug for tracking moving the debug build to 10.6 machines. This can be done with mostly the same configs we have now, since XCode < 4 ships with both SDKs. I think all that is needed is to pass -arch i386 explicitly since 64 bits is the default on 10.6.

With this the opt side would be unchanged and the debug side would become

* we build 10.5 debug build on 10.6 (by passing -arch i386 and using the 10.5 sdk)
** we test that build on the 10.5 machines (unchanged)
* we build 10.6 debug build on 10.6 (unchanged)
** we test that build on the 10.6 machines (unchanged)

* Bug 674655 keeps track of stop using the 10.5 SDK (and depends on this bug). With that the opt side would be similar, but we would use the 10.6 sdk for both 32 and 64 bits. The debug build would become.

* we build 10.5 debug build on 10.5 (by passing -arch i386 and using the 10.6 sdk with a 10.5 deployment target)
** we test that build on the 10.5 machines (unchanged)
* we build 10.6 debug build on 10.6 (unchanged)
** we test that build on the 10.6 machines (unchanged)

With this we can then, if desired, make debug builds universal too.

Sounds reasonable?

Comment 8

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: add a file with common setting for 10.5

The idea is to replace

ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk

with

. $topsrcdir/build/macosx/mozconfig.leopard

in the mozconfigs in http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla2/macosx

That will let them build on 10.6 machines.

Comment 9

[alexander :surkov (:asurkov)]

(In reply to comment #1)

> > I am not sure why accessibility was enabled in the debug build but not the
> > nightly. We should probably have it enabled in none or both.
> 
> accessibility is not ready to be shipped to our users and that is why we
> chose to only enable it debug builds.
> Enabling it by default on nightly builds would make the builds slower and we
> don't want that until ready.

that's right, few notes.

Enabling a11y on debug builds allows us to catch compilation errors and failing mochitests on mac. Additional reason to what you said why we shouldn't enable it on release builds is we don't want nightly testers experience potential problems because of a11y since a11y implementation is far from ideal state (for example, has noticeable perf problems).

Comment 10

[Armen [:armenzg]]

Comment on attachment 548988 [details] [diff] [review]
add a file with common setting for 10.5

I am removing myself from the review as I won't be here next week.

FTR this are the mozconfig files that are used for leopard:
http://hg.mozilla.org/build/buildbot-configs/file/b0c3644daa95/mozilla2/macosx/mozilla-central/nightly/mozconfig
http://hg.mozilla.org/build/buildbot-configs/file/b0c3644daa95/mozilla2/macosx/mozilla-central/debug/mozconfig

Also worth bring it up that there will be changes in the mozconfig world in bug 558180.

Comment 11

[Armen [:armenzg]]

(In reply to comment #7)
> > Can you elaborate more? I am afraid I don't know much about those two
> > targets (perhaps is what we already use on the opt 10.6 testing scenario --
> > see below).
> > 
> > Perhaps explaining how the opt side works might help here (not sure if we
> > use such flags for that scenario):
> > * we build 10.6 opt universal on 10.6
> > ** we test that build on the 10.5 machines
> > ** we test that build on the 10.6 machines
> 
> This part is "perfect" :-) So mozilla2/macosx/try/nightly/mozconfig doesn't
> get used at all?
> 
I don't think it gets used since we don't have nightly builds for try.

> > On the other side the debug tests work like this:
> > * we build 10.5 debug build on 10.5
> > ** we test that build on the 10.5 machines
> > * we build 10.6 debug build on 10.6
> > ** we test that build on the 10.6 machines
> > 
> > What you seem to want is to change it to:
> > * we build 10.6 *debug* universal on 10.6 (no 10.5 sdk anymore)
> > ** we test that build on the 10.5 machines
> > ** we test that build on the 10.6 machines
> > 
> > Does it make sense? Does it match what you want?
> 
> Yes. Doing two independent builds on 10.6 would also match.
> 
To be completely sure, you want 10.6 *opt* universal (unchanged) and 10.6 *debug* on 10.6, correct?

This means we would run 2 different debug builds on 10.6.

> * we build 10.5 debug build on 10.6 (by passing -arch i386 and using the
> 10.5 sdk)
> * we build 10.5 debug build on 10.5 (by passing -arch i386 and using the
> 10.6 sdk with a 10.5 deployment target)

These two lines either have typos or I am extremely confused of what you mean.

Comment 12

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

> > This part is "perfect" :-) So mozilla2/macosx/try/nightly/mozconfig doesn't
> > get used at all?
> > 
> I don't think it gets used since we don't have nightly builds for try.

Can we delete it?
 
> > * we build 10.5 debug build on 10.6 (by passing -arch i386 and using the
> > 10.5 sdk)
> > * we build 10.5 debug build on 10.5 (by passing -arch i386 and using the
> > 10.6 sdk with a 10.5 deployment target)
> 
> These two lines either have typos or I am extremely confused of what you
> mean.

yes, sorry, it should have been:

* we build 10.5 debug build on **10.6** (by passing -arch i386 and using the 10.6 sdk with a 10.5 deployment target)
** we test that build on the 10.5 machines (unchanged)
* we build 10.6 debug build on 10.6 (unchanged)
** we test that build on the 10.6 machines (unchanged)

Comment 13

[Armen [:armenzg]]

Comment on attachment 548988 [details] [diff] [review]
add a file with common setting for 10.5

This looks good to me but I would like ted or someone from the build side that can look at it.

Comment 14

[Armen [:armenzg]]

Attachment: [untested] running 10.5 leak jobs on 10.6 machines

Before I head out for my week, coop will be on point to try to find out someone that could test this change (as catlee and I are gone next week).

Rafael, if the following mozconfig is grabbed by a 10.6 machine to do a 10.5 leak build and your patch is landed, should it be all that is needed at first?
> http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla2/macosx/mozilla-central/debug/mozconfig

These are the only 2 changes I see between a leak build for 10.6 on a moz2-darwin10 slave and a leak build for 10.5 on a moz2-darwin10 slave (ignoring accessibility).
--- mozilla2/macosx/mozilla-central/debug/mozconfig     2011-02-22 11:42:57.000000000 -0500
+++ mozilla2/macosx64/mozilla-central/debug/mozconfig   2011-06-24 15:22:44.000000000 -0400
@@ -1,7 +1,2 @@
-# Don't use the standard mozconfig. We don't want universal for a debug build. 
-#. $topsrcdir/build/macosx/universal/mozconfig
-
-ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk
-
 ac_add_options --disable-optimize
 ac_add_options --enable-debug


These need to happen if *no* mozconfig changes are needed:
* the patch that respindola asks for review needs to land first on mozilla-central
* I believe the patch I attached is all that is needed
* test on staging by triggering a build (apply my patch locally)
* trigger unit tests (by grabbing a sendchange)

If a mozconfig is needed:
* we would need to land the mozconfig change on a user repo (on the production branch)
* point the staging master to that user repo (applying a mozconfig change locally does not do the trick)

Comment 15

[Armen [:armenzg]]

After talking with nthomas we will have to modify the mozconfig to include this:
> . $topsrcdir/build/macosx/mozconfig.leopard

Right Rafael?

BTW my patch needs not to break 1.9.2.

Comment 16

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

(In reply to comment #15)
> After talking with nthomas we will have to modify the mozconfig to include
> this:
> > . $topsrcdir/build/macosx/mozconfig.leopard
> 
> Right Rafael?

The idea is:

* check in my patch. That creates topsrcdir/build/macosx/mozconfig.leopard
* in a followup patch, change every mozcofig that says

ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk

to say

. $topsrcdir/build/macosx/mozconfig.leopard

instead.

> BTW my patch needs not to break 1.9.2.

Comment 17

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

ted, ping

Comment 18

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

In bug 674655 Josh noted that 10.5 is the default in configure. Let me know if you want me to drop the

ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk

line from this patch.

Comment 19

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Sorry, the default is just for the deployment target, not the sdk, ignore comment 18.

Comment 20

[(not currently active) Ted Mielczarek]

I'm a little confused, what are we actually planning on doing here? Are we just moving the existing 10.5 debug builds to be built on 10.6 slaves? Comment 0 talks about making them universal builds.

Comment 21

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Yes, sorry, I mixed some unrelated stuff in the early comments. This bug is really just about doing the builds as we do now (with the 10.5 sdk) but in 10.6 slaves so that in a followup change we can try switching to the 10.6 sdk.

Comment 22

[(not currently active) Ted Mielczarek]

Comment on attachment 548988 [details] [diff] [review]
add a file with common setting for 10.5

Review of attachment 548988 [details] [diff] [review]:
-----------------------------------------------------------------

Ok then.

Comment 23

[Justin Wood (:Callek)]

Comment on attachment 548988 [details] [diff] [review]
add a file with common setting for 10.5

http://hg.mozilla.org/integration/mozilla-inbound/rev/5b62f52daa1b

Comment 24

[Mounir Lamouri (:mounir)]

Backed-out from inbound due to a perma-orange, see:
http://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=0adaae140642

The patch in this bug might not bee related, the entire push has been backed out.

Comment 25

[(no longer active)]

Relanded on inbound.

Comment 26

[(no longer active)]

http://hg.mozilla.org/mozilla-central/rev/646c9d3de3f0

Comment 27

[Armen [:armenzg]]

Now that the code landed which mozconfig should we be using:
http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla2/macosx/mozilla-central/debug/mozconfig
http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla2/macosx64/mozilla-central/debug/mozconfig

I tried doing a debug build on a 10.6 machine with the "macosx" mozconfig and did not work with:

checking for correct temporary object destruction order... no
configure: error: Your compiler does not follow the C++ specification for temporary object destruction order.
*** Fix above errors and then restart with               "make -f client.mk build"
make[2]: *** [configure] Error 1
make[1]: *** [obj-firefox/Makefile] Error 2
make: *** [build] Error 2

Comment 28

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: test patch

Armen, I don't have a 10.6 machine with me at work (I can try at home), but if you get to it first, can you give the attached patch a try?

Thanks!

Comment 29

[Armen [:armenzg]]

It didn't work for me.


http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1313163209.1313163275.15952.gz&fulltext=1

# Don't use the standard mozconfig. We don't want universal for a debug build. 
#. $topsrcdir/build/macosx/universal/mozconfig

. $topsrcdir/build/macosx/mozconfig.leopard

ac_add_options --disable-optimize
ac_add_options --enable-debug
ac_add_options --enable-libxul

ac_add_options --enable-application=browser
ac_add_options --enable-tests
ac_add_options --enable-trace-malloc

# For NSS symbols
export MOZ_DEBUG_SYMBOLS=1
ac_add_options --enable-debug-symbols="-gdwarf-2"

# Enable parallel compiling
mk_add_options MOZ_MAKE_FLAGS="-j4"

# Needed to enable breakpad in application.ini
export MOZILLA_OFFICIAL=1

moz2-darwin10-slave10:~ cltbld$ cat /builds/slave/m-cen-osx-dbg/build/build/macosx/mozconfig.leopard 
if test -z "$CC" ; then
   CC=gcc-4.2
fi

if test -z "$CXX" ; then
   CXX=g++-4.2
fi

# We do 32 bit builds for leopard
CC="$CC -arch i386"
CXX="$CXX -arch i386"

ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk

Comment 30

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Pass --target to configure too

It looks like that with clang or the gcc in xcode 4, just passing -isysroot (--with-macos-sdk in configure) is sufficient for gcc to know it is targeting an older version of OS X.

With the gcc-4.2 in the xcode included in 10.6, passing just -isysroot causes gcc to try to like with the 10.6 crt files which causes every configure test to fail.

Adding -target=i386-apple-darwin$DARWIN_VERSION (like mozconfig.common does), fixes the problem.

Comment 31

[Armen [:armenzg]]

Comment on attachment 552736 [details] [diff] [review]
Pass --target to configure too

ted, do you see any problem with this 2 line change?

Comment 32

[(not currently active) Ted Mielczarek]

Comment on attachment 552736 [details] [diff] [review]
Pass --target to configure too

Review of attachment 552736 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/macosx/mozconfig.leopard
@@ +10,5 @@
>  CC="$CC -arch i386"
>  CXX="$CXX -arch i386"
>  
> +DARWIN_VERSION=`uname -r`
> +ac_add_options --target=i386-apple-darwin$DARWIN_VERSION

Is the DARWIN_VERSION bit necessary? I don't think we actually use the kernel version in the configure target anywhere, do we?

Comment 33

[(not currently active) Ted Mielczarek]

Oh, I guess that was the whole point of the patch. That's interesting, what does that actually change in configure?

Comment 34

[(not currently active) Ted Mielczarek]

*sigh*, I should think before commenting in the morning. I presume the only thing you actually need there is:
ac_add_options --target=i386-apple-darwin

because GCC on 10.6 targets 64-bit by default.

Comment 35

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: don't include the kernel version

I used an equivalent line to the one in universal/mozconfig.common. This patch removes the kernel version from both. I will test it once I get in the office (the 10.6 machine went to sleep :-( )

Comment 36

[Daniel Holbert [:dholbert] (PTO until 2/24)]

This landed earlier today as:
  http://hg.mozilla.org/integration/mozilla-inbound/rev/7369baed29ef
but I backed it out:
  http://hg.mozilla.org/integration/mozilla-inbound/rev/4e895b191eb2
because it appears to have caused perma-orange in OS X 64-bit xpcshell, with a crash in test_jsctypes.js.

For example:
http://tbpl.allizom.org/php/getParsedLog.php?id=5981245
{
PROCESS-CRASH | /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/ctypes/tests/unit/test_jsctypes.js | application crashed (minidump found)
Crash dump filename: /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/ctypes/tests/unit/CC8A85D6-A9C9-437F-9839-6773BF37B3F7.dmp
/var/www/html/stackwalker/minidump_stackwalk /tmp/tmpjt6MGV /builds/stackwalker_symbols/2cebe01fbf4bed6b837eabafbfaa72d5
Operating system: Mac OS X
                  10.6.6 10J567
CPU: amd64
     family 6 model 23 stepping 10
     2 CPUs

Crash reason:  EXC_BAD_ACCESS / KERN_PROTECTION_FAILURE
Crash address: 0x80bd180

Thread 0 (crashed)
 0  0x1080bd180
    rbx = 0x0000000000000001   r12 = 0x0000000000000004
    r13 = 0x0000000000000001   r14 = 0x00000001080bc0d8
    r15 = 0x00007fff5fbfcd00   rip = 0x00000001080bd180
    rsp = 0x00007fff5fbfccf8   rbp = 0x00007fff5fbfcd00
    Found by: given as instruction pointer in context
 1  XUL!ffi_call [ffi64.c:01ce7e95d7b7 : 485 + 0x2d]
    rip = 0x000000010134f034   rsp = 0x00007fff5fbfcd20
    Found by: stack scanning
 2  XUL!js::ctypes::FunctionType::Call [CTypes.cpp:01ce7e95d7b7 : 5074 + 0x1c]
    rip = 0x00000001013422df   rsp = 0x00007fff5fbfce10
    Found by: stack scanning
 3  XUL!js::Invoke [jscntxtinlines.h:01ce7e95d7b7 : 281 + 0x7]
    rbx = 0x00000001056333f0   r12 = 0x0000000000000001
    r13 = 0x00007fffffffffff   r14 = 0x0000000105633380
    r15 = 0x00007fff5fbfd0e0   rip = 0x000000010113ffb7
    rsp = 0x00007fff5fbfd030   rbp = 0x00000001080bc5d0
    Found by: call frame info
 4  XUL!js::ExternalInvoke [jsinterp.h:01ce7e95d7b7 : 169 + 0xc]
    rbx = 0x00000001080bc5d0   r12 = 0x0000000107f9e758
    r13 = 0x0000000105633378   r14 = 0x0000000105633380
    r15 = 0x00007fff5fbfd0e0   rip = 0x00000001011407aa
    rsp = 0x00007fff5fbfd0d0   rbp = 0x0000000000000001
    Found by: call frame info
 5  XUL!js::JSProxyHandler::call [jsproxy.cpp:01ce7e95d7b7 : 279 + 0x18]
    rbx = 0x0000000105633370   r12 = 0x00000001080bc5d0
    r13 = 0x0000000000000001   r14 = 0x0000000107f9e6f8
    r15 = 0x0000000000000001   rip = 0x00000001011aadd6
    rsp = 0x00007fff5fbfd150   rbp = 0x0000000000000001
    Found by: call frame info
 6  XUL!JSCrossCompartmentWrapper::call [jswrapper.cpp:01ce7e95d7b7 : 272 + 0x17]
[...]
}

Comment 37

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Thanks. I am trying to reproduce it. Another thing I noticed from the non clobbered builds is that the bots are detected as really old versions of OS X:

configure: error: `host_alias' has changed since the previous run:
configure:   former value:  `i386-apple-darwin10.2.0'
configure:   current value: `i386-apple-darwin'

Comment 38

[(not currently active) Ted Mielczarek]

libffi's configure is really sensitive to the host triplet. It requires a clobber every time you install an OS update that increments the minor version number.

Comment 39

[Phil Ringnalda (:philor)]

And that's not a really old version, it's the really annoying Darwin numbering, where darwin10.2.0 == OS X 10.6.2.

Comment 40

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I found the problem, it is this chunk in configure.in:

case "$target" in
     *-apple-darwin10* | *-*-freebsd* | *-*-openbsd* | *-pc-solaris*)
       AC_DEFINE(FFI_MMAP_EXEC_WRIT, 1,
                 [Cannot use malloc on this target, so, we revert to
                   alternative means])
     ;;
esac

which doesn't match "x86_64-apple-darwin". I will upload an updated patch with just the major version.

Comment 41

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: updated patch with the major version in the --target

Comment 42

[John Ford [:jhford] CET/CEST Berlin Time]

I beleive this bug debugs on bug 679825

Comment 43

[John Ford [:jhford] CET/CEST Berlin Time]

I believe this bug debugs on bug 679825

Comment 44

[Armen [:armenzg]]

I hope I followed correctly your instructions from IRC but the build failed.
respindola would it be better if you loan a 10.6 machine from us and figure out the right mozconfig with our set of tools?

I tried running it with this mozconfig:
# Don't use the standard mozconfig. We don't want universal for a debug build. 
#. $topsrcdir/build/macosx/universal/mozconfig

if test -z "$CC" ; then
   CC=gcc-4.2
fi

if test -z "$CXX" ; then
   CXX=g++-4.2
fi

# We do 32 bit builds for leopard
CC="$CC -arch i386"
CXX="$CXX -arch i386"

# Note, the version (10) is usued by libffi's configure.
ac_add_options --target=i386-apple-darwin10

ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.5.sdk

ac_add_options --disable-optimize
ac_add_options --enable-debug
ac_add_options --enable-libxul

ac_add_options --enable-application=browser
ac_add_options --enable-tests
ac_add_options --enable-trace-malloc

# For NSS symbols
export MOZ_DEBUG_SYMBOLS=1
ac_add_options --enable-debug-symbols="-gdwarf-2"

# Enable parallel compiling
mk_add_options MOZ_MAKE_FLAGS="-j4"

# Needed to enable breakpad in application.ini
export MOZILLA_OFFICIAL=1


and I got this:
######################
gcc-4.2 -arch i386 -o xpidl_java.o -c -fvisibility=hidden -DOSTYPE=\"Darwin10.2.0\" -DOSARCH=Darwin  -I/builds/slave/m-cen-osx-dbg/build/xpcom/typelib/xpidl -I. -I../../../dist/include -I../../../dist/include/nsprpub  -I/builds/slave/m-cen-osx-dbg/build/obj-firefox/dist/include/nspr -I/builds/slave/m-cen-osx-dbg/build/obj-firefox/dist/include/nss       -fPIC -Wall -W -Wno-unused -Wpointer-arith -Wdeclaration-after-statement -Wcast-align -W -fno-strict-aliasing -fno-common -pthread -DNO_X11 -pipe  -DDEBUG -D_DEBUG -DTRACING -gdwarf-2 -I/opt/local/include/libIDL-2.0 -I/opt/local/include/glib-2.0 -I/opt/local/lib/glib-2.0/include -I/opt/local/include     -include ../../../mozilla-config.h -DMOZILLA_CLIENT -MD -MF .deps/xpidl_java.pp /builds/slave/m-cen-osx-dbg/build/xpcom/typelib/xpidl/xpidl_java.c
/builds/slave/m-cen-osx-dbg/build/obj-firefox/config/nsinstall -D ../../../dist/sdk/bin
/tools/buildbot/bin/python2.6 /builds/slave/m-cen-osx-dbg/build/config/pythonpath.py -I../../../config /builds/slave/m-cen-osx-dbg/build/config/expandlibs_exec.py --uselist --  gcc-4.2 -arch i386 -o xpidl -Wall -W -Wno-unused -Wpointer-arith -Wdeclaration-after-statement -Wcast-align -W -fno-strict-aliasing -fno-common -pthread -DNO_X11 -pipe  -DDEBUG -D_DEBUG -DTRACING -gdwarf-2 -I/opt/local/include/libIDL-2.0 -I/opt/local/include/glib-2.0 -I/opt/local/lib/glib-2.0/include -I/opt/local/include   xpidl.o xpidl_idl.o xpidl_util.o xpidl_header.o xpidl_typelib.o xpidl_doc.o xpidl_java.o   -framework Cocoa -lobjc  -framework ExceptionHandling -Wl,-executable_path,/builds/slave/m-cen-osx-dbg/build/obj-firefox/dist/bin -L../../../dist/bin -L../../../dist/lib ../../../dist/lib/libxpt.a -L/opt/local/lib -lIDL-2 -lgobject-2.0 -lglib-2.0 -lintl -liconv      
ld: warning: in /opt/local/lib/libIDL-2.dylib, file is not of required architecture
ld: warning: in /opt/local/lib/libgobject-2.0.dylib, file is not of required architecture
ld: warning: in /opt/local/lib/libglib-2.0.dylib, file is not of required architecture
ld: warning: in /opt/local/lib/libintl.dylib, file is not of required architecture
ld: warning: in /opt/local/lib/libiconv.dylib, file is not of required architecture
Undefined symbols:
  "_g_slist_last", referenced from:
      _add_deprecated in xpidl_java.o
  "_IDL_tree_property_get", referenced from:
...
ld: symbol(s) not found
collect2: ld returned 1 exit status
make[8]: *** [xpidl] Error 1
make[7]: *** [export] Error 2
make[6]: *** [export] Error 2
make[5]: *** [export] Error 2
make[4]: *** [export_tier_platform] Error 2
make[3]: *** [tier_platform] Error 2
make[2]: *** [default] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2

Comment 45

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #44)
> I hope I followed correctly your instructions from IRC but the build failed.
> respindola would it be better if you loan a 10.6 machine from us and figure
> out the right mozconfig with our set of tools?

It would be awesome to be able to loan one. Are you in the office today?

> ld: warning: in /opt/local/lib/libIDL-2.dylib, file is not of required
> architecture


I normally get that when I install IDL with

$ sudo port install libidl

instead of

$ sudo port install libidl +universal

Can you run "file /opt/local/lib/libIDL-2.dylib" both in the machine you used for testing and in a regular build bot? Thanks.

Comment 46

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

On my machine I get:

/opt/local/lib/libIDL-2.dylib: Mach-O universal binary with 2 architectures
/opt/local/lib/libIDL-2.dylib (for architecture i386):	Mach-O dynamically linked shared library i386
/opt/local/lib/libIDL-2.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64

Comment 47

[(not currently active) Ted Mielczarek]

Comment on attachment 553512 [details] [diff] [review]
updated patch with the major version in the --target

Review of attachment 553512 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/macosx/mozconfig.leopard
@@ +10,4 @@
>  CC="$CC -arch i386"
>  CXX="$CXX -arch i386"
>  
> +# Note, the version (10) is usued by libffi's configure.

spelling nit: "used".

Comment 48

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #46)
> On my machine I get:
> 
> /opt/local/lib/libIDL-2.dylib: Mach-O universal binary with 2 architectures
> /opt/local/lib/libIDL-2.dylib (for architecture i386):	Mach-O dynamically
> linked shared library i386
> /opt/local/lib/libIDL-2.dylib (for architecture x86_64):	Mach-O 64-bit
> dynamically linked shared library x86_64

I just tried this on a bot and it is a 64 bit only file. My guess is that we have to set HOST_CC, but I don't know why that never worked for me with an non-universal idl. Debugging.

Comment 49

[Marco Bonardo [:mak]]

http://hg.mozilla.org/mozilla-central/rev/5bfb2f2d1464

I'm unsure if this should be resolved, looks like not.

Comment 50

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Build as a cross compilation

The problem was really that the bots have only 64 bit versions of some libraries, so we have to do a cross compile "for real"

The attached patch fixes it.

Comment 51

[Armen [:armenzg]]

Comment on attachment 555441 [details] [diff] [review]
Build as a cross compilation

Feel free to land with DONTBUILD in your comment.

Comment 52

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Use build/macosx/mozconfig.leopard

The updated mozconfig.leopard is on inbound, so it should be possible to test this now.

Comment 53

[Matt Brubeck (:mbrubeck)]

http://hg.mozilla.org/mozilla-central/rev/45bce33d2bd7

Comment 54

[Ed Morley [:emorley]]

Reopened for remaining patch (comment 52).

Comment 55

[Armen [:armenzg]]

It seems that compilation passed on moz2-darwin10-slave02 but it failed in make package.

> /bin/sh: 1: command not found
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1314289111.1314293988.20217.gz&fulltext=1

On the other hand, here is a log to compare against for a darwin9 slave doing a leak test build:
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1314288788.1314294903.23034.gz&fulltext=1

FTR this is the change that I tested:
http://hg.mozilla.org/users/armenzg_mozilla.com/buildbot-configs/rev/0ff0e62a1058

Comment 56

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Add an explicit MOZ_CAN_RUN_PROGRAMS variable

The "1" is coming from GENERATE_CACHE=1 and we deciding that we cannot run programs and therefore not updating it to an actual command.

This patch defines a new variable MOZ_CAN_RUN_PROGRAMS which is set automatically for the trivial cases but can be set manually for the x86_64 -> i686 cross compilation case.

A try of just this patch is at

http://tbpl.allizom.org/?usebuildbot=1&tree=Try&rev=f014fe6bcba8

I am also building on a bot with my modified mozconfig so that I can test the combination.

Comment 57

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: set MOZ_CAN_RUN_PROGRAMS

With this patch (and the one I posted before) I was able to build a .dmg in the machine you assigned to me.

Comment 58

[Armen [:armenzg]]

Comment on attachment 555831 [details] [diff] [review]
Add an explicit MOZ_CAN_RUN_PROGRAMS variable

I really don't much about the build system.
Passing it to ted and he can determine if he has time for it or someone else can review it.

Comment 59

[(not currently active) Ted Mielczarek]

Comment on attachment 555942 [details] [diff] [review]
set MOZ_CAN_RUN_PROGRAMS

Review of attachment 555942 [details] [diff] [review]:
-----------------------------------------------------------------

This doesn't look like the correct patch...

Comment 60

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: set MOZ_CAN_RUN_PROGRAMS

Comment 61

[(not currently active) Ted Mielczarek]

Comment on attachment 556111 [details] [diff] [review]
set MOZ_CAN_RUN_PROGRAMS

Review of attachment 556111 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/macosx/mozconfig.leopard
@@ +20,4 @@
>  LD=ld
>  STRIP="strip -x -S"
>  
> +MOZ_CAN_RUN_PROGRAMS=1

I don't think this is going to do anything for you. The mozconfig is sourced as a shell script twice: once during configure, and once in client.mk. You're not saving this variable anywhere, so it won't exist by the time you get to packager.mk.

Comment 62

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

> > +MOZ_CAN_RUN_PROGRAMS=1
> 
> I don't think this is going to do anything for you. 

I does, I am not sure what the path is, but is is available. Note that it is similar to UNIVERSAL_BINARY which is set is mozconfig.common and used in  packager.mk.

Comment 63

[Marco Bonardo [:mak]]

backed out from inbound investigating a permaorange leak in Windows reftests. While it may appear strange both changesets in the range (this included) looked unrelated, but the backout fixed the leak, so please get a try run of this to ensure Windows XP Debug Reftests is green before pushing.

Comment 64

[(not currently active) Ted Mielczarek]

UNIVERSAL_BINARY gets handled in configure.in, though:
http://mxr.mozilla.org/mozilla-central/source/configure.in#1080
http://mxr.mozilla.org/mozilla-central/source/config/autoconf.mk.in#660

The usage in packager.mk is referencing the variable that's set in autoconf.mk.

Comment 65

[Ed Morley [:emorley]]

Comment on attachment 555831 [details] [diff] [review]
Add an explicit MOZ_CAN_RUN_PROGRAMS variable

http://hg.mozilla.org/mozilla-central/rev/14271cf8f6f2

Comment 66

[Ed Morley [:emorley]]

Not sure what left to do here, so leaving open for now. Please close if appropriate.

Comment 67

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: updated patch

I have no idea how make package worked for me with the previous patch. In any case, I now did a "hg clean --all", applied this patch and was able to, in a 10.6 machine, build a package using the 10.5 sdk.

Comment 68

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Comment on attachment 558306 [details] [diff] [review]
updated patch

I think we should call this UNIVERSAL_BINARY_CAN_RUN_PROGRAMS instead, because this doesn't handle the generic cross-compilation case.

r=me with that

Comment 69

[Ed Morley [:emorley]]

Comment on attachment 558306 [details] [diff] [review]
updated patch

https://hg.mozilla.org/mozilla-central/rev/90de3b7ddca9
(I'm presuming this changeset was for this patch).

For bugs with multiple patches, please could you use the per patch checkin?/+/- flags and also in the reply with the inbound changeset rev (which appears to be missing here) annotate which patch it applies to. Otherwise merges take a lot longer, due to having to sift through all attached patches to work out what is going on.

For more info, see the recently updated https://wiki.mozilla.org/Inbound_Sheriff_Duty#Please_do_the_following_after_landing_on_mozilla-inbound

Thanks :-)

Comment 70

[Ed Morley [:emorley]]

(Leaving bug open since patches with pending reviews)

Comment 71

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Cool. Armen, can you try this on your test setup? Let me know if there is something else I need to do.

Comment 72

[Armen [:armenzg]]

I am doing a release at the moment but will get to it as soon as I have an opening.

Comment 73

[Armen [:armenzg]]

I booked moz2-darwin10-slave01 to trigger a 10.5 debug build.
I will report tomorrow as it is still running the previous job.

Comment 74

[Armen [:armenzg]]

respindola, how does this look?
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1316734982.1316746212.27926.gz&fulltext=1

I will upload the builds to my people account.
You will see them under here when they are ready:
http://people.mozilla.com/~armenzg/builds/bug674647-respindola

I have also triggered unittests for the build with:
buildbot sendchange --master dev-master01.build.mozilla.org:9041 --username sendchange-unittest --branch mozilla-central-macosx-debug-unittest --revision 45db8c3384e1 --property buildid:20110922164315 --property builduid:e84bff1b6d6e41e0811efa40c5cebb57 http://dev-stage01.build.sjc1.mozilla.com/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1316734995/firefox-9.0a1.en-US.mac.dmg http://dev-stage01.build.sjc1.mozilla.com/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1316734995/firefox-9.0a1.en-US.mac.tests.zip

Comment 75

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I can give them a try on an old 10.5 machine on Monday. Is there anything else I should look for? What would be the next step?

Comment 76

[Armen [:armenzg]]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #75)
> I can give them a try on an old 10.5 machine on Monday. Is there anything
> else I should look for? What would be the next step?

advertise, advertise and perhaps advertise. As soon as things look good we can make the change happen. I will let my people know.

I also have to write a patch to ensure that it won't affect releases and older branches.

Can you please verify that the mozconfig is what you want?

Comment 77

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

> advertise, advertise and perhaps advertise. As soon as things look good we
> can make the change happen. I will let my people know.

I have email dev-planning and I have been to weekly meeting. What else is needed? Remember that this bug just changes the host being used to make the build.

> I also have to write a patch to ensure that it won't affect releases and
> older branches.
> 
> Can you please verify that the mozconfig is what you want?

The mozconfig in the log you posted looks reasonable. It matches what is done in 

https://bugzilla.mozilla.org/attachment.cgi?id=555485&action=diff

Comment 78

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

> I will upload the builds to my people account.
> You will see them under here when they are ready:
> http://people.mozilla.com/~armenzg/builds/bug674647-respindola

I gave it a try on 10.5 and it looks like it is working correctly.

Comment 79

[Armen [:armenzg]]

I found the thread and replied to it as we need fresh attention. We want everyone to be on the loop.

BTW I have found a LEAK on mochitest-other, would you be able to have a look at it?
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1316813420.1316817806.30862.gz&fulltext=1#err0

Meanwhile I have re-triggered another mochitest-other job.

Comment 80

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

It looks like: https://bugzilla.mozilla.org/show_bug.cgi?id=687857

Comment 81

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

OK. You replied to it, everyone is on the loop. What should I do next?

Comment 82

[Armen [:armenzg]]

I don't understand what is going on staging. The build has gone orange [1] and the tests have gone red [2].

I have triggered a new set.

I just realized that my post was on dev.platform which is not as wide spread.
I have now posted on dev.planning (https://groups.google.com/d/topic/mozilla.dev.planning/4sANJZlal1c/discussion).

Feel free to check http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest if you want to check the results of the new triggered set.

[1]
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1317132832.1317138964.24546.gz&fulltext=1

The following step is what is failing:
python leaktest.py -- --trace-malloc malloc.log --shutdown-leaks=sdleak.log

[2]
http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest&maxdate=1317110050&hours=24&legend=0&norules=1

TEST-UNEXPECTED-FAIL | automation.py | application timed out after 330 seconds with no output
Can't trigger Breakpad, just killing process
INFO | automation.py | Application ran for: 0:07:52.994201
INFO | automation.py | Reading PID log: /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmp6WNs8hpidlog
TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | missing output line for total leaks!

Comment 83

[Phil Ringnalda (:philor)]

Welcome to the wonderful world of bug 573631 and bug 581077 - we don't do trace-malloc on 10.6, because it's too slow and nobody really cares enough to have ever particularly tried to fix it.

Comment 84

[Armen [:armenzg]]

Build to build (10.6 leak vs 10.5 leak) differ in that after doing the "alive_test1" and "make check" we have a bunch of small tests that I assume they are related to "alive_test2" (excepting macosx64 as a platform is done in factory.py [1].

On a darwin9 machine the alive_test2 takes around 1 min while on a darwin10 machine takes around 25-30mins if it does not timeout.

trace mallok got disabled on bug 573631 and no progress has been done since then.
Disabling the leak test suite was discussed in https://groups.google.com/forum/#!topic/mozilla.dev.platform/0lkjbtBK8eQ

If we do switch to do 10.5 leak builds on 10.6 machines we are going to resuscitate the intermittent orange (bug 558496) but being macosx-debug rather than macosx64-debug.

respindola and philor I propose we ask in dev.platform that we want to disable it and anyone objecting should be fixing either bug 573631 or find a replacement for this suite (that has been long being ignored) as it does not report on dev.tree-management.

Makes sense?

[1] hg.mozilla.org/build/buildbotcustom/file/23247502024e/process/factory.py#l1112

Comment 85

[Phil Ringnalda (:philor)]

Desperately needs a clear and precise resummarization of why we might want to do this. After reading through this bug and bug 674655, I know what we're being asked to give up, any chance we have of noticing when someone adds new Foo without delete in Mac-specific code, but I don't have any idea what we're getting in return.

Is it "when at some point in the future we want to add code using 10.7 APIs, we will have to create a new refplatform for our builders, which will have to use a new version of XCode, which will not have the 10.5 SDK"? Or is it "developers using machines which shipped with Lion shouldn't have to navigate the morass of installing an unsupported XCode 3 to build with the same mozconfig that we use for buildbot"?

Comment 86

[Armen [:armenzg]]

respindola is bug 573631 something you can help with?

philor IIUC we are two SDKs behind. There are many bugs fixes in the newer ones and even APIs needed for features wanted to use.

Comment 87

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

There are a few reasons to move forward:

* No single SDK has 10.5, 10.6 and 10.7
* Machines running 10.5 are going to get harder and harder to get, better use them to test on 10.5 and build on easier to get 10.6 ones.
* Easier to reproduce the build. I had at least one issue I had to fetch a 10.5 machine to reproduce.

Unfortunately I had no time to look at why trace malloc is so slow in 10.6, so I don't have an opinion right now at what we should do. I am currently debugging a more pressing issue on windows, but will get back to this.

Comment 88

[Armen [:armenzg]]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #87)
> 
> Unfortunately I had no time to look at why trace malloc is so slow in 10.6,
> so I don't have an opinion right now at what we should do. I am currently
> debugging a more pressing issue on windows, but will get back to this.

Perfect. Let me know once you get back to it.

Comment 89

[Armen [:armenzg]]

Once bug 696376 is on production I believe we won't have the slowness on 10.6.

I believe we then have to test once more on staging with a 10.6 slave and see if there is no surprises.

We don't depend on bug 573631 and bug 679825 anymore, correct?

Comment 90

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I don't think we depend on 679825 and 573631 is probably fixed.

Comment 91

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Btw, is
https://bug674647.bugzilla.mozilla.org/attachment.cgi?id=555485

OK? I should not change the behavior when running on 10.5, so it should be safe for it to go in before we switch to building on 10.6.

Comment 92

[Armen [:armenzg]]

Comment on attachment 555485 [details] [diff] [review]
Use build/macosx/mozconfig.leopard

Ah! this patch, if you tried this on the try server feel free to land it at any time (I understand that it could go now before switching to building on 10.6).

Comment 93

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I just pushed to try a much simpler version thanks to the mozconfigs being in tree :-)

https://tbpl.mozilla.org/?tree=Try&rev=943136201616

Comment 94

[Ed Morley [:emorley]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/96a33324e54d

Moving to Core::Build Config to get a milestone field.

Note: For whomever merges to mozilla-central, OS X will need a clobber.

Comment 95

[Mark Banner (:standard8)]

I may be misunderstanding here, but if we end up running the 32-bit leak tests on the 10.6 builders, then isn't that one of the configurations we were explicitly not-supporting when we decided to go for 32/64 bit universal binaries? I remember the thread on .planning which we said we'd only support running 32 bit on 10.5 and 64 bit on 10.6, and not 32 bit on 10.6 (even though it is theoretically possible).

Now this is only debug builds and looks like its being done for operational reasons, but it does seem a bit strange to moving to testing something that is effectively unsupported (which is why I'd have expected a universal debug build which then is tested on both 10.5 & 10.6).

Comment 96

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

With the fixes already in place, we should be able to run leak tests on 10.6 (and eventually 10.7) too.

Comment 97

[Armen [:armenzg]]

Even for end-to-end times purposes I would rather have the 10.6 machines do two leak builds rather than one universal leak build since the former takes longer to build.

Comment 98

[Marco Bonardo [:mak]]

https://hg.mozilla.org/mozilla-central/rev/96a33324e54d

Comment 99

[Armen [:armenzg]]

Thanks for landing the patches but those are preparatory patches.
We have not yet switched doing 10.5 leak builds on 10.6.

Should we remove the mozilla10 target until we actually do the switch?

Comment 100

[Ed Morley [:emorley]]

Yeah good point, I've removed the target for now.

To avoid bugs being closed prematurely in the future, whiterboard annotations (along the lines of https://wiki.mozilla.org/Tree_Rules/Inbound#Please_do_the_following_after_pushing_to_inbound) can be left for people doing the merges :-)

Comment 101

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I am investigating two possibilities. One is that the stack walking itself is causing the orange on 64 bits. If that is the case, this should work:

https://tbpl.mozilla.org/?tree=Try&rev=69714951bd04

Another option is that our hack for finding out when new_sem_from_pool calls realloc needs to be present also on the 64 bit stack walking.

Comment 102

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Armen, can you try this again? I think all the dependencies have finally been fixed. Let me know if there anything else I need to do in this bug.

Comment 103

[Armen [:armenzg]]

Added to my TODO. I hope to get some results this week.

Comment 104

[Armen [:armenzg]]

On bug 698827 we're working on the last release engineering steps to run the 10.5 leak builds on the 10.6 machines.

Comment 105

[Armen [:armenzg]]

What is left on this bug after bug 698827?

Should we use this bug for the mozilla-aurora approvals?

Comment 106

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I would vote to close it. It is already very noisy.

I opened 719499 for the aurora backport. Will add a patch as soon and hg finishes cloning the tree.

Comment 107

[Armen [:armenzg]]

Thanks espindoal. It works for me.