410637 - Mac UB XULRunner SDK packages have empty bin directory

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Matthew Gertner]

I downloaded a XULRunner SDK nightly for Mac from the FTP server (specifically http://ftp.mozilla.org/pub/mozilla.org/xulrunner/nightly/latest-trunk/xulrunner-1.9b3pre.en-US.mac-pkg.sdk.tar.bz2). When I unpacked it, the bin directory under the root was empty.

Comment 1

[Matthew Gertner]

Also, this may or may not be related, but I get the following error if I copy over the bin/ files manually:

/usr/bin/ld: warning /Users/Matt/xulrunner-sdk/lib/libxpcomglue_s.a archive's cputype (18, architecture ppc) does not match cputype (7) for specified -arch flag: i386 (can't load from it)

Maybe the libraries aren't i386 compatible?

Comment 2

[Matthew Gertner]

Also XUL.framework doesn't appear to be in there.

Comment 3

[Chris Cooper [:coop] (he/him)]

We recently had to change build machines for XULRunner due to a hardware failure. It's quite possible that there is something wrong with the configuration of the new machine.

Do you have a regression range for the when the bin/ directory was last populated correctly?

Comment 4

[Matthew Gertner]

I don't personally, no. The first SDK I ever downloaded is referenced in https://bugzilla.mozilla.org/show_bug.cgi?id=410637#c0.

Comment 5

[Chris Cooper [:coop] (he/him)]

(In reply to comment #4)
> I don't personally, no. The first SDK I ever downloaded is referenced in
> https://bugzilla.mozilla.org/show_bug.cgi?id=410637#c0.

That link points to the latest-trunk directory which gets overwritten with the latest build every night. One of the date-stamped directories under http://ftp.mozilla.org/pub/mozilla.org/xulrunner/nightly/ is what I was hoping for.

Comment 6

[Chris Cooper [:coop] (he/him)]

The oldest Mac SDK nightly on ftp right now is from 2007-11-22, and it's bin/ dir is empty as well. This looks like a larger packaging problem.

Comment 7

[Matthew Gertner]

I'm going to take a crack at this. If anyone has ideas about what the problem is, please let me know.

Comment 8

[Matthew Gertner]

Attachment: Copy dist/bin/ directory into staging directory when building SDK

packager.mk is used for several purposes. It looks like the part that copies dist/bin/ into the staging directory for packaging into the SDK is broken. There is something there to copy _APPNAME into the staging directory when UNIVERSAL_BINARY is not set, and since when building the SDK _APPNAME is set to something weird, I'm actually not sure why the build doesn't simply break when the UB make files try to make the SDK for i386 and ppc without UNIVERSAL_BINARY set.

Nonetheless, here's my first crack at a solution. Can someone tell me exactly how the tinderbox version is built so I can test locally? Building the SDK individually in i386 and ppc now works fine (as long as UNIVERSAL_BINARY is not set), but this isn't entirely reassuring since, as I mention, I don't understand how the build is even completing right now.

Comment 9

[Matthew Gertner]

Comment on attachment 304951 [details] [diff] [review]
Copy dist/bin/ directory into staging directory when building SDK

Shouldn't affect anything except the Mac XR SDK.

Comment 10

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 304951 [details] [diff] [review]
Copy dist/bin/ directory into staging directory when building SDK

a1.9=beltzner

Comment 11

[Dave Townsend [:mossop]]

Checking in toolkit/mozapps/installer/packager.mk;
/cvsroot/mozilla/toolkit/mozapps/installer/packager.mk,v  <--  packager.mk
new revision: 1.54; previous revision: 1.53
done

Comment 12

[Dave Townsend [:mossop]]

The nightly sdks still have an empty bin directory, reopening

Comment 14

[Dave Townsend [:mossop]]

Attachment: Fix sdk packaging in UB setup

This is probably not quite ready yet and I haven't tested that other apps still build right with it.

This simply unsets UNIVERSAL_BINARY for the stage-package pass so we properly copy stuff into sdk-stage. This will let the tinderbox correctly build either a ppc or an i386 sdk after a UB xulrunner build. Might want to modify the tarball name for this.

I've also added in dist/host/bin to the sdk under host/bin. It is here that xpidl is found for the ppc pass as host_xpidl so I think it is necessary. Perhaps some of the other things aren't there though I'm not sure.

flight.mk contains a couple of tweaks to make it work right for apps built against the sdk. It assumes they have built themselves an app bundle and copied the <sdk>/bin into Contents/Frameworks/XUL.Framework of the bundle. postflight will then fix the buildconfig and clear out the .chk files and unify correctly.

This patch and a couple of lines changed in McCoy's makefiles were all that were necessary to make a UB of McCoy against the two resulting sdks.

Comment 15

[Matthew Gertner]

My hero!

Comment 16

[Dave Townsend [:mossop]]

Attachment: Fix sdk packaging

Two parts to this patch, this is to the main build system.

flight.mk is fixed to attempt to find the toolkit.jar file and the nss chk files regardless of the build type.

The sdk package name is changed on mac to include the processor. I couldn't find a better way to do this but it generates names like:

xulrunner-1.9pre.en-US.mac-i386.sdk.tar.bz2
xulrunner-1.9pre.en-US.mac-powerpc.sdk.tar.bz2

rather than the current:

xulrunner-1.9pre.en-US.mac-pkg.sdk.tar.bz2

The dist/host/bin directory is included in the sdk such that the powerpc sdk includes host/bin/xpidl in it.

The sdk is no longer build at the same time as the installer. The installer will build a UB version of xulrunner, then separately we must build the ppc and i386 sdks. I couldn't see an easier way to do that. Next patch makes tinderbox perform that additional step.

Comment 17

[Dave Townsend [:mossop]]

Attachment: Make tinderbox build sdk

This adds code to tinderbox so that if configured it will perform an sdk build for the objdir, or for both objdirs in a UB case.

Comment 18

[Dave Townsend [:mossop]]

Comment on attachment 314348 [details] [diff] [review]
Make tinderbox build sdk

This adds an additional build step post build which makes the sdk and copies it to the packages directory. In a unversal build case it repeats this step for the i386 objdir.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 314348 [details] [diff] [review]
Make tinderbox build sdk

A bit painful to be duplicating crappy code, but at least it's self contained.

Comment 20

[Dave Townsend [:mossop]]

Comment on attachment 314347 [details] [diff] [review]
Fix sdk packaging

Requesting approval to land this for 1.9. This makes it possible to generate working gecko sdks for Mac. The only impact to Firefox is some changes to the makefiles that performs the final unification on Mac. There should be no chance of any hidden regressions from this, if t were wrong the Firefox build would just fail. I have of course tested this myself and seen no issue however.

Comment 21

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 314347 [details] [diff] [review]
Fix sdk packaging

a1.9=beltzner

Comment 22

[Dave Townsend [:mossop]]

Landed

Checking in build/macosx/universal/flight.mk;
/cvsroot/mozilla/build/macosx/universal/flight.mk,v  <--  flight.mk
new revision: 1.9; previous revision: 1.8
done
Checking in toolkit/mozapps/installer/packager.mk;
/cvsroot/mozilla/toolkit/mozapps/installer/packager.mk,v  <--  packager.mk
new revision: 1.58; previous revision: 1.57
done
Checking in xulrunner/installer/Makefile.in;
/cvsroot/mozilla/xulrunner/installer/Makefile.in,v  <--  Makefile.in
new revision: 1.15; previous revision: 1.14
done

Checking in tinderbox/post-mozilla-rel.pl;
/cvsroot/mozilla/tools/tinderbox/post-mozilla-rel.pl,v  <--  post-mozilla-rel.pl
new revision: 1.145; previous revision: 1.144
done
Checking in tinderbox/tinder-defaults.pl;
/cvsroot/mozilla/tools/tinderbox/tinder-defaults.pl,v  <--  tinder-defaults.pl
new revision: 1.127; previous revision: 1.126
done
Checking in tinderbox-configs/xulrunner/linux/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/xulrunner/linux/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.9; previous revision: 1.8
done
Checking in tinderbox-configs/xulrunner/macosx/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/xulrunner/macosx/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.7; previous revision: 1.6
done
Checking in tinderbox-configs/xulrunner/win32/tinder-config.pl;
/cvsroot/mozilla/tools/tinderbox-configs/xulrunner/win32/tinder-config.pl,v  <--  tinder-config.pl
new revision: 1.12; previous revision: 1.11
done