324855 - Need a script to produce universal binary .app from cpu-specific bits

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Mark Mentovai]

Working on this, making progress, almost done.

Comment 1

[Mark Mentovai]

Attachment: unify (draft)

I haven't finished documenting/commenting, and there are a couple more things I want to add, but this script is functionally complete and I'm now running the same unifox.app it produced on both ppc and x86.

Comment 2

[Mark Mentovai]

Attachment: fix-buildconfig

This is a small helper script I wrote to gracefully handle the about:buildconfig situation for universal builds.  It takes two arguments, each a path to a toolkit.jar file.  toolkit/content/buildconfig.html is extracted from each, their contents are somewhat intelligently glued together with an <hr> separator, and the zip files are rewritten with the new buildconfig.

Comment 3

[Mark Mentovai]

unify runs lipo on Mach-O files and compares non-Mach-O files a la diff.  If two zip files (including jars) fail the diff check, not all is doomed: the zips' directories are scrutinized and CRCs compared.

Using ppc and x86 builds produced using the cross build system under development in bug 322578, the following items fail the diff check:

 - toolkit/content/buildconfig.html in chrome/toolkit.jar.  fix-buildconfig
   (attached) accounts for this.
 - On the branch, components/nsExtensionManager.js.  This was fixed in
   bug 323330, but hasn't yet made it to the branch.

Additionally, the lipo merge fails for the following item:

 - JEP.  Since JEP is a precompiled binary in our tree, we'll need to check in
   a fat JEP.  Until then, even native x86 builds are produced with a ppc
   version of JEP.

Along these lines, keep in mind that Camino builders need a fat SharedMenusCocoa.

Comment 4

[Mark Mentovai]

A functionally complete implementation along with in-tree support to drive the build process has been checked in on CAMINO_1_0_BRANCH.  A trunk patch based on that work will appear shortly.

Comment 5

[Mark Mentovai]

Attachment: unify

This version is review-worthy, or at least glance'n'test-worthy.  It is intended to live in the tree at mozilla/build/macosx/universal/unify.  It includes perldoc documentation that should be sufficient to help use it.  Briefly:

usage: unify <ppc-path> <x86-path> <universal-path>
            [--dry-run]           (print what would be done)
            [--only-one <action>] (skip, copy, fail; default=copy)
            [--verbosity <level>] (0, 1, 2, 3; default=2)

(--only-one is not self-explanatory.  It determines the action taken when a file exists in only one of <ppc-path> and <x86-path>.  I couldn't come up with a better name for this option that was short enough to be useful.)

Comment 6

[Mark Mentovai]

Attachment: fix-buildconfig

The only non-executable bits in ppc and x86 builds that differ is the buildconfig page.  If left as-is, unify will die with an error when it encounters the buildconfig file.  This script, fix-buildconfig, preprocesses the architecture-specific toolkit.jar or embed.jar files by combining their content/global/buildconfig.html members and replacing them in each archive.

usage: fix-buildconfig <zipfile1> <zipfile2>

Comment 7

[Mark Mentovai]

Attachment: Sample buildconfig.html

This is the buildconfig.html file from the Camino 1.0 release, which was built using the unify and fix-buildconfig scripts posted here.

Comment 8

[Josh Aas]

Comment on attachment 212245 [details]
unify

# copmareZipArchives($zip1, $zip2)

typo - it is "compareZipArchives"

Is a stat cache really necessary? Do you have perf numbers to validate its existence? For maintainability reasons, we probably shouldn't do that unless it is really necessary. KISS is key for these packaging perl scripts.

Passes the "perl -c ..." test!

Comment 9

[Mark Mentovai]

Comment on attachment 212245 [details]
unify

The cache is good for about 5%.  Perl will internally cache successive stats if you're repeatedly looking at the same file, but because this script looks at two files back and forth a few times, that optimization would be lost.  The cache isn't just for stats, it's to avoid reopening the same file repetitively to read its magic number.

I can disable the cache, but it won't change the code all that much because I'm not about to rip out that entire object interface at this point.

Comment 10

[Benjamin Smedberg]

Comment on attachment 212246 [details]
fix-buildconfig

I kinda wish you could strip the <h1>about:config</h1> from the second instance, but that's not essential.

Are we calling this in the same order every time (always PPC > intel)?

Comment 11

[Benjamin Smedberg]

Comment on attachment 212245 [details]
unify

typo "copmare" in comment.

Seems to me like the stat cache is overkill.

Comment 12

[Mark Mentovai]

Comment on attachment 212246 [details]
fix-buildconfig

I'll strip out the second about:buildconfig and the empty <p> that follows it by reading up to "<p> </p>" instead of "<body>" when reading the second file.

This script imposes no restriction on the order of the files.  flight.mk (bug 327848) always calls this script with ppc before x86.

Comment 13

[Mark Mentovai]

unify and fix-buildconfig have been checked in on the trunk with the requested changes.

Comment 14

[Daniel Veditz [:dveditz]]

Comment on attachment 212246 [details]
fix-buildconfig

approved for 180 branch, a=dveditz for drivers

Comment 15

[Daniel Veditz [:dveditz]]

Comment on attachment 212245 [details]
unify

approved for 180 branch, a=dveditz for drivers

Comment 16

[Mark Mentovai]

Checked in, 1.8.0 and 1.8.