Last Comment Bug 324855 - Need a script to produce universal binary .app from cpu-specific bits
: Need a script to produce universal binary .app from cpu-specific bits
: fixed1.8.0.2, fixed1.8.1
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: 1.8 Branch
: PowerPC Mac OS X
: -- normal (vote)
: ---
Assigned To: Mark Mentovai
: Gregory Szorc [:gps]
Depends on:
Blocks: 324651 327848
  Show dependency treegraph
Reported: 2006-01-26 13:37 PST by Mark Mentovai
Modified: 2006-02-21 20:02 PST (History)
3 users (show)
dveditz: blocking1.8.0.2+
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---

unify (draft) (22.94 KB, text/plain)
2006-01-29 20:52 PST, Mark Mentovai
no flags Details
fix-buildconfig (2.49 KB, text/plain)
2006-01-29 20:56 PST, Mark Mentovai
no flags Details
unify (40.32 KB, text/plain)
2006-02-17 12:32 PST, Mark Mentovai
jaas: review+
benjamin: review+
benjamin: approval‑branch‑1.8.1+
dveditz: approval1.8.0.2+
fix-buildconfig (4.18 KB, text/plain)
2006-02-17 12:40 PST, Mark Mentovai
benjamin: review+
benjamin: approval‑branch‑1.8.1+
dveditz: approval1.8.0.2+
Sample buildconfig.html (3.86 KB, text/html)
2006-02-17 12:45 PST, Mark Mentovai
no flags Details

Description Mark Mentovai 2006-01-26 13:37:28 PST
Working on this, making progress, almost done.
Comment 1 Mark Mentovai 2006-01-29 20:52:38 PST
Created attachment 210111 [details]
unify (draft)

I haven't finished documenting/commenting, and there are a couple more things I want to add, but this script is functionally complete and I'm now running the same it produced on both ppc and x86.
Comment 2 Mark Mentovai 2006-01-29 20:56:36 PST
Created attachment 210113 [details]

This is a small helper script I wrote to gracefully handle the about:buildconfig situation for universal builds.  It takes two arguments, each a path to a toolkit.jar file.  toolkit/content/buildconfig.html is extracted from each, their contents are somewhat intelligently glued together with an <hr> separator, and the zip files are rewritten with the new buildconfig.
Comment 3 Mark Mentovai 2006-01-29 21:23:55 PST
unify runs lipo on Mach-O files and compares non-Mach-O files a la diff.  If two zip files (including jars) fail the diff check, not all is doomed: the zips' directories are scrutinized and CRCs compared.

Using ppc and x86 builds produced using the cross build system under development in bug 322578, the following items fail the diff check:

 - toolkit/content/buildconfig.html in chrome/toolkit.jar.  fix-buildconfig
   (attached) accounts for this.
 - On the branch, components/nsExtensionManager.js.  This was fixed in
   bug 323330, but hasn't yet made it to the branch.

Additionally, the lipo merge fails for the following item:

 - JEP.  Since JEP is a precompiled binary in our tree, we'll need to check in
   a fat JEP.  Until then, even native x86 builds are produced with a ppc
   version of JEP.

Along these lines, keep in mind that Camino builders need a fat SharedMenusCocoa.
Comment 4 Mark Mentovai 2006-02-02 21:02:41 PST
A functionally complete implementation along with in-tree support to drive the build process has been checked in on CAMINO_1_0_BRANCH.  A trunk patch based on that work will appear shortly.
Comment 5 Mark Mentovai 2006-02-17 12:32:47 PST
Created attachment 212245 [details]

This version is review-worthy, or at least glance'n'test-worthy.  It is intended to live in the tree at mozilla/build/macosx/universal/unify.  It includes perldoc documentation that should be sufficient to help use it.  Briefly:

usage: unify <ppc-path> <x86-path> <universal-path>
            [--dry-run]           (print what would be done)
            [--only-one <action>] (skip, copy, fail; default=copy)
            [--verbosity <level>] (0, 1, 2, 3; default=2)

(--only-one is not self-explanatory.  It determines the action taken when a file exists in only one of <ppc-path> and <x86-path>.  I couldn't come up with a better name for this option that was short enough to be useful.)
Comment 6 Mark Mentovai 2006-02-17 12:40:52 PST
Created attachment 212246 [details]

The only non-executable bits in ppc and x86 builds that differ is the buildconfig page.  If left as-is, unify will die with an error when it encounters the buildconfig file.  This script, fix-buildconfig, preprocesses the architecture-specific toolkit.jar or embed.jar files by combining their content/global/buildconfig.html members and replacing them in each archive.

usage: fix-buildconfig <zipfile1> <zipfile2>
Comment 7 Mark Mentovai 2006-02-17 12:45:02 PST
Created attachment 212247 [details]
Sample buildconfig.html

This is the buildconfig.html file from the Camino 1.0 release, which was built using the unify and fix-buildconfig scripts posted here.
Comment 8 Josh Aas 2006-02-17 15:55:50 PST
Comment on attachment 212245 [details]

# copmareZipArchives($zip1, $zip2)

typo - it is "compareZipArchives"

Is a stat cache really necessary? Do you have perf numbers to validate its existence? For maintainability reasons, we probably shouldn't do that unless it is really necessary. KISS is key for these packaging perl scripts.

Passes the "perl -c ..." test!
Comment 9 Mark Mentovai 2006-02-17 16:19:44 PST
Comment on attachment 212245 [details]

The cache is good for about 5%.  Perl will internally cache successive stats if you're repeatedly looking at the same file, but because this script looks at two files back and forth a few times, that optimization would be lost.  The cache isn't just for stats, it's to avoid reopening the same file repetitively to read its magic number.

I can disable the cache, but it won't change the code all that much because I'm not about to rip out that entire object interface at this point.
Comment 10 Benjamin Smedberg [:bsmedberg] 2006-02-21 08:44:29 PST
Comment on attachment 212246 [details]

I kinda wish you could strip the <h1>about:config</h1> from the second instance, but that's not essential.

Are we calling this in the same order every time (always PPC > intel)?
Comment 11 Benjamin Smedberg [:bsmedberg] 2006-02-21 09:28:18 PST
Comment on attachment 212245 [details]

typo "copmare" in comment.

Seems to me like the stat cache is overkill.
Comment 12 Mark Mentovai 2006-02-21 12:08:20 PST
Comment on attachment 212246 [details]

I'll strip out the second about:buildconfig and the empty <p> that follows it by reading up to "<p> </p>" instead of "<body>" when reading the second file.

This script imposes no restriction on the order of the files. (bug 327848) always calls this script with ppc before x86.
Comment 13 Mark Mentovai 2006-02-21 12:39:08 PST
unify and fix-buildconfig have been checked in on the trunk with the requested changes.
Comment 14 Daniel Veditz [:dveditz] 2006-02-21 15:35:44 PST
Comment on attachment 212246 [details]

approved for 180 branch, a=dveditz for drivers
Comment 15 Daniel Veditz [:dveditz] 2006-02-21 15:35:50 PST
Comment on attachment 212245 [details]

approved for 180 branch, a=dveditz for drivers
Comment 16 Mark Mentovai 2006-02-21 20:02:51 PST
Checked in, 1.8.0 and 1.8.

Note You need to log in before you can comment on or make changes to this bug.