Need a script to produce universal binary .app from cpu-specific bits

RESOLVED FIXED

Status

()

Core
Build Config
RESOLVED FIXED
12 years ago
12 years ago

People

(Reporter: Mark Mentovai, Assigned: Mark Mentovai)

Tracking

({fixed1.8.0.2, fixed1.8.1})

1.8 Branch
PowerPC
Mac OS X
fixed1.8.0.2, fixed1.8.1
Points:
---
Dependency tree / graph
Bug Flags:
blocking1.8.0.2 +

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [camino-1.0])

Attachments

(3 attachments, 2 obsolete attachments)

(Assignee)

Description

12 years ago
Working on this, making progress, almost done.
(Assignee)

Comment 1

12 years ago
Created attachment 210111 [details]
unify (draft)

I haven't finished documenting/commenting, and there are a couple more things I want to add, but this script is functionally complete and I'm now running the same unifox.app it produced on both ppc and x86.
(Assignee)

Comment 2

12 years ago
Created attachment 210113 [details]
fix-buildconfig

This is a small helper script I wrote to gracefully handle the about:buildconfig situation for universal builds.  It takes two arguments, each a path to a toolkit.jar file.  toolkit/content/buildconfig.html is extracted from each, their contents are somewhat intelligently glued together with an <hr> separator, and the zip files are rewritten with the new buildconfig.
(Assignee)

Comment 3

12 years ago
unify runs lipo on Mach-O files and compares non-Mach-O files a la diff.  If two zip files (including jars) fail the diff check, not all is doomed: the zips' directories are scrutinized and CRCs compared.

Using ppc and x86 builds produced using the cross build system under development in bug 322578, the following items fail the diff check:

 - toolkit/content/buildconfig.html in chrome/toolkit.jar.  fix-buildconfig
   (attached) accounts for this.
 - On the branch, components/nsExtensionManager.js.  This was fixed in
   bug 323330, but hasn't yet made it to the branch.

Additionally, the lipo merge fails for the following item:

 - JEP.  Since JEP is a precompiled binary in our tree, we'll need to check in
   a fat JEP.  Until then, even native x86 builds are produced with a ppc
   version of JEP.

Along these lines, keep in mind that Camino builders need a fat SharedMenusCocoa.
(Assignee)

Updated

12 years ago
Blocks: 324651
(Assignee)

Comment 4

12 years ago
A functionally complete implementation along with in-tree support to drive the build process has been checked in on CAMINO_1_0_BRANCH.  A trunk patch based on that work will appear shortly.
Whiteboard: [camino-1.0]
(Assignee)

Comment 5

12 years ago
Created attachment 212245 [details]
unify

This version is review-worthy, or at least glance'n'test-worthy.  It is intended to live in the tree at mozilla/build/macosx/universal/unify.  It includes perldoc documentation that should be sufficient to help use it.  Briefly:

usage: unify <ppc-path> <x86-path> <universal-path>
            [--dry-run]           (print what would be done)
            [--only-one <action>] (skip, copy, fail; default=copy)
            [--verbosity <level>] (0, 1, 2, 3; default=2)

(--only-one is not self-explanatory.  It determines the action taken when a file exists in only one of <ppc-path> and <x86-path>.  I couldn't come up with a better name for this option that was short enough to be useful.)
Attachment #210111 - Attachment is obsolete: true
Attachment #212245 - Flags: review?(joshmoz)
(Assignee)

Comment 6

12 years ago
Created attachment 212246 [details]
fix-buildconfig

The only non-executable bits in ppc and x86 builds that differ is the buildconfig page.  If left as-is, unify will die with an error when it encounters the buildconfig file.  This script, fix-buildconfig, preprocesses the architecture-specific toolkit.jar or embed.jar files by combining their content/global/buildconfig.html members and replacing them in each archive.

usage: fix-buildconfig <zipfile1> <zipfile2>
Attachment #210113 - Attachment is obsolete: true
Attachment #212246 - Flags: review?(benjamin)
(Assignee)

Comment 7

12 years ago
Created attachment 212247 [details]
Sample buildconfig.html

This is the buildconfig.html file from the Camino 1.0 release, which was built using the unify and fix-buildconfig scripts posted here.

Comment 8

12 years ago
Comment on attachment 212245 [details]
unify

# copmareZipArchives($zip1, $zip2)

typo - it is "compareZipArchives"

Is a stat cache really necessary? Do you have perf numbers to validate its existence? For maintainability reasons, we probably shouldn't do that unless it is really necessary. KISS is key for these packaging perl scripts.

Passes the "perl -c ..." test!
Attachment #212245 - Flags: review?(joshmoz) → review+
(Assignee)

Comment 9

12 years ago
Comment on attachment 212245 [details]
unify

The cache is good for about 5%.  Perl will internally cache successive stats if you're repeatedly looking at the same file, but because this script looks at two files back and forth a few times, that optimization would be lost.  The cache isn't just for stats, it's to avoid reopening the same file repetitively to read its magic number.

I can disable the cache, but it won't change the code all that much because I'm not about to rip out that entire object interface at this point.
Attachment #212245 - Flags: superreview?(benjamin)
(Assignee)

Updated

12 years ago
Attachment #212245 - Flags: superreview?(benjamin) → review?(benjamin)
(Assignee)

Updated

12 years ago
Blocks: 327848
Comment on attachment 212246 [details]
fix-buildconfig

I kinda wish you could strip the <h1>about:config</h1> from the second instance, but that's not essential.

Are we calling this in the same order every time (always PPC > intel)?
Attachment #212246 - Flags: review?(benjamin) → review+
Comment on attachment 212245 [details]
unify

typo "copmare" in comment.

Seems to me like the stat cache is overkill.
Attachment #212245 - Flags: review?(benjamin) → review+
(Assignee)

Comment 12

12 years ago
Comment on attachment 212246 [details]
fix-buildconfig

I'll strip out the second about:buildconfig and the empty <p> that follows it by reading up to "<p> </p>" instead of "<body>" when reading the second file.

This script imposes no restriction on the order of the files.  flight.mk (bug 327848) always calls this script with ppc before x86.
(Assignee)

Comment 13

12 years ago
unify and fix-buildconfig have been checked in on the trunk with the requested changes.
Status: NEW → RESOLVED
Last Resolved: 12 years ago
Resolution: --- → FIXED
(Assignee)

Updated

12 years ago
Attachment #212245 - Flags: approval1.8.0.2?
Attachment #212245 - Flags: approval-branch-1.8.1?(benjamin)
(Assignee)

Updated

12 years ago
Attachment #212246 - Flags: approval1.8.0.2?
Attachment #212246 - Flags: approval-branch-1.8.1?(benjamin)
Attachment #212246 - Flags: approval-branch-1.8.1?(benjamin) → approval-branch-1.8.1+
Attachment #212245 - Flags: approval-branch-1.8.1?(benjamin) → approval-branch-1.8.1+
Flags: blocking1.8.0.2+
Comment on attachment 212246 [details]
fix-buildconfig

approved for 180 branch, a=dveditz for drivers
Attachment #212246 - Flags: approval1.8.0.2? → approval1.8.0.2+
Comment on attachment 212245 [details]
unify

approved for 180 branch, a=dveditz for drivers
Attachment #212245 - Flags: approval1.8.0.2? → approval1.8.0.2+
(Assignee)

Comment 16

12 years ago
Checked in, 1.8.0 and 1.8.
Keywords: fixed1.8.0.2, fixed1.8.1
You need to log in before you can comment on or make changes to this bug.