298670 - Build Firefox tarballs from the installer manifests

Status: RESOLVED FIXED

(Toolkit Graveyard :: Build Config, defect)

Description

[Benjamin Smedberg]

In order for the update MARs to be accurate, they must be built from the same
set of object files that are being shipped in the installer. This patch changes
the tarball build system to use the installer manifests when available (keeping
the status-quo on mac).

Comment 1

[Benjamin Smedberg]

Attachment: Build tarballs using the installer manifests, rev. 1 [backed out]

Chase, I'm going to be flying all day tomorrow and taking a personal day
Monday, so I hoping that you have the energy to land this or delegate it to
someone else. It has been tested on linux but not windows: I don't expect
windows difficulties, but it should be tested just to make sure that silly
as/cygwin perl or backslash delimiters don't trip us up.

Comment 2

[Chase Phillips]

Comment on attachment 187193 [details] [diff] [review]
Build tarballs using the installer manifests, rev. 1 [backed out]

Landing this needs to happen asap to test the update mechanism.

Comment 3

[Chase Phillips]

Comment on attachment 187193 [details] [diff] [review]
Build tarballs using the installer manifests, rev. 1 [backed out]

Checking in toolkit/mozapps/installer/packager.mk;
/cvsroot/mozilla/toolkit/mozapps/installer/packager.mk,v  <--  packager.mk
new revision: 1.10; previous revision: 1.9
done
Checking in xpinstall/packager/Packager.pm;
/cvsroot/mozilla/xpinstall/packager/Packager.pm,v  <--	Packager.pm
new revision: 1.7; previous revision: 1.6
done
Checking in xpinstall/packager/xptlink.pl;
/cvsroot/mozilla/xpinstall/packager/xptlink.pl,v  <--  xptlink.pl
new revision: 1.7; previous revision: 1.6
done
Checking in browser/installer/Makefile.in;
/cvsroot/mozilla/browser/installer/Makefile.in,v  <--  Makefile.in
new revision: 1.16; previous revision: 1.15
done

Comment 4

[Scott MacGregor]

I'll put together the thunderbird patch for this.

Comment 5

[Chase Phillips]

The only .xpt files the latest .zip build had was browser.xpt where the previous
.zip had many .xpt files.  I thought this patch had worked but after installing
the installer I see many .xpt files in my components directory.

Upon further inspection, the Firefox respin had this error in its xptlink.pl output:

[adt]
[browser]
FAILED: get_file_length: No such file or directory
'/cygdrive/c/builds/tinderbox/Fx-Trunk/WINNT_5.2_Depend/mozilla//dist/bin/xpt_link
/cygdrive/c/builds/tinderbox/Fx-Trunk/WINNT_5.2_Depend/mozilla//stage/browser/bin/components/browser.xpt.new
/cygdrive/c/builds/tinderbox/Fx-Trunk/WINNT_5.2_Depend/mozilla//stage/browser/bin/components/accessibility-msaa.xpt
...
/cygdrive/c/builds/tinderbox/Fx-Trunk/WINNT_5.2_Depend/mozilla//stage/browser/bin/components/xultmpl.xpt'
failed at xptlink.pl line 144.

For comparison, a yesterday's release nightly had this output:

Linking .xpt files...
[abe]
[adt]
[browser]
[en-US]
[reporter]
[talkback]
[xpcom]
Linking .xpt files completed.

The only thing that springs out at me is the extra forward slash in the
arguments sent to xpt_link.  Since we're building those arguments in a different
way now I suspect this might be the cause of the error.  I committed a patch
just now to remove those extra slashes and started a respin on pacifica.

Comment 6

[Scott MacGregor]

(In reply to comment #5)
> The only .xpt files the latest .zip build had was browser.xpt where the previous
> .zip had many .xpt files.  I thought this patch had worked but after installing
> the installer I see many .xpt files in my components directory.

Chase, what do you mean by a lot of .xpt files in your components directory
(i.e. how many)? the xpt linker should be linking almost all of them together
into browser.xpt, so I would expect you to only see browser.xpt and maybe one
for qfa and another one for xpcom. 

Or maybe I misunderstood your comment and you are saying that installer builds
now have a bunch of .xpt files with this patch where before you would have only
seen a couple. In which case, ignore my question.

Comment 7

[Chase Phillips]

(In reply to comment #6)
> Or maybe I misunderstood your comment and you are saying that installer builds
> now have a bunch of .xpt files with this patch where before you would have only
> seen a couple. In which case, ignore my question.

This is what I was describing.

Comment 8

[Chase Phillips]

Comment on attachment 187193 [details] [diff] [review]
Build tarballs using the installer manifests, rev. 1 [backed out]

I backed these changes out.

Checking in browser/installer/Makefile.in;
/cvsroot/mozilla/browser/installer/Makefile.in,v  <--  Makefile.in
new revision: 1.17; previous revision: 1.16
done
Checking in xpinstall/packager/xptlink.pl;
/cvsroot/mozilla/xpinstall/packager/xptlink.pl,v  <--  xptlink.pl
new revision: 1.9; previous revision: 1.8
done
Checking in xpinstall/packager/Packager.pm;
/cvsroot/mozilla/xpinstall/packager/Packager.pm,v  <--	Packager.pm
new revision: 1.8; previous revision: 1.7
done
Checking in toolkit/mozapps/installer/packager.mk;
/cvsroot/mozilla/toolkit/mozapps/installer/packager.mk,v  <--  packager.mk
new revision: 1.11; previous revision: 1.10
done

Comment 9

[Chase Phillips]

Bug 298976 pointed out other interesting problems that resulted from this patch.

Comment 10

[Benjamin Smedberg]

Attachment: Fix * globbing in the flat case, rev. 1 [checked in]

I ended up fixing all the hideous tabs in Packager.pm, so this patch is a bit
rough: I'll attach a -w version for easier review.

Comment 11

[Benjamin Smedberg]

Attachment: Fix * globbing in the flat case, rev. 1 (-w)

Comment 12

[Chase Phillips]

Comment on attachment 187193 [details] [diff] [review]
Build tarballs using the installer manifests, rev. 1 [backed out]

Minusing since I backed out due to regressions.

Comment 13

[Chase Phillips]

Comment on attachment 187510 [details] [diff] [review]
Fix * globbing in the flat case, rev. 1 [checked in]

land asap, i'll respin, and we can test result with quick turnaround

Comment 14

[Benjamin Smedberg]

Fixed on trunk, with a secondary fix for xptlink.pl to use `cygpath` when
necessary to convert cygwin-style paths.

Comment 15

[Scott MacGregor]

Attachment: thunderbird windows changes

Untested changes for Thunderbird windows builds. I also renamed basemail-win to
packages-static to be consistent with Firefox. Will test this later tonight at
home (where my static installer build lives)

Comment 16

[Mike Kaply [:mkaply]]

Attachment: OS/2 build bustage fix

This broke OS/2 because we follow the MSDOS path.

I'm removing $os completely because it is not used anymore.

I'm using $^O = to cygwin to decide to do the cygpath stuff,

Comment 17

[Benjamin Smedberg]

Comment on attachment 197452 [details] [diff] [review]
OS/2 build bustage fix

Won't this break the toolkit usages?

http://lxr.mozilla.org/mozilla/source/toolkit/mozapps/installer/build_static.pl
#134
http://lxr.mozilla.org/mozilla/source/toolkit/mozapps/installer/packager.mk#259