modify pkgcp.pl to merge all .xpt files into one file

VERIFIED FIXED in M15

Status

P2
normal
VERIFIED FIXED
19 years ago
14 years ago

People

(Reporter: granrosebugs, Assigned: granrosebugs)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [PDT+] 3/9 will verify after talking with reporter, URL)

(Assignee)

Description

19 years ago
Per thread on n.p.m.performance, we can get a 15% improvement in startup times 
if we package all the .xpt files into one large .xpt file.  Need to modify 
pkgcp.pl to call xptlink after parsing the packages-* files to create this one 
all inclusive xpt file:

----

We found that merging the ~50 xpt files into one big file gave winnt a
13% performance boost at startup. Hence we are trying to figure out the
best way to do it:

2. We can do this at package time
Advantage is we can package it however we want. One uber xpt file. 3
logical xpt files...
Disadvantage is developers wont run with this uber xpt file.

For beta, I think going with package time or installer time is better
for now since it gives us flexibility.

The command to run is:
    xpt_link components/all.xpt components/*.xpt
(Assignee)

Updated

19 years ago
Status: NEW → ASSIGNED
Keywords: beta1
Target Milestone: M15
(Assignee)

Updated

19 years ago
Severity: major → normal
Priority: P3 → P2

Updated

19 years ago
Blocks: 28964
(Assignee)

Comment 1

19 years ago
I have this 95% completed.  It works on linux, but win32 doesn't like 
system("xpt_link foo.xpt *.xpt"), it doesn't expand the glob.  So I'm going to 
have to do manual globbing by reading all the files in the directory, then if it 
ends in .xpt, add it to the list, then do a system("xpt_link foo.xpt $list").  
I'd probably have to do that for the mac anyway since Macperl doesn't like 
globbing unless it's just '*'.  But this will have to wait til tomorrow or the 
next day.

Putting PDT in the whiteboard so I'll have approval when it's ready to check in.
Whiteboard: [PDT]

Comment 2

19 years ago
Putting on PDT+ for beta1.  But will move to PDT- if not fixed by 03/03 or if 
this regresses.
Whiteboard: [PDT] → [PDT+] w/b minus on 03/03
When I saw the perl book with slamm, it said

@files = <*.xpt>
(or)
@files = Glob("*.xpt")

should work. It said we could use the nsFile::KGlob package

If that is of any help ... I strongly suggest refering to a perl book for the 
correct package name.
(Assignee)

Comment 4

19 years ago
right.  but unfortunately, macperl is very very picky about what it accepts for 
globbing.  Basically, it just doesn't work on the mac.  It makes writing XP 
Perl and interesting challenge.  100 lines so far and counting...  I've made 
most of the changes, now I'm testing.  Should be able to hit the 3/3 PDT 
deadline assuming the mac testing goes well.
*all* .xpt files in one glob?

So installing the browser gives you the mail .xpt files?  Plus the secret 
netscape-only .xpt files?

This really should be done at build time, and done per-"component". Or 
introduce extra syntax into the packager list that indicates which .xpt files 
for that component need to be globbed together for that component.

Or maybe you're doing this globbing after the copy in the per-component staging 
areas. In that case never mind :-), you can combine 
[component]/bin/components/*.xpt into [component].xpt
(Assignee)

Comment 6

19 years ago
I'm doing the xpt_link after pkgcp.pl has created the various components as 
defined in the packages-win/mac/unix files.  For commercial builds, it creates a 
new .xpt file including the mozilla one and any new ones installed by running 
pkgcp.pl on the ns tree.

This works on linux (although we don't really use the packages-unix files yet), 
and I just completed a test run on win32 (try 
ftp://sweetlou/products/client/seamonkey/windows/32bit/x86/2000-03-01-16-M15/Net
scapeSetup.exe).  The build finished, it installs, it has only 4 xpt files 
(browser, mail, AIMGlue, and qfaservices.  It then crashes in about 10 seconds.

mac doesn't like the system() call, not to mention I can't seem to find xpt_link 
in dist:viewer which is where it is on linux and win32.

I'm pretty well stalled on this until someone can convince me that the 
repackaging isn't causing the  win32 crash, and/or we get xpt_link for the mac.
(Assignee)

Comment 7

19 years ago
I've pulled the xptlink stuff out of pkgcp.pl since we can't make it work on the 
mac and pkgcp.pl is XP perl.  So I have a new perl script, xptlink.pl, which 
will work on win32 and unix only, and will check it in to the mozilla tree once 
it opens (mozilla/xpinstall/packager/xptlink) unless someone can suggest a 
better location.
I don't see why mozilla/xpinstall/packager/xptlink.pl wouldn't work, in the 
same dir as the package lists and the copying script. Either way there's 
a new entry in the packager directory, might as well be the file itself rather 
than introduce another level in the tree.

Bummer about the Mac not working, it's the one that could probably have 
benefitted the most from this change.
(Assignee)

Comment 9

19 years ago
xptlink.pl checked in.  I'll add it to the automation and do some test builds.
(Assignee)

Comment 10

19 years ago
Done.  see the win32 bits in the url field above (or any daily bits starting 
3/3) for verification.  I get an error the first time I run the commercial 
build, but running it a second time shows no problems, and I get the same error 
when running this morning's non-xptlink'ed build, so I don't think it's caused 
by xptlink.pl.  Looking at Netscape\Netscape 6\components shows only 
aimglue.xpt, browser.xpt, mail.xpt, qfaservices.xpt.

ftp://sweetlou/products/client/seamonkey/windows/32bit/x86/2000-03-02-15-M15/Net
scapeSetup.exe

changes to linux and win32 build automation complete as well.  jj and beard are 
working on the mac side of this.  marking resolved/fixed, and jj or beard can 
open a mac-specific bug since it's going to be a rather different process from 
this.
Status: ASSIGNED → RESOLVED
Last Resolved: 19 years ago
Resolution: --- → FIXED
Your startup problems are due to a change by dp yesterday; see thread in the 
.builds group between him and jband. Nothing to do with this.

Comment 12

19 years ago
so before I mark this verified, I'm a bit confused on what I should see on linux
- I see about 50 xpt files, as opposed to 4 on windows - is this correct?

also, has a mac-specific bug been written up?
QA Contact: cyeh → paulmac
Whiteboard: [PDT+] w/b minus on 03/03 → [PDT+] 3/9 will verify after talking with reporter
(Assignee)

Comment 13

19 years ago
well, in order to know what files to link together, the xpt files have to be 
copied into their component directories by pkgcp.pl.  pkgcp.pl copies the files 
to /h/lithium/root/media/exposed/unix/Linux2.1x86glibc/en/5.0 and then 
xptlink.pl runs xptlink on each subdirectory of that directory.  So if you look 
there, you will find only 3 xpt files:
----
autobahn.mcom.com:41% ls
AIM  browser  install  mail  psm  security  spellchecker
autobahn.mcom.com:42% pwd
/.automount/lithium/root/media/exposed/unix/Linux2.1x86glibc/en/5.0
autobahn.mcom.com:43% find . -name "*.xpt" -print
./security/bin/components/security.xpt
./browser/bin/components/browser.xpt
./mail/bin/components/mail.xpt                                  
----

The problem is that for Linux we don't have an installer yet and won't for 
beta1, so the actual tar.gz file people install won't have the linked xpt files.  
There has been talk of making xptlink.pl use a file to control what xpt files 
are linked together.  But that's not going to happen for beta1 either, so at 
this point it looks like win32 is the only platform that's getting linked xpt 
files for beta1.  jj is still working on the mac xptlink process last I heard.
We do NOT need a native install stub to make the .xpi files useful. The .XPI 
files could be produced *now* and then tested from an html page.

This would catch problems with the package lists, would result in a reduction 
in the number of .xpt files that could then be clumped into the tar ball, and 
other goodness.
(Assignee)

Comment 16

19 years ago
I'm all for that.  But as far as shipping (and I think the bits that QA tests), 
they'll need an installer.  The Linux automation already runs makeall to create 
psm and security xpi files.  If you want to have that make all the rest of the 
xpi files and an update.html page, that would be great.

Comment 17

19 years ago
marking verified for windows case, separate bug on mac, linux is a different 
story - let me know if you need a bug for the linux case
Status: RESOLVED → VERIFIED
Product: Browser → Seamonkey
You need to log in before you can comment on or make changes to this bug.