investigate xpt file loading performance/caching

RESOLVED INCOMPLETE

Status

()

defect
P3
normal
RESOLVED INCOMPLETE
19 years ago
2 years ago

People

(Reporter: warrensomebody, Unassigned)

Tracking

({memory-footprint, perf})

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

From the footprint meeting 7/26/00:

6) .xpt file loading
  Description: Currently we're loading more type library information than we 
necessarily use because of the way xpt files are 
  created as part of the module build process. Better factoring of the xpt files 
could result in less loading at startup time and less 
  memory usage. Also, the type library info needs to be a memory flush listener.
  Module owner: jband@netscape.com, mccabe@netscape.com
  Task owner: jband@netscape.com
  Status: JBand to investigate either better factoring of typelibs in xpt files 
or the idea of putting all of the individual small xpt files 
  into a single zip file.
Keywords: footprint, nsbeta3
Keywords: perf
Status: NEW → ASSIGNED
Adding dependency on jar file packaging. 
Depends on: 18433
Sounds like this is going to be a significant win for beta3. Marking nsbeta3+
Whiteboard: [nsbeta3+]
It looks like work I did for bug 
http://bugzilla.mozilla.org/show_bug.cgi?id=30753 already got the bigger 
part of the space win. This part is to take better advantage of that win.

Here are some rough mem usage numbers I got from running taskman on NT.
This was a release build with symbols. 
I just started mozilla and let it go to the mozilla.org site.
Numbers are in bytes and show average size after a couple of runs

1) What we ship now with a few big xpt files:           18534
2) What developers currently run (97 .xpt files):       18190
3) Those 99 files zipped to one zip:                    18270
4) All .xpt files if we don't run xpt_link (479 files): 18252
5) Those 479 files xtp_link'd to one .xpt file:         18550
6) Those 479 files zipped:                              18280

So, I had thought that the best tradeoff of loading speed and 
footprint would be '6'. But, '3' is near enough the same not
too matter. It looks like the added space of the additional
zip entry names for all the file closely offsets the addition 
bits of the unused xpt stuff we actually load when we don't
keep the .xpt file to the smallest granularity. this is good.
It means that the additional build change of *not* running
xpt_link for each module would not save us anything.

So running zip rather than xpt_link at package time is the minimal
build/package change to get us the best win.

This gets us about 250K gain over what we currently ship.

I still need to do some caching of the zip files if we want to not
loose on speed. I'll hack this and make sure the added space cost
doesn't whack the entire gain here.
Not holding PR3 for this. Marking nsbeta3-. Please nominate for rtm if there's
something we really need to do for Seamonkey RTM.
Whiteboard: [nsbeta3+] → [nsbeta3-]
I have not touched this for months. I'd be happy if someone else took it over. 
The changes to support xpt files in zips got implemented and worked when last I 
tried it. The work that is yet to be done is to change the packaging perl 
scripts to use zip.exe on the xpt files that it currently uses on xpt_link.exe 
on. This would give us browser.zip, mail.zip, etc in place of browser.xpt, 
mail.xpt, etc.

I started in on that back then and ran out of time and clues. I'm not seeing 
myself pick this up again anytime soon. If someone with a clue on these 
packaging scripts wants to dig in then I can certainly answer questions and fix 
bugs in xpti if they are uncovered.

I'll attach the stuff I had at the time. 

IIRC, it was sort of working on Windows, untested on Linux, and expected to not 
work yet on Mac. I was puzzled by the differences in the packaging between the 
mozilla distribution and the netscape stuff. The strategy was to copy xptlink.pl 
to a new file called xptzip.pl and hack it to do the zipping. The build.pl 
script would then call xptzip.pl rather than xptlink.pl and all would be well. 
Again. I think it would be great if someone would pick this up and go with it!
Keywords: nsbeta3
Whiteboard: [nsbeta3-]
Blocks: 27510
No longer depends on: 18433
mass reassign of bugs to dbradley@netscape.com
Assignee: jband → dbradley
Status: ASSIGNED → NEW
Is 256K memory savings worth escalating this for?
Status: NEW → ASSIGNED
I never managed to get anyone interested enough in this likely 256K of footprint 
savings for them to actually do the work. The reamining work is just changes in 
the packaging scripts. This ought to be fairly easy for the packaging script 
gurus. SEE ABOVE

Vidur suggested an innovative solution: just reassign the bug until someone 
pushy enough talks the right person into doing the work. Let's see how pushy 
cathleen is :)
Assignee: dbradley → cathleen
Status: ASSIGNED → NEW
(shaver pushes himself around all the time! :-)
I hate you, Brendan Eich.
Assignee: cathleen → shaver
I don't think shaver is looking at this, so I'll take it.
Assignee: shaver → dveditz
Target Milestone: --- → mozilla0.9.8
Blocks: 7251
No longer blocks: 27510
OS: other → All
Target Milestone: mozilla0.9.8 → mozilla0.9.9
only nsbeta1+ bugs can have milestones, resetting to ---
Target Milestone: mozilla0.9.9 → ---
Target Milestone: --- → mozilla1.0
Status: NEW → ASSIGNED
Moving Netscape owned 0.9.9 and 1.0 bugs that don't have an nsbeta1, nsbeta1+,
topembed, topembed+, Mozilla0.9.9+ or Mozilla1.0+ keyword.  Please send any
questions or feedback about this to adt@netscape.com.  You can search for
"Moving bugs not scheduled for a project" to quickly delete this bugmail.
Target Milestone: mozilla1.0 → mozilla1.2
Changing target milestone as 1.2a has been and gone. Mind you, is this bug still
an issue?
Target Milestone: mozilla1.2alpha → ---
It seems that this one is no longer valid, as the current
Mozilla, Firefox and TB releases have now only a very few
XPT's included.
So either close this one, or keep it open to see if something
can be done in the loading of one XPT file can be done.
That's precisely the problem -- all the little interfaces are combined into a
very few giant files with an .xpt extension, which are completely loaded at
startup. This adds to startup time and memory bloat since loading an .xpt file
is all-or-nothing.

The perf and footprint gain would come from analyzing which interfaces are
actually used at startup or other performance critical tasks and put only those
into a few .xpt files. The infrequently used interfaces coul be put into .zip
archives from which individual interfaces could be read as needed without having
to load all of them (but with a bit more overhead than raw mini-.xpt files).
QA Contact: pschwartau → xpconnect
Assignee: dveditz → nobody
Status: ASSIGNED → NEW
per bug 510309, xpt files are linked on all tier 1 platforms now.
XPT stuff is more of an XPCOM thing. But think is old enough I think I can just close it.
Status: NEW → RESOLVED
Closed: 2 years ago
Component: XPConnect → XPCOM
Resolution: --- → INCOMPLETE
I'd probably say it should go to "WONTFIX", since it is a known item that won't be getting fixed.
You need to log in before you can comment on or make changes to this bug.