Closed Bug 468607 Opened 11 years ago Closed 10 years ago

Investigate refactoring .xpt files into startup and runtime components

Categories

(Firefox for Android Graveyard :: General, defect)

defect
Not set

Tracking

(Not tracked)

VERIFIED WONTFIX

People

(Reporter: fred23, Unassigned)

Details

Attachments

(2 files)

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.4) Gecko/2008111317 Ubuntu/8.04 (hardy) Firefox/3.0.4
Build Identifier: fennec-1.0a2pre.en-US

Currently, Fennec is loading all Firefox-required JS and XPT components at startup whereas we know at that time that nsDefaultCLH.js is not necessary.

So the idea here is to remove any unnecessary component from the initial fennec loading to gain some time at startup.

Initially, two components were unloaded without affecting fennec'S startup and runtime behavior : 

 - nsDefaultCLH.js
 - filePicker.xpt


more and more components will be unloaded and tested as the bug's threads go by.

... Eventually, the reduced JS and XPT subset obtained from this work could be merged into a single XPT file... like it's suggested in the fennec startup time optimization page : https://wiki.mozilla.org/Mobile/StartupOptimization.

Feedback from senior programmers with a more in-depth knowledge of Fennec' modules would be very appreciated.

numbers about startup time gains are not yet available, but are coming soon.

Reproducible: Always

Actual Results:  
Actually, two components were unloaded without affecting fennec'S startup and runtime behavior : 

 - nsDefaultCLH.js
 - filePicker.xpt

Need feedback from other developers about those two.

Expected Results:  
startup time reduction is expected.
nsDefaultCLH.js is a critical file. It's responsible for loading the primary XUL window (browser.xul in the case of Fennec).

XPT files should be bundled into a single XPT file for release, like Firefox does. In our case we should have an XPT bundle for XULRunner and a XPT bundle for any Fennec components. If this isn't happening, we should fix the makefiles
At the moment, 104 xpt files are in the xulrunner/components directory after packaging... I'll be more than happy to take on this task of merging all those into a single XTP file. 

And I'm sure there will be an associated startup time reduction with it.

BUT concerning the present bug #468607 - "Fennec loads unnecessary JS and XPT components at startup time", I still think Fennec is loading unnecessary components that used to be crucial for Firefox but not anymore in the mobile version. What do you think?

So I'd like to discuss that statement here in the bug's thread with the interested people to see if there's anything to do with that, at all. If not, we'll close the present bug.
xptlink.pl exists to combine XPT files at build time, in case you weren't already aware of it.

I don't know of any unnecessary components at the moment. Perhaps you should post a list of all of the loaded components and we can look them over? (I think you might have already posted such a list on IRC, but I don't remember the details.)
I've used the REGLOG and STATS mechanisms of the XPTI module in "xptiinterfaceinfoManager.cpp" by setting the MOZILLA_XPTI_REGLOG and MOZILLA_XPTI_STATS environment variables. At every fennec reboot, I'd remove the compreg.dat and xpti.dat manifest files and check the final reglog.log after extensive browsing with fennec.

In those browsing sessions, here are the things that have been done:

  - clicked every clickable widgets : buttons, radios, checkboxes...
  - typed adresses in the addressbar
  - scrolled up-down, left-right
  - added favorites, added/removed bookmarks
  - got the downloads list, got to general settings, got to adds-on, plugins, themes and extensions.
  - enabled and disabled shockwave flash plugin
  - added/removed an extension e.g. URL Fixer
  - modified all "content" and "privacy" settings
  - cleared the private data
  - visit the following sites
      - google.com
      - wikipedia.com
      - cnn.com
      - yahoo.com
      - anwsers.com

That being said, I comparatively checked the Stats.log file for "USED XPT FILES" and "USED INTERFACES", and discovered that for those browsing sessions mentionned above, only 80 XPT files (out of 130) were actually used.

Here are the 51 XPT files that were left out :

./xulrunner/components/necko_res.xpt
./xulrunner/components/commandhandler.xpt
./xulrunner/components/chrome.xpt
./xulrunner/components/directory.xpt
./xulrunner/components/mozbrwsr.xpt
./xulrunner/components/prefetch.xpt
./xulrunner/components/unicharutil.xpt
./xulrunner/components/parentalcontrols.xpt
./xulrunner/components/autoconfig.xpt
./xulrunner/components/htmlparser.xpt
./xulrunner/components/content_xtf.xpt
./xulrunner/components/zipwriter.xpt
./xulrunner/components/mozfind.xpt
./xulrunner/components/proxyObjInst.xpt
./xulrunner/components/filepicker.xpt
./xulrunner/components/chardet.xpt
./xulrunner/components/widget.xpt
./xulrunner/components/content_xmldoc.xpt
./xulrunner/components/dom_loadsave.xpt
./xulrunner/components/necko_viewsource.xpt
./xulrunner/components/oji.xpt
./xulrunner/components/xpcom_xpti.xpt
./xulrunner/components/profile.xpt
./xulrunner/components/layout_printing.xpt
./xulrunner/components/webbrowserpersist.xpt
./xulrunner/components/imglib2.xpt
./xulrunner/components/inspector.xpt
./xulrunner/components/jsdservice.xpt
./xulrunner/components/appstartup.xpt
./xulrunner/components/alerts.xpt
./xulrunner/components/toolkitprofile.xpt
./xulrunner/components/composer.xpt
./xulrunner/components/lwbrk.xpt
./xulrunner/components/exthelper.xpt
./xulrunner/components/webBrowser_core.xpt
./xulrunner/components/cookie.xpt
./xulrunner/components/embed_base.xpt
./xulrunner/components/find.xpt
./xulrunner/components/necko_ftp.xpt
./xulrunner/components/windowds.xpt
./xulrunner/components/webshell_idls.xpt
./xulrunner/components/dom.xpt
./xulrunner/components/necko_socket.xpt
./xulrunner/components/txtsvc.xpt
./xulrunner/components/toolkitremote.xpt
./xulrunner/components/xulapp_setup.xpt
./xulrunner/components/imgicon.xpt
./xulrunner/components/necko_strconv.xpt
./xulrunner/components/spellchecker.xpt
./xulrunner/components/pippki.xpt
./xulrunner/components/dom_sidebar.xpt

Shorlty after, I've completely removed those 51 XPT files from my system, removed the compreg.dat and xpti.dat files, rebuilt the components manifest, rebooted fennec, and it still ran smoothly for a whole browsing session! I couldn't make it crash.

Now, I know that many of these XPT files are STILL NECESSARY, OF COURSE, for specific fennec behaviours that I did not come across during the browsing sessions, but my idea is :

 1 - First, see if there are any of these files that ARE NOT used at all. Like filePicker.xpt ?  Is there a way to 
     pick a file in fennec ? That's where I need your inputs : Help from you guys about the XPT files that you know of would be very appreciated.

 2 - Second, once we have cleared this list of all the xpt's that are still necessary, I think we could end up with an hybrid solution of preloading only the FREQUENTLY USED XPT's and leave the rest out for DYNAMIC LOADING ON REQUEST. We could then load only the 80-some badly important XPT files on startup, and leave the rest for LOADING ON REQUEST. 

Current manifest and componentManager loading strategy does not allow differed loading of XPT interfaces. Currently, if an XPT file is missing when some requestInterface() is issued, fennec terminates. What I'm proposing here is to tweak the componentManager and xptiintercace to be able to load the missing XPT file on request.

I think we could earn a lot in terms of startup time here.
As Gavin mentioned, we should be linking all the xpt files together as part of the build packaging step.  Normally when I've seen xpt files effecting startup is due to stat and other issues opening all the different files, but when they're packaged together it usually falls out of my profiles.  What amount of time is being used when we do link them all together?
Indeed, after packaging, the number of XPT files copied to xulrunner/components is down to 80... those ones that were said to be necessary above. So everything's seems fine on that front. I still want to check if the same applies to scripts (.js .jsm,...) to see if we're stricly loading what's needed.

more to come....

About that XPT merging issue, I'm on it, I'll create a new bug for that and I'll be posting numbers soon I hope.
Have you seen comments #17 and #18 of bug #46707 (https://bugzilla.mozilla.org/show_bug.cgi?id=46707) ??

It talks about several interfaces not needed for firefox that get loaded at startup from those few big .xpt files.

does that ring a bell to anyone ?  Do you think this issue has been resolved for firefox by now ?  And if not, is XPT refactoring could get us a big win on loading time for fennec?
Some notes:
* Fennec itself contains no XPT files.
* XULRunner is not xpt_link'ing any XPT files, so we have 132 individual files.
* Individual files mean only files that are used will be loaded (I think)
* Individual files mean extra time needed to access and load each file.
* xpt_link'ed files mean only 1 file to load.
* xpt_link'ed files mean all XPT information is loaded into memory right away. (bug 46707)
* We will need _all_ of the XPT files since we have no idea what interfaces an extension could be using.

Perhaps we could separate the most common XPT files into one xpt_link'ed bundle and all the rest in a separate xpt_link'ed bundle.
Frederic - Some startup timing numbers for the 132 XPT files and the 80 XPT files cases would be nice.
As I said, the "132-XPT-file case" doesn't happen after packaging. It was only the "dist/bin" version that loaded, as Stuart suggested, the whole bunch of interfaces because of the stats mechanism. After packaging, we're down to 80 XPT files in the /xulrunner/component directory, so it seems fine there.

Besides, I like the idea of creating 2 XPT bundles, one for common interfaces, and another for not so common ones that might get accessed by extensions...
finkle: it sounds like we should have xulrunner xpt_linking the files.
If nobody minds, I'm on it (xpt_linking). It should imply only hacks to the Makefile if I'm correct. It should be no big deal. After that, I'll be able to get some nice numbers of comparative startup time between several-XPT and 1-or-2-XPT scenarios.

In the meantime, I've gathered other results about loaded and used components interface-wise. I've attached two texfiles "loadedInterfaces" and "usedInterfaces" that respectively show, for the packaged version of fennec, the interfaces currently loaded at startup time, and the interfaces actually accessed at runtime for a reasonably long and diversified browsing session.

We easily see that close to 66% of all the loaded interfaces never get touched. Maybe those figures could help us conduct some XPT refactoring in terms of "what should get bundled in the *common-XPT*"...
Attached file LoadedInterfaces
This lists the XPCOM interfaces loaded at startup time by xulrunner for fennec packaged version 1.0 alpha2.
Attached file UsedInterfaces
This lists the XPCOM interfaces used at runtime by xulrunner for fennec
packaged version 1.0 alpha2.. for a reasonably complete browsing session.
i take it you didn't try to upload any files.

if you want the browser to crash, this sounds like a great way to go about it.

xpt_linking would be a much better idea.
timeless : what do you mean by "i take it you didn't try to upload any files" ?
The file picker is probably used to select a file when a website offers you to upload one. Try Flickr to see if Fennec crashes.
i had a very long comment listing a number of problems with your testing methodology.

among other things, you didn't file inputs, spell checker, you didn't care about accessibility / automated testing, and you don't understand what can go wrong if typelib information is not available.

last i checked, the primary xpt system didn't support incremental loading.

if you want to xpt_link the whole blob into one file, that's fine, and everyone is encouraged to do that when they try to ship a product.

commandline handling is something which you'll miss when someone tries to use it.

your entire methodology is fatally flawed and you're running around like a bull in a china shop, except that you're so blind and deaf you won't notice if you broke everything.

could someone please file a bug against the xulrunner build so that it uses xptlink?

atm there's no point in using more than one bundle, because we don't have a paging or incremental anything.

note that firefox already uses xpt_link which means you want to copy the build logic from that.
I'm really sorry, timeless that my contribution to this project appears to upset you. I really do think that people should use the present threads to make constructive comments and contribute with each other, instead of trying to put people down.

Concerning your request for someone filing a bug about xulrunner using xptlink, it's already been taken care of. See bug #469873.
> atm there's no point in using more than one bundle, because we don't have a
> paging or incremental anything.

I disagree. I'm investigating streamlining some stuff in case we have no extensions installed. In this case, we could possibly avoid loading a bunch of useless xpt files if they were separate.
Summary: Fennec loads unnecessary JS and XPT components at startup time → Investigate refactoring .xpt files into startup and runtime components
I changed the bug's name because that's really what it's about : the idea of making 2 XPT bundles instead of one.

See Bug 469873 for XPT_linking patches.

As an attempt to summarize what's presented here, the general idea would be to investigate the impact of having 2 XPT bundles : One for startup stuff (mandatory) and an optional one for accessory stuff mainly used by extensions. This second XPT file's loading could be done when necessary (presence of extensions), thus relieving fennec's startup in many cases.

This idea came out while discussing the fact that the majority of (firefox?) users have no extensions.

Stuart: this bug is very similar to Bug 311968. Do you think we should merge them?
We should not merge with bug 311968. It's too different.

In fact, should we close this out? I think bug 469873 did a great job. Do we still think this bug is worth working on?
(In reply to comment #22)
> We should not merge with bug 311968. It's too different.
> 
> In fact, should we close this out? I think bug 469873 did a great job. Do we
> still think this bug is worth working on?

I agree. I revisited this bug yesterday, played with it a little and timed ts for a reduced subset of xpt components at startup... and got no significant "ts" gain ... The numbers aren't super reliable, but I feel we won't get much gain out of this bug.

Please someone, back me up on this, and we'll close this one out.
Ping, should this bug be closed?
(In reply to comment #24)
> Ping, should this bug be closed?

I think so. For now anyway.
Status: UNCONFIRMED → RESOLVED
Closed: 10 years ago
Resolution: --- → WONTFIX
verified per comment 22-25.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.