Closed Bug 562406 Opened 14 years ago Closed 13 years ago

Omnijar js components from startupcache during build

Categories

(Core :: XPCOM, defect)

x86
Linux
defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla2.0b9
Tracking Status
blocking2.0 --- -

People

(Reporter: taras.mozilla, Assigned: taras.mozilla)

References

Details

(Whiteboard: [ts])

Attachments

(1 file, 6 obsolete files)

Once js files are moved into a jar we can avoid excess filesystem io during fastload generation. However since the js files are not going to change, we might as well stick their binary form into the jar. Then those can bypass the fastload infrastructure altogether. This should reduce out startup time(ie less computation in the fastload code), reduce ram usage(by using jars more effectively), and reduce our disk footprint.
Whiteboard: [ts]
Depends on: omnijar
Basically a "compile" step, like Java's javac .java -> .class?
Makes sense to me, as long as developers can opt out of this during dev, so that they can still edit plain files, restart FF and have them used, without compilation.
(In reply to comment #1)
> Basically a "compile" step, like Java's javac .java -> .class?
> Makes sense to me, as long as developers can opt out of this during dev, so
> that they can still edit plain files, restart FF and have them used, without
> compilation.

Yeah I think flat chrome should force plain files
Am I seeing a symptom of this bug or is this a different one?

At this list of Symbian bugs: http://developer.symbian.org/bugs/buglist.cgi?bug_file_loc=&bug_file_loc_type=allwordssubstr&bug_id=&bug_status=NEW&bug_status=ASSIGNED&bug_status=PROPOSED&bug_status=RESOLVED&bugidtype=include&chfieldfrom=&chfieldto=Now&chfieldvalue=&email1=&email2=&emailassigned_to1=1&emailassigned_to2=1&emailcc2=1&emailreporter2=1&emailtype1=substring&emailtype2=substring&field-1-0-0=bug_status&field-1-1-0=keywords&field0-0-0=noop&keywords=gecko&keywords_type=allwords&long_desc=&long_desc_type=substring&query_format=advanced&remaction=&short_desc=&short_desc_type=allwordssubstr&type-1-0-0=anyexact&type-1-1-0=allwords&type0-0-0=noop&value-1-0-0=NEW,ASSIGNED,PROPOSED,RESOLVED&value-1-1-0=gecko&value0-0-0=&order=bugs.resolution,bugs.bug_id&query_based_on=

Mozilla/5.0 (Windows NT 6.0; rv:2.0b5pre) Gecko/20100822 Minefield/4.0b5pre ID:20100822040607 becomes unresponsive for some seconds and then I get this:
"Warning: Unresponsive script
A script on this page may be busy, or it may have stopped responding. You can stop the script now, or you can continue to see if the script will complete.
Script: jar:file:///C:/Program Files/firefox/omni.jar!/components/nsProxyAutoConfig.js:118"
(In reply to comment #3)
> Am I seeing a symptom of this bug or is this a different one?

No. Sounds like broken js on the symbian site.
(In reply to comment #4)
> No. Sounds like broken js on the symbian site.
Why is Minefield itself blaming "omni.jar!/components/nsProxyAutoConfig.js:118" ?
http://mxr.mozilla.org/comm-central/source/mozilla/netwerk/base/src/nsProxyAutoConfig.js#118
118         return dns.resolve(dns.myHostName, 0).getNextAddrAsString();
Looks like a DNS failure.
Just because "omni" is in the error message doesn't mean it's caused by this. If omnijar was the problem, it wouldn't load the nsProxyAutoConfig.js file at all. Please try the same with a nightly directly before omnijar.
Esp. given that it say "Warning: Unresponsive script", and it's using sync APIs (!), that's most likely a problem with the DNS server/client not responding/resolving, i.e. a site problem. And nsProxyAutoConfig.js should use dns.asyncResolve(), but that's also a different bug.
(In reply to comment #7)
> Esp. given that it say "Warning: Unresponsive script", and it's using sync APIs
> (!), that's most likely a problem with the DNS server/client not
> responding/resolving, i.e. a site problem. And nsProxyAutoConfig.js should use
> dns.asyncResolve(), but that's also a different bug.
Do I file the bug?
(In reply to comment #7)
Filed bug 590247 to followup.
(In reply to comment #2)
> (In reply to comment #1)
> > Basically a "compile" step, like Java's javac .java -> .class?
> > Makes sense to me, as long as developers can opt out of this during dev, so
> > that they can still edit plain files, restart FF and have them used, without
> > compilation.
> 
> Yeah I think flat chrome should force plain files

How do you get the plain files when unpacking omni.jar gives you binaries?
At this point, it's not clear whether we would only have the compiled version, or both versions, and whether we should keep a separate build path for allowing plain files in the build, but let's have that discussion once we have a patch to play with, please?
mwu points out that this is conceptually "easy" with startupcache + omnijar. At packaging time can merge startupcache + omnijar jars. Then "merely" add a fallback to startupcache code to look in the omnijar. Worth a try.
Attached patch proof of concept (obsolete) — Splinter Review
Turns out it's also proof-of-concept-easy.
Assignee: nobody → tglek
Summary: Generate fastload info for .js files at build time → Omnijar js components from startupcache during build
So I asked benh if we can force-populate the cache instead of doing it lazily as is done during normal ff execution.

so it looks like the entry points are http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/loader/mozJSComponentLoader.cpp#1283 
http://mxr.mozilla.org/mozilla-central/source/js/src/xpconnect/loader/mozJSComponentLoader.cpp#1373

"but I'm not sure it's possible to duplicate those calls without starting up firefox"
Attached patch omnijar startup cache (obsolete) — Splinter Review
This is what the omnijar looks like as a result of this patch:
-rw-rw-r--  2.3 unx      153 t- defX  1-Jan-10 00:00 chrome.manifest
-rw-rw-r--  2.3 unx       23 t- stor  1-Jan-10 00:00 components/interfaces.manifest
-rw-rw-r--  2.3 unx   389310 b- defX  1-Jan-10 00:00 components/browser.xpt
-rw-rw-r--  2.3 unx    20871 t- defX  1-Jan-10 00:00 components/components.manifest
-rw-rw-r--  2.3 unx     1771 t- defX  1-Jan-10 00:00 chrome/nonlocalized.manifest
-rw-rw-r--  2.3 unx    81959 t- defX  1-Jan-10 00:00 greprefs.js
-rw-rw-r--  2.3 unx    47200 t- defX  1-Jan-10 00:00 defaults/pref/firefox.js
-rw-rw-r--  2.3 unx      213 t- defX  1-Jan-10 00:00 defaults/pref/firefox-l10n.js
-rw-rw-r--  2.3 unx     1580 t- defX  1-Jan-10 00:00 defaults/pref/services-sync.js
-rw-rw-r--  2.3 unx     1246 t- defX  1-Jan-10 00:00 defaults/pref/firefox-branding.js
-rw-rw-r--  2.3 unx     1113 t- defX  1-Jan-10 00:00 chrome/localized.manifest
-rw-r--r--  2.3 unx     7204 b- defX  1-Jan-10 00:00 startupcache/omni-components/Weave.js.bin
-rw-r--r--  2.3 unx     6816 b- defX  1-Jan-10 00:00 startupcache/omni-gre/modules/XPCOMUtils.jsm.bin
-rw-r--r--  2.3 unx    42356 b- defX  1-Jan-10 00:00 startupcache/omni-components/WebContentConverter.js.bin
-rw-r--r--  2.3 unx    60084 b- defX  1-Jan-10 00:00 startupcache/omni-components/nsBrowserGlue.js.bin
-rw-r--r--  2.3 unx     4828 b- defX  1-Jan-10 00:00 startupcache/omni-gre/modules/Services.jsm.bin
-rw-r--r--  2.3 unx    29188 b- defX  1-Jan-10 00:00 startupcache/omni-components/nsPrivateBrowsingService.js.bin
-rw-r--r--  2.3 unx    11472 b- defX  1-Jan-10 00:00 startupcache/omni-components/nsSessionStartup.js.bin
-rw-r--r--  2.3 unx     2376 b- defX  1-Jan-10 00:00 startupcache/omni-components/nsTryToClose.js.bin
-rw-rw-r--  2.3 unx     2924 t- defX  1-Jan-10 00:00 chrome/en-US/locale/en-US/global/intl.properties
-rw-rw-r--  2.3 unx      242 t- defX  1-Jan-10 00:00 chrome/en-US/locale/en-US/global-platform/unix/intl.properties
-rw-r--r--  2.3 unx    11872 b- defX  1-Jan-10 00:00 startupcache/omni-components/addonManager.js.bin
-rw-r--r--  2.3 unx    45768 b- defX  1-Jan-10 00:00 startupcache/omni-gre/modules/AddonManager.jsm.bin
-rw-r--r--  2.3 unx     7016 b- defX  1-Jan-10 00:00 startupcache/omni-gre/modules/AddonLogging.jsm.bin
-rw-r--r--  2.3 unx     3492 b- defX  1-Jan-10 00:00 startupcache/omni-gre/modules/FileUtils.jsm.bin
-rw-r--r--  2.3 unx   241144 b- defX  1-Jan-10 00:00 startupcache/omni-gre/modules/XPIProvider.jsm.bin
-rw-r--r--  2.3 unx    63932 b- defX  1-Jan-10 00:00 startupcache/omni-gre/modules/AddonRepository.jsm.bin
-rw-r--r--  2.3 unx     8512 b- defX  1-Jan-10 00:00 startupcache/omni-gre/modules/NetUtil.jsm.bin
-rw-r--r--  2.3 unx    37692 b- defX  1-Jan-10 00:00 startupcache/omni-gre/modules/LightweightThemeManager.jsm.bin
-rw-r--r--  2.3 unx    16280 b- defX  1-Jan-10 00:00 startupcache/omni-gre/modules/PluginProvider.jsm.bin
-rw-r--r--  2.3 unx     2600 b- defX  1-Jan-10 00:00 startupcache/omni-components/nsUpdateServiceStub.js.bin
-rw-r--r--  2.3 unx    10188 b- defX  1-Jan-10 00:00 startupcache/omni-components/nsUpdateTimerManager.js.bin
-rw-rw-r--  2.3 unx       73 t- defX  1-Jan-10 00:00 chrome/en-US/locale/branding/brand.properties
-rw-rw-r--  2.3 unx      117 t- defX  1-Jan-10 00:00 chrome/toolkit/res/hiddenWindow.html
-rw-rw-r--  2.3 unx    15751 t- defX  1-Jan-10 00:00 chrome/browser/content/browser/sanitize.js
-rw-rw-r--  2.3 unx    13734 t- defX  1-Jan-10 00:00 modules/distribution.js
-rw-rw-r--  2.3 unx     2333 t- defX  1-Jan-10 00:00 chrome/en-US/locale/browser-region/region.properties
-rw-r--r--  2.3 unx    35812 b- defX  1-Jan-10 00:00 startupcache/omni-components/nsBrowserContentHandler.js.bin
-rw-r--r--  2.3 unx     2380 b- defX  1-Jan-10 00:00 startupcache/omni-components/jsconsole-clhandler.js.bin
-rw-r--r--  2.3 unx     1788 b- defX  1-Jan-10 00:00 startupcache/omni-components/nsSetDefaultBrowser.js.bin
-rw-rw-r--  2.3 unx     6451 t- defX  1-Jan-10 00:00 chrome/toolkit/res/ua.css
-rw-rw-r--  2.3 unx    16052 t- defX  1-Jan-10 00:00 chrome/toolkit/res/html.css
-rw-rw-r--  2.3 unx     7972 t- defX  1-Jan-10 00:00 chrome/en-US/locale/en-US/global/css.properties
-rw-rw-r--  2.3 unx    26111 t- defX  1-Jan-10 00:00 chrome/toolkit/content/global/xul.css
-rw-rw-r--  2.3 unx    11356 t- defX  1-Jan-10 00:00 chrome/toolkit/res/quirk.css
-rw-rw-r--  2.3 unx     8258 t- defX  1-Jan-10 00:00 chrome/toolkit/skin/classic/global/scrollbars.css
-rw-rw-r--  2.3 unx    17018 t- defX  1-Jan-10 00:00 chrome/toolkit/res/forms.css
-rw-rw-r--  2.3 unx     2288 t- defX  1-Jan-10 00:00 chrome/toolkit/content/mozapps/plugins/pluginFinderBinding.css
-rw-rw-r--  2.3 unx     2315 t- defX  1-Jan-10 00:00 chrome/toolkit/content/mozapps/plugins/pluginProblemBinding.css
-rw-rw-r--  2.3 unx     9470 t- defX  1-Jan-10 00:00 chrome/toolkit/res/charsetData.properties
-rw-rw-r--  2.3 unx     6101 t- defX  1-Jan-10 00:00 res/langGroups.properties
-rw-rw-r--  2.3 unx   102644 t- defX  1-Jan-10 00:00 chrome/browser/content/browser/browser.xul
-rw-r--r--  2.3 unx     4852 b- defX  1-Jan-10 00:00 startupcache/omni-components/nsDefaultCLH.js.bin

This patch:
* Sanitizes paths in startup cache to make them portable
* produces a binary representation of well-known(ie hardcoded) js components+modules. I'm hoping that eventually this will contain jagermonkey AOT output.
* This is then omnijarred and run through our jar-reordering routine.

The goals here are:
* to make first startup significantly faster(no need for our users to generate startup cache)
* To make subsequent startups faster by loading more stuff from less files(ie only addons should exist in user-generated startup cache)

This turned out a lot hairier/hackier than it seemed. For now I package xpcshell so I can use it to startupcache stuff out of omnijar. Our url-resolution code(ie for resource uris) is not exposed properly so for now I just hardcode paths in SimplifyURI.

I think this is the dirtiest patch I ever posted to bugzilla :)
Attachment #474869 - Attachment is obsolete: true
Great, keep going -- dirt will wash off.

Just a note that JM is not an AOT compiler and probably shouldn't become one. But there's a plan to build an AOT JS compiler for our chrome, starting with research intern work. Cc'ing gal.

/be
(In reply to comment #16)
> Great, keep going -- dirt will wash off.
> 
> Just a note that JM is not an AOT compiler and probably shouldn't become one.
> But there's a plan to build an AOT JS compiler for our chrome, starting with
> research intern work. Cc'ing gal.

I talked to dmandelin&friends about it, they said it's doable. It would also be nice to cache webpage JS in our disk cache. Seems like it would require similar infrastructure on JS side.
Would it be possible to also cache framescripts? Those are scripts loaded through the messagemanager, in Fennec (basically the parent process tells the child process, "load these scripts and run them in each tab").

We cache the results of compiling these scripts, as they are run once per tab, but improving initial startup as well would be great.
We have built a Narcissus-based infrastructure with SSA and a pretty strong type inference. We will try to bolt nanojit and/or llvm to this, with spidermonkey providing basic VM services. There will be some trade-offs (native code is less compact than bytecode or even JS source), but it will definitely help startup latency and most likely also performance. We are looking for Winter interns for this as we speak.
Depends on: 595924
Attached patch omnijar pgo startup cache (obsolete) — Splinter Review
Comments welcome. http://stage.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/tglek@mozilla.com-7add41e587d9/tryserver-win32/firefox-4.0b8pre.en-US.win32.zip is a sample build(run with -console) for debug spew
The idea is to pregenerate the startup cache without running firefox by Cu.import()ing various things in xpcshell. To do this we currently need a make packaged build so this increases pgo make packages to 3(from 2).

Only limitation is that currently sync is not part of this due to the funny resource://services-sync/ url. I tried to set that up in my xpcshell script, but it didn't take. I need to look at sync next.
Attachment #478108 - Attachment is obsolete: true
Depends on: 609785
Attached patch startup cache (obsolete) — Splinter Review
Run the part in client.mk manually after make package and then do make package again
Attachment #482445 - Attachment is obsolete: true
blocking2.0: --- → final+
Attached patch wip (obsolete) — Splinter Review
This rewrites resource:/// urls into resource://gre/. Also worked around some unzipping bugs, etc. Mostly finished just need to get rid of debugging printfs
Attachment #488592 - Attachment is obsolete: true
Attached patch caching omnijar (obsolete) — Splinter Review
Attachment #489621 - Attachment is obsolete: true
Attachment #489653 - Flags: review?(dwitte)
This patch effectively allows a clean browser install to skip cache generation. It saves somewhere > 150ms (out of ~3s) on my windows laptop.
Comment on attachment 489653 [details] [diff] [review]
caching omnijar

>diff --git a/browser/installer/precompile_cache.js b/browser/installer/precompile_cache.js

>+// see http://mxr.mozilla.org/mozilla-central/source/services/sync/Weave.js#76

See what? If this is still relevant, move it down to the part where you do the crypto bits?

>+function setenv(name, val) {
>+  try {
>+        var environment = Components.classes["@mozilla.org/process/environment;1"].
>+          getService(Components.interfaces.nsIEnvironment);
>+        environment.set(name, val);
>+  }
>+  catch(e) {
>+    displayError("setenv", e);
>+  }
>+}

Inconsistent indenting. Also, can you make your usage of displayError, print, and dump consistent? You use all three and it's not especially clear why.

>+// Check that files can be read from after closing zipreader

Is this meant to be a TODO?

>+function populate_omnijar() {

>+  // the sync part below doesn't work as smoothly

Not a very descriptive statement. Explain why?

>+function extract_files(jar, pattern, dest) {

>+      file.parent.create(1 /* Ci.nsIFile.DIRECTORY doesn't work*/, 0700);

Probably because you want DIRECTORY_TYPE. Also why not 0600? Probably doesn't need to be executable. ;)

>+    try {
>+      jar.extract(c, file);
>+      print("extracted "+file.path+":"+file.fileSize);
>+    }catch(e) {
>+      print("Failed to extract " + file.path);
>+      print(e)

Combine messages onto one line, like you do elsewhere?

Also, need more spaces on the catch(e) line.

>+function extract_jsloader_to_dist_bin() {

>+  // this should really be zipReader.extract(...), but that method is too broken

Comment on why?

>diff --git a/startupcache/StartupCache.cpp b/startupcache/StartupCache.cpp
>@@ -230,6 +240,15 @@ StartupCache::GetBuffer(const char* id, 

>+  if (mozilla::OmnijarReader()) {

A little comment here about how we've prestashed entries in the omnijar and we're falling back to pulling them from it?

Looks good, r=dwitte with fixes.
Attachment #489653 - Flags: review?(dwitte) → review+
Oh, also, needs a test. Should be pretty simple to write -- maybe an xpcshell to open omni.jar and check a few resources inside it? And maybe even instantiate SC and ask it for an entry? r=dwitte is conditional on a test.
- This desperately needs some comments on 1. what it does and 2. why
  and 3. when.
- What is this new env var MOZ_STARTUP_CACHE for? When is it used and
  for what purpose? Please document it.
- print("Failed to extract " + file.path);
  And then? What happens in that case? Will the non-cached files be used?
  Please make a comment.
Would be nice, but does not block. We need to ship.
blocking2.0: final+ → -
(In reply to comment #27)
> - This desperately needs some comments on 1. what it does and 2. why
>   and 3. when.
> - What is this new env var MOZ_STARTUP_CACHE for? When is it used and
>   for what purpose? Please document it.

It's used to specify an alternate cache file. It's especially useful when running from xpcshell when there is no cache by default.

> - print("Failed to extract " + file.path);
>   And then? What happens in that case? Will the non-cached files be used?
>   Please make a comment.

Right, non-cached files are used.

I'll add these comments in the code.
This disables cache generation for Firefox. This is a good footprint/startup win on mobile, would like to land it so mobile can start using it.

Will keep it disabled for ff4 since it increases installer size. Should fix js serialization to be more compact(ie use a string table), then this will be a win in all dimensions.
Attachment #489653 - Attachment is obsolete: true
Attachment #502041 - Flags: approval2.0?
Attachment #502041 - Flags: approval2.0? → approval2.0+
http://hg.mozilla.org/mozilla-central/rev/80dcb82a0cb7
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Blocks: FF2SM
Target Milestone: --- → mozilla2.0b9
Version: unspecified → Trunk
Blocks: 633645
If I modify a js file in omnijar, what's the easiest way to generate the compiled binary for it?  Is it necessary or sensible to recompile?  Just deleting the binary seems to work.  Is there a perf hit?  Only on startup?  Negligible?
(In reply to comment #33)
> If I modify a js file in omnijar, what's the easiest way to generate the
> compiled binary for it?  Is it necessary or sensible to recompile?  Just
> deleting the binary seems to work.  Is there a perf hit?  Only on startup? 
> Negligible?

It just means the cached entry will be recreated in startup cache. Perf hit would be pretty small
How to turn on this feature during building?
(In reply to Taras Glek (:taras) from comment #34)
> (In reply to comment #33)
> > If I modify a js file in omnijar, what's the easiest way to generate the
> > compiled binary for it?  Is it necessary or sensible to recompile?  Just
> > deleting the binary seems to work.  Is there a perf hit?  Only on startup? 
> > Negligible?
> 
> It just means the cached entry will be recreated in startup cache. Perf hit
> would be pretty small

I noticed that the startup cache is not automatically invalidated.  If the js in omni.jar changes, or even if a new unmodified omni.jar is installed with its own binary, Fx will continue to use the old startup cached binary (until manually purged)

Is that expected behavior?  Shouldn't there be some kind of invalidation?
(In reply to al_9x from comment #36)
> (In reply to Taras Glek (:taras) from comment #34)
> > (In reply to comment #33)
> > > If I modify a js file in omnijar, what's the easiest way to generate the
> > > compiled binary for it?  Is it necessary or sensible to recompile?  Just
> > > deleting the binary seems to work.  Is there a perf hit?  Only on startup? 
> > > Negligible?
> > 
> > It just means the cached entry will be recreated in startup cache. Perf hit
> > would be pretty small
> 
> I noticed that the startup cache is not automatically invalidated.  If the
> js in omni.jar changes, or even if a new unmodified omni.jar is installed
> with its own binary, Fx will continue to use the old startup cached binary
> (until manually purged)
> 
> Is that expected behavior?  Shouldn't there be some kind of invalidation?

We don't really support people messing with js in omni.jar. That said, the perf impact of comparing omnijar modtime with the startup cache modtime is probably insignificant. There's just little incentive to support such a thing.
(In reply to Michael Wu [:mwu] from comment #37)
> We don't really support people messing with js in omni.jar. That said, the
> perf impact of comparing omnijar modtime with the startup cache modtime is
> probably insignificant. There's just little incentive to support such a
> thing.

I've been annoyed several times by the invalidation not happening, when fiddling with modules on a release build. Arguably,  I could have used a "flat chrome", but sometimes it's just easier to edit a release build.
No longer blocks: FF2SM
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: