696095 - Create Fennec startupCache at build time

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: General, defect, P3)

Description

[Geoff Brown [:gbrown]]

Create the Fennec startupCache at build time rather than run-time.

From email:

On October 19, 2011 10:20:28 AM EDT, Mike Hommey wrote:

On Wed, Oct 19, 2011 at 10:10:41AM -0400, Doug Turner wrote:


    On Oct 19, 2011, at 9:48 AM, Taras Glek wrote:


        On 10/19/2011 2:19 AM, Mike Hommey wrote:

            On Tue, Oct 18, 2011 at 10:35:13PM -0400, Doug Turner wrote:

                Mike,


                I heard that we can get some small improvement if we constructed the startup cache during the build process instead of constructing it at first run.  Is this true?  If so, how do we proceed?

            There are essentially two ways to do that:

            - Generate it with xpcshell on a device

            - Generate it with xpcshell on the build host


            The latter requires building a host xpcshell or grabbing one from the

            archive, or using qemu-arm-static. I don't know if the latter actually

            works, but if it does, it could very much simplify things.

        I think at the moment the files are portable from x86 to arm since both are little endian, so no need for funny cross-device games.


    If true, how do I try this out?


You need to do something similar to the GENERATE_CACHE sequence in
toolkit/mozapps/installer/packager.mk, but instead of omni.jar, you
probably need to use the apk, and instead of xpcshell from the build
directory, an x86 xpcshell.

Mike

Comment 1

[Geoff Brown [:gbrown]]

It looks like GENERATE_CACHE in packager.mk can be used, with minor modifications, to create a Fennec startupCache at build time. GENERATE_CACHE invokes xpcshell; an arm, or possibly x86, xpcshell is required. 

Ideas are:
1. execute an arm xpcshell on the build host via qemu emulation
2. build a host xpcshell and execute on the build host (perhaps require a parallel objdir-x86, or something like that)
3. run arm xpcshell on device (require device connection at build time -- seems impractical)

Comment 2

[Geoff Brown [:gbrown]]

qemu-arm-static can easily run a statically linked ARM executable on our build host...but xpcshell is dynamically linked!

Comment 3

[Geoff Brown [:gbrown]]

The difficulties with running dynamically linked executables with qemu are illustrated here: https://wiki.edubuntu.org/UbuntuDevelopment/Ports. We would need to use "full system emulation", which seems an impractical complication to our build system.

Comment 4

[Doug Turner (:dougt)]

dynamically linked to what?  is it just xpcom or something?  we could just package x86 bits up, and use that binary in the build process?

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Doug Turner (:dougt) from comment #4)
> dynamically linked to what?  is it just xpcom or something?  we could just
> package x86 bits up, and use that binary in the build process?

creating a x86 xpcshell essentially means building all of gecko for x86. The more sensible thing in my mind is to create a statically linked xpcshell which we can run with qemu-arm-static.

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Brad Lassey [:blassey] from comment #5)
> creating a x86 xpcshell essentially means building all of gecko for x86. The
> more sensible thing in my mind is to create a statically linked xpcshell
> which we can run with qemu-arm-static.

A static xpcshell requires a few static libs that we don't normally generate: 
libmozutils.a
libxpcom.a
libmozalloc.a
libxul.a

More troubling, there is a dependency on liblog.so and there is no static version of this in the NDK.

Comment 7

[Geoff Brown [:gbrown]]

Another complication: Once we generate a startup cache, how can it be installed, and updated, in the profile? (see also bug 695145).

Comment 8

[Geoff Brown [:gbrown]]

Attachment: patch to generate startupCache at build time and include in APK (using MOZ_HOST_BIN)

I can build a startupCache.4.little file at build (package) time using this patch. If a MOZ_HOST_BIN environment variable is set, "make package" includes a step to generate a startupCache from the files in <objdir-android>/dist/bin/components and <objdir-android>/dist/bin/modules. MOZ_HOST_BIN should point to a directory containing an xpcshell (and required libraries) that can execute on the build host, eg. objdir-x86/dist/bin. The startupCache.4.little file is included in the APK. This adds about 1 MB to the APK.

I have verified that manually extracting the startupCache file from the APK and moving it to the <profile>/startupCache directory enables startupCache lookups from the file.

Comment 9

[Geoff Brown [:gbrown]]

Attachment: patch to generate startupCache at build time and include in APK (using MOZ_HOST_BIN)

Now I see that the StartupCache class already makes allowance for pre-built cache files using the APK: If not found in the startupCache.4.little file, the GRE and/or APP omnijar (APK on Android) is searched. This simplifies my task and addresses my concerns about installing the pre-built cache at first run time. This patch simply adds the files to the APK with a jsloader/resource/gre path; StartupCache should automatically find the files in the APK.

Comment 10

[Geoff Brown [:gbrown]]

Attachment: patch to generate startupCache at build time and include in APK (using MOZ_HOST_BIN)

This works fine for me on local builds with $MOZ_HOST_BIN set to a regular Linux desktop build (eg. objdir-firefox/dist/bin). Will this strategy work for nightly builds, etc? How would I set MOZ_HOST_BIN to enable this feature in nightlies?

Comment 11

[(not currently active) Ted Mielczarek]

It's possible we can use this for nightly builds, but we'll have to do some more work to use it. The build machines aren't guaranteed to have a desktop build of Firefox laying around (although they probably do, since the same slaves build desktop and mobile Firefox), so we'll probably have to put a package somewhere that the slaves can download, unpack, and use for this step.

Comment 12

[Geoff Brown [:gbrown]]

The package would "only" require xpcshell and its dependent libs:

libmozalloc.so
libmozsqlite3.so
libnspr4.so
libnss3.so
libnssutil3.so
libplc4.so
libplds4.so
libsmime3.so
libssl3.so
libxpcom.so
libxul.so
xpcshell

Comment 13

[(not currently active) Ted Mielczarek]

Are we going to run into issues if the desktop xpcshell we use isn't precisely in sync with the Fennec version we're running it for? Presumably if we ever tweak the startupCache format we'd be in trouble.

Comment 14

[Geoff Brown [:gbrown]]

I think this sort of incompatibility problem would be rare, but if it did happen, it would be a terrible thing to debug!

In a recent irc exchange, cjones noted that b2g uses qemu, with images. With an arm image for qemu, the freshly-built arm xpcshell could be used to build the startupCache, eliminating incompatibility concerns.

Setting this up on the build machines is outside of my expertise and experience -- perhaps I should hand-off this bug to someone else?

Comment 15

[(not currently active) Ted Mielczarek]

How about you coordinate with someone who can do that, and we can figure out how to make it work in our existing setup, and if that's successful we can file a bug to get qemu set up on the build machines.

Comment 16

[(not currently active) Ted Mielczarek]

Comment on attachment 572026 [details] [diff] [review]
patch to generate startupCache at build time and include in APK (using MOZ_HOST_BIN)

Review of attachment 572026 [details] [diff] [review]:
-----------------------------------------------------------------

I'm going to clear this review for now, since I think everyone agreed that running the ARM xpcshell in an emulator was the way to go.

Comment 17

[Geoff Brown [:gbrown]]

I'm finally getting back to this now....

I downloaded and built the b2g repository and found that the emu.sh script there very handily starts an emulator with the built images. By adding a -no-window argument, the emulator can be run in a "headless" manner. Then I can use adb commands to push xpcshell, run xpcshell, and pull back the startup cache. I'm sorting out the xpcshell command line in the new environment now, but this approach looks like it will work.

Comment 18

[Geoff Brown [:gbrown]]

Attachment: shell script that runs arm xpcshell in emulator to generate startup cache

I can generate a startup cache and update the fennec APK using this script. The script relies on the emulator and images found in a B2G build, and the adb utility from the Android SDK. 

This takes about 75 seconds to execute, which breaks down approximately like this:
20 s - start the emulator and wait for it to boot
35 s - push xpcshell, fennec APK, and support files to emulator
15 s - execute xpcshell to generate the startup cache
5 s - pull back the updated fennec APK and kill the emulator

The startup cache adds about 800K to the size of the APK.

Does this seem promising? Where do we go from here?

Comment 19

[(not currently active) Ted Mielczarek]

Comment on attachment 592323 [details]
shell script that runs arm xpcshell in emulator to generate startup cache

This definitely looks promising! I'll leave it to other mobile folks to comment on the size/speed tradeoff.

In terms of what we'd need to do to get this into official builds, the first step would be to file a bug about getting the B2G build environment (or at least the emulator part) onto our Linux build slaves. Then we'd need some machinery in configure to specify where that lives (--with-qemu? --with-android-emulator? I don't know.) Then we'd need to either take the existing "populate startupcache" routine:
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#460
and modify it to support pushing things to the emulator and running that way, or just write a separate script to do it. I'd lean towards the former, to keep desktop and mobile in sync. We could structure it like the test harnesses, with a normal and remote path.

Comment 20

[Mike Hommey [:glandium]]

At some point, we'll need to use valgrind on arm as well, and that is not going to be workable on the emulator. So it might be better to be able to do such things directly with a real device.

Comment 21

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #20)
> At some point, we'll need to use valgrind on arm as well

Or fennec itself, for that matter, for PGO.

Comment 22

[Geoff Brown [:gbrown]]

Attachment: patch to generate fennec startup cache at package time via emulator

This basically moves the previous script into packager.mk. It also adds the ability to "trim" the startup cache to a list of files found in a manifest.

It's still rough, but works for me.

Comment 23

[Geoff Brown [:gbrown]]

I have held off on opening a bug for the b2g environment while I looked into the performance gain achieved by the pre-built startup cache. Unfortunately, I have not been able to identify any performance gain.

I measured first-run fennec startup time (first run after installation) with/without a pre-built startup cache on a Samsung Galaxy S. I took the mean time for 10 runs of each: The mean startup time with the pre-built startup cache was about 200 ms greater than the mean startup time without a pre-built startup cache. 

I thought the result might be caused by the increase in APK size, so removed all the startup cache files except those that I determined were accessed during startup. I repeated the experiment and got the same result: Even with a customized, minimal pre-built startup cache containing only those files resulting in startup cache hits during startup, the mean startup time increases by about 200 ms. (This new APK is only 260K larger than normal.)

I cannot account for the performance regression -- ideas welcome!

Comment 24

[Geoff Brown [:gbrown]]

I repeated startup tests using Talos and a new build, running 50 cycles for each configuration. These tests showed slightly less performance degradation: mean startup time with a pre-built startup cache was about the same as without. Tests were repeated with some variations:
 - full startup cache, normal compression
 - startup cache trimmed to those files known to be hit during startup, normal compression
 - startup cache trimmed to those files known to be hit during startup, no compression

All the results were about the same: no improvement in Ts.


At this point then, we have the proof of concept for a method for adding a pre-built startup cache to the APK using the android emulator and B2G images, but no known performance improvement. I cannot justify investigating this bug further.

Comment 25

[Brad Lassey [:blassey] (use needinfo?)]

Geoff, thanks for working through this. CC'ing Taras so he's aware of your results and can look into it if he wants.

Comment 26

[(dormant account)]

Geoff, was this using the nested omnijar? Android apk code gets slow when the .apk has too many entries in it.

Comment 27

[Geoff Brown [:gbrown]]

(In reply to Taras Glek (:taras) from comment #26)

Startup cache entries (eg. jsloader/resource/gre/components/nsPrompter.js, etc) were added to omni.ja, the existing omnijar in the apk.

Comment 28

[Geoff Brown [:gbrown]]

Didn't mean to change the status there....

Comment 29

[Nathan Froyd [:froydnj]]

Vladan, can you find somebody to see if things have changed over the last 1.5 years?

Comment 30

[Nathan Froyd [:froydnj]]

So, to try to get all the relevant information into the bug as to why we are reopening this: Brad mentioned that there were some startup cache changes which might make this more worthwhile.  Bad, what specific changes were you thinking of?

Maybe the passage of time has changed things regardless.

Geoff, how hard is it to get a setup for this to experiment with locally?  Is just building all the B2G bits sufficient to get all the pieces one needs?

Comment 31

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Nathan Froyd (:froydnj) from comment #30)
> So, to try to get all the relevant information into the bug as to why we are
> reopening this: Brad mentioned that there were some startup cache changes
> which might make this more worthwhile.  Bad, what specific changes were you
> thinking of?
The networking team has been examining the cache in general.
> Maybe the passage of time has changed things regardless.
This is more the point. We looked at this a long time ago. Things may have changed since then, so I wouldn't rule out finding a win here.

> Geoff, how hard is it to get a setup for this to experiment with locally? 
> Is just building all the B2G bits sufficient to get all the pieces one needs?

Comment 32

[Nathan Froyd [:froydnj]]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #31)
> (In reply to Nathan Froyd (:froydnj) from comment #30)
> > So, to try to get all the relevant information into the bug as to why we are
> > reopening this: Brad mentioned that there were some startup cache changes
> > which might make this more worthwhile.  Bad, what specific changes were you
> > thinking of?
> The networking team has been examining the cache in general.

Ah, so, the startup cache is completely distinct from the networking cache.  No interrelation between the two.  (Although maybe the networking cache would be an ok place to store startup-cachy things nowadays?)

Comment 33

[Geoff Brown [:gbrown]]

(In reply to Nathan Froyd (:froydnj) from comment #30)
> Geoff, how hard is it to get a setup for this to experiment with locally? 
> Is just building all the B2G bits sufficient to get all the pieces one needs?

As far as I can remember (this was almost 2 years ago!), it is pretty easy to set this up locally -- you just need to get the B2G bits in place, have adb in your path, and run the script.

Comment 34

[Richard Newman [:rnewman]]

nfroyd: did you do any more investigation in this?

We're interested in this less for the performance improvement (though that would be nice), but as a way to eliminate some source JS from the APK altogether.

Comment 35

[Nathan Froyd [:froydnj]]

(In reply to Richard Newman [:rnewman] from comment #34)
> nfroyd: did you do any more investigation in this?

No, I did not.

Comment 36

[Rachelle Yang [:ryang][[e-mail]]]

This bug is old and no more progress for more than 2 years, so I am removing the tracking flag now.
feel free to renominate. 

Thank you !