1518203 - Investigate creating warmup service that can warm up critical files for Firefox launch on OS startup

Status: NEW

(Firefox :: General, enhancement, P3)

Description

[Alex Thayer [:alexical] (she/her)]

For the purposes of improving Firefox startup performance (or at least its perceived performance), we want to create a proof of concept for a Windows service which runs on startup which would open and read a list of the most critical files that Firefox needs on startup. In theory this could dramatically improve Firefox's perceived startup performance, and even potentially reduce the overall time from PC power button pressed to Firefox being up and running, since we might be able to better fit into existing gaps in disk utilization.

Having a service like this which runs on OS startup could in theory be useful for other purposes, like maintaining resources needed during Firefox startup which we need to update via the network, but for right now we are just concerned with the potential performance wins.

Comment 1

[Alex Thayer [:alexical] (she/her)]

I created a simple service to test this which just maps the file to memory and reads a byte from every page. The results weren't night and day, but there may have been an improvement. There's a bit too much noise to tell. Unfortunately it really doesn't seem to cut down on the cost of reading xul.dll, which seems to be the biggest offender if I'm reading Procmon's durations right[1]. In any case, I ended up taking a look at RAMMap's measures of things to get an understanding. After OS startup, before launching Firefox, RAMMap shows xul.dll as being in the cache, using up 100KB, which is the size of Nightly's xul.dll. However, after launching firefox, it seems like that cache entry is blown away, and a new one using only 44KB is there in its place. I'm wondering if windows handles the caching of files loaded as libraries differently, caching some processed form of it rather than the raw file contents. In that case, I would need to make the service call LoadLibrary on the dll files rather than simply mapping and reading the file. However, this sounds to me like it might have security implications. I'm going to at first try the LoadLibrary approach though and see if it cuts the time down any.

[1] xul.dll registers a total of around 120 seconds in ReadFile, which is longer than the interval that I'm recording. There is some overlap across processes, but a single process registers 52 seconds in ReadFile, which is certainly longer than it took to start up but right around the duration of the recorded interval.

Comment 2

[Alex Thayer [:alexical] (she/her)]

Calling LoadLibraryA seems to just cache things separately. Looking in RAMMap, the footprint of xul.dll doubles after loading it from both firefox.exe and my test.exe. If this is true, I think our only option for preloading xul.dll might be to simply run firefox.exe in some kind of inert mode that prevents it from doing anything substantial. Possibly headless mode but I think that's likely more than we're looking for.

Comment 3

[(No longer employed by Mozilla) Aaron Klotz]

You may also want to investigate PrefetchVirtualMemory. https://docs.microsoft.com/en-us/windows/desktop/api/memoryapi/nf-memoryapi-prefetchvirtualmemory

Comment 4

[Alex Thayer [:alexical] (she/her)]

Attachment: times.png

Some fun results to share. This is a graph of me putting a 2017 reference device into a special hell where it loads firefox, records its startup times, and then restarts again for eternity. Blue is time till painting the blank window, red is first paint. The dip you see in the graph is when I enabled the service. (The weird noisy chunk right before that is when I enabled the service with a bug in it.)

What the service does right now is it just executes "firefox.exe -warmup". "-warmup" is a flag I added to an opt build of Firefox which has it just load a few dlls and then immediately exit.

Right now this is just a proof of concept - we'll need to figure out all the implications of doing this, but it's at least some solid evidence that something like this could noticeably improve the startup experience for users.

Comment 5

[Laurențiu Nicola]

Is this really a good idea? Warming a cache at startup seems to save 5 seconds or so of IO and makes Firefox appear to start faster.

But on the other hand, that's some 5 seconds of IO that's taken away from the rest of the system. Especially on low-end computers, the first couple of minutes after startup are especially painful, and implementing this for Firefox will only contribute to that.

It reminds me of the "quick starters" that larger apps like OpenOffice used to have (although those remained resident).

Comment 6

[Alex Thayer [:alexical] (she/her)]

(In reply to Laurentiu Nicola from comment #5)

Is this really a good idea? Warming a cache at startup seems to save 5 seconds or so of IO and makes Firefox appear to start faster.

But on the other hand, that's some 5 seconds of IO that's taken away from the rest of the system. Especially on low-end computers, the first couple of minutes after startup are especially painful, and implementing this for Firefox will only contribute to that.

It reminds me of the "quick starters" that larger apps like OpenOffice used to have (although those remained resident).

Yeah, outside of this bug I've had a few conversations about that, and that's a large part of what I'm talking about when I say "we need to figure out all the implications of doing this." For one, I want to measure the impact that this has on the time until the machine is started up and basically usable. I haven't sorted out a good way to measure this yet - one idea might be to have the machine open, say, MS Word right after startup and measure the time taken from power button pressed to a Word window open. If it doesn't impact that measure then I think it's likely we could put the question to bed.

However if it does impact that measure then we need to have subtler conversations, and I think it will depend on exactly how much impact we're having on overall startup vs how much benefit we see for Firefox startup. If Firefox is always the first application a user opens, then it still makes sense to run this, since in the worst case it still consolidates the time that the user is waiting. I.e., they would wait 2 minutes for their machine to start up and 5 seconds for Firefox to start up, rather than 1 minute and 55 seconds for their machine to start up and 10 seconds for Firefox to start up. I would argue that it feels more frustrating to have two long waits than one longer wait and one short wait, if that makes sense.

Comment 7

[Eric Shepherd [:sheppy]]

In addition to checking on the impact this would have on an older machine, it would be useful to check the impact on a more recent machine that's already set up to preload or automatically launch a bunch of stuff at boot. Adding more things to the "automatically start this up" list can grind a machine to a near halt, even if it's got a ton of cores and a decently fast storage system with a lot of memory.

And it should be something that is not difficult to turn off if the user finds that boot performance is harmed by it.

Comment 8

[Alex Thayer [:alexical] (she/her)]

(In reply to Eric Shepherd [:sheppy] from comment #7)

In addition to checking on the impact this would have on an older machine, it would be useful to check the impact on a more recent machine that's already set up to preload or automatically launch a bunch of stuff at boot. Adding more things to the "automatically start this up" list can grind a machine to a near halt, even if it's got a ton of cores and a decently fast storage system with a lot of memory.

Could you clarify by what mechanism it's bogging the newer machine down in a way that it wouldn't with an older machine? Or maybe an example setup where adding additional IO to start up costs a disproportionate amount? I assume we're using newer/older here as a proxy for higher/lower end hardware, though, yes? Since our reference hardware is not old, just low-end.

And it should be something that is not difficult to turn off if the user finds that boot performance is harmed by it.

Absolutely. I get very angry when programs try to do "smart" things on my behalf which I don't like and have to jump through a thousand hoops to disable. In my opinion we should have an in-Firefox setting for disabling it, without forcing the user to go into their Windows Services list and manually disable it. Additionally though it's on my wishlist to ensure that we're not running this code if the user doesn't typically open Firefox soon after starting up their machine. If Firefox is installed but the user only runs it occasionally I don't want to give them a bad image of us by eating up even a second of their startup time. I haven't sorted out a mechanism for this, and I think there's some kinks that would have to be worked out for us to effectively disable ourselves, but it's definitely a goal of mine.

Comment 9

[Alex Thayer [:alexical] (she/her)]

The current measurements for startup impact, as measured by time to start Chrome, are:

• Median: 75687ms with service, 72890ms without, 2797ms difference
• 95th percentile: 84812ms with service, 82788ms without, 2024ms difference
• 5th percentile: 61846ms with service, 58448ms without, 3398ms difference

So, ballpark 2-3s extra Windows startup time for 5s less Firefox startup time. To me this sounds like a good trade?

Accordingly my current plan is to try to get this cleaned up to be landable in tree, and to try to integrate with the installer/updater(?), in order to trial it on Nightly and collect metrics. Before I do that I need to sort out the security implications and whether it's possible to install a service through the updater.

Regarding the latter question, rstrong, do we have sufficient privileges when updating in Windows to install a service on the user's machine? I would think that we do in the common case since we're writing into Program Files but I wanted to double check. And while we're here, do you have any objections to going down that route?

Comment 10

[Robert Strong (they/them - no direct email)]

When we created the maintenance service it was intentionally separated from app update so it could be used for things such as this and at one point it was used for a separate project to improve startup by changing the Firefox related Windows prefetch files though that was later removed since it didn't improve perf.

Any reason not to go this route?

Comment 11

[Robert Strong (they/them - no direct email)]

BTW: we don't always have the privileges to do this via the installer or the updater. The installer always tries to elevate when installing but the user can choose to install without elevating so it is possible to install Firefox on systems such as kiosks, computers in libraries, etc. The updater only uses the service or requests elevation if it needs to so if it can update without additional privileges it will.

Comment 12

[Alex Thayer [:alexical] (she/her)]

(In reply to Robert Strong (Robert he/him) [:rstrong] (use needinfo to contact me) from comment #10)

When we created the maintenance service it was intentionally separated from app update so it could be used for things such as this and at one point it was used for a separate project to improve startup by changing the Firefox related Windows prefetch files though that was later removed since it didn't improve perf.

Any reason not to go this route?

I tried to go down the route of mucking with the prefetch files since they didn't include, among other things, xul.dll, but that didn't improve performance, and I couldn't sort out a way by playing with the files to run the prefetch immediately on boot. If you're aware of a way to do so, please do let me know, but it sounds like the previous project also hit a dead-end around here?

Comment 13

[Robert Strong (they/them - no direct email)]

Not the route of mucking with the prefetch files... the route of using the existing maintenance service.

Comment 14

[Alex Thayer [:alexical] (she/her)]

(In reply to Robert Strong (Robert he/him) [:rstrong] (use needinfo to contact me) from comment #13)

Not the route of mucking with the prefetch files... the route of using the existing maintenance service.

Ah, yeah that sounds fine for me.

Comment 15

[Robert Strong (they/them - no direct email)]

If you use the maintenance service then everything related to your questions in comment #9 should already handled as best as they can be.

Comment 16

[Laurențiu Nicola]

How would multiple versions of the browser be handled? Imagine having Release, Beta and Nightly installed.

Comment 17

[Mike Conley (:mconley) (:⚙️)]

Setting performance impact to high because of dthayer's findings in comment 9 (although I'd be curious to get some more modern measurements).

I wonder if the newish background task mode of Firefox could be used in this way?

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is set to S3. However, the Performance Impact field flags this bug as having a high impact on the performance.
:mossop, could you consider increasing the severity of this performance-impacting bug? Alternatively, if you think the performance impact is lower than previously assessed, could you request a re-triage from the performance team by setting the Performance Impact flag to ??

For more information, please visit BugBot documentation.

Comment 19

[Mike Conley (:mconley) (:⚙️)]

As this is an enhancement request, I don't think a severity level actually makes sense on this bug.