1358927 - Initializing gfxPlatform to get telemetry data is quite slow

Status: REOPENED

(Core :: Graphics, defect, P3)

Description

[Kris Maglione [:kmag]]

On my system, it spends about 15ms at startup just in getGfxField calls. It looks like all of those getters just copy or truncate strings, so I guess this is either just XPConnect overhead, or it takes a surprisingly long time to parse the glxinfo process data on the first call.

Comment 1

[Kris Maglione [:kmag]]

Looking at the C++ stack more closely, it looks like most of this is actually in initializing gfxPlatform, which needs to happen anyway. There's only about 2-3ms of additional overhead.

Comment 2

[Florian Quèze [:florian]]

(In reply to Kris Maglione [:kmag] from comment #1)
> Looking at the C++ stack more closely, it looks like most of this is
> actually in initializing gfxPlatform, which needs to happen anyway.

Could it happen on a different thread without blocking the rest of startup?

getGfxField is 10ms on my Windows startup profile: https://perfht.ml/2pUofVu and 30ms on my Mac startup profile: https://perfht.ml/2pUHVZk (both taken on very fast modern hardware)

gfxPlatformFontList::InitFontList() is 63% of it on my mac profile (20% on Windows).

Do we really need a whole list of fonts from the platform before continuing startup?

Comment 3

[Kris Maglione [:kmag]]

Yeah, initializing the font list seemed to be the slowest part for me, too.

I suspect we do need it before we can show a window, but doing it on a different thread might be a possibility.

Comment 4

[Florian Quèze [:florian]]

(In reply to Kris Maglione [:kmag] from comment #3)

> I suspect we do need it before we can show a window,

When I asked "Do we really need a whole list of fonts" I was wondering if we could instead just check that the 2-3 fonts that are going to be used by the browser window exist (but maybe that question just shows my complete ignorance of how we handle fonts)

Comment 5

[Kris Maglione [:kmag]]

Font matching can be quite complicated, and you often need the entire list of available fonts before you know which ones you'll actually be using, or where they're stored. But I don't know much about how the font list is actually used in our graphics subsystem.

Comment 6

[Florian Quèze [:florian]]

I also wonder how much it matters to have telemetry record data during early startup, and if it's worth the 10+% of my Windows startup profile that we spend there. Eg. It spends 11ms to load the search service (and getting no actual data out of this, because it will initialize asynchronously).

Comment 7

[Florian Quèze [:florian]]

(In reply to Kris Maglione [:kmag] from comment #5)
> Font matching can be quite complicated, and you often need the entire list
> of available fonts before you know which ones you'll actually be using

For web pages in general, sure. But could we ensure our primary UI (ie. the XUL of the browser window outside of the content area) uses a very small set of fonts that are very clearly selected?

Comment 8

[Kris Maglione [:kmag]]

(In reply to Florian Quèze [:florian] [:flo] from comment #6)
> I also wonder how much it matters to have telemetry record data during early
> startup, and if it's worth the 10+% of my Windows startup profile that we
> spend there. Eg. It spends 11ms to load the search service (and getting no
> actual data out of this, because it will initialize asynchronously).

Yeah, I've been investigating other ways to speed that up, e.g., bug 1358907. I also found the search service part suspicious.

(In reply to Florian Quèze [:florian] [:flo] from comment #7)
> For web pages in general, sure. But could we ensure our primary UI (ie. the
> XUL of the browser window outside of the content area) uses a very small set
> of fonts that are very clearly selected?

I don't think so, unless we bundled all of the fonts with the browser. We use a combination of system fonts and CSS font matching. We need the font list for the CSS font matching, but even if we knew the specific system fonts we were going to use, and tried to circumvent the CSS font matching, we'd still need to load the system font list to know where to find them.

Comment 9

[Jonathan Kew [:jfkthame]]

(In reply to Florian Quèze [:florian] [:flo] from comment #7)
> (In reply to Kris Maglione [:kmag] from comment #5)
> > Font matching can be quite complicated, and you often need the entire list
> > of available fonts before you know which ones you'll actually be using
> 
> For web pages in general, sure. But could we ensure our primary UI (ie. the
> XUL of the browser window outside of the content area) uses a very small set
> of fonts that are very clearly selected?

For a single localization of the browser, maybe so (though even then there are edge cases that might be tricky, such as if an add-on can insert extra bits of UI that might require "unexpected" fonts, or if the platform supports user-configurable "themes" that might change the default UI fonts). It gets harder once you also consider localizations that may use entirely different scripts (and hence fall back to different fonts) for the UI text.

(In reply to Kris Maglione [:kmag] from comment #3)
> Yeah, initializing the font list seemed to be the slowest part for me, too.
> 
> I suspect we do need it before we can show a window, but doing it on a
> different thread might be a possibility.

I've experimented a bit with this in the past (several years ago, IIRC), and at that time I wasn't able to see much benefit. I think one issue may have been that the font-list initialization thread just ended up competing with the main thread for file i/o anyway. But it might be worth re-visiting this, as no doubt lots has changed in the meantime.

Comment 10

[Georg Fritzsche [:gfritzsche]]

If we would move querying the graphics info from Telemetry to a later time, would this meaningfully impact startup times?
If yes i'm happy to check into this in a separate bug.

Comment 11

[Kris Maglione [:kmag]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #10)
> If we would move querying the graphics info from Telemetry to a later time,
> would this meaningfully impact startup times?
> If yes i'm happy to check into this in a separate bug.

I don't think so, which is why I initially closed it as invalid. Most of the time here is in initializing the graphics platform code, and we need to do that before showing a window, regardless.

Comment 12

[Florian Quèze [:florian]]

On Windows it seems some of this startup cost is already/also paid at http://searchfox.org/mozilla-central/rev/ce5ccb6a8ca803271c946ccb8b43b7e7d8b64e7a/toolkit/components/gfx/SanityTest.js#243

Comment 13

[Florian Quèze [:florian]]

Here is another profile captured today on the reference hardware: https://perfht.ml/2tkYkXV

This represents a large part of cold startup (about 9% of the time before first paint in this profile).

The code doing the expensive system call is http://searchfox.org/mozilla-central/rev/5e1e8d2f244bd8c210a578ff1f65c3b720efe34e/gfx/thebes/gfxDWriteFontList.cpp#1860

Comment 14

[(no longer active)]

This code is the bundled font loading support which was added in bug 998844.  As far as I can tell, the only bundled fonts that we currently ship is the EmojiOneMozilla.ttf font: https://searchfox.org/mozilla-central/source/browser/fonts/moz.build#9

Do we need this font to be available at startup?

Comment 15

[Mike de Boer [:mikedeboer]]

(In reply to :Ehsan Akhgari (Away 7/10-7/17; needinfo please, extremely long backlog) from comment #14)
> Do we need this font to be available at startup?

Jonathan can and should probably answer this, but the answer is no.

Comment 16

[Jonathan Kew [:jfkthame]]

Right, I don't think it's critical that the bundled emoji font is available at startup, so from that point of view we could defer loading it. One downside of doing that is that whenever we do eventually load the font, we'll force re-layout of everything (required if the list of available fonts changes), so we'll potentially do more work later.

The other issue, if we don't load the font until after the initial page load, is that users may see a page initially displayed with B/W emoji characters or (perhaps more likely, on old Windows systems) just missing glyphs, and then after a moment it gets refreshed and the proper emoji pop into place. If the user is restoring a session where their initially-displayed page is some kind of social media site or similar, with lots of usage of emoji, this might be quite disconcerting.

So....trade-offs. I'm not sure what's the best option here. Partly, it depends how we prioritize the completeness of the user experience on older Windows systems (which is why we're bundling an emoji font rather than just relying on what the platform offers) vs startup time on newer systems.

Comment 17

[Florian Quèze [:florian]]

(In reply to Jonathan Kew (:jfkthame) from comment #16)

Can we load the emoji font only in content processes so that the parent process starts faster?

Comment 18

[Jonathan Kew [:jfkthame]]

Yes, that might be possible. In principle it sounds trivial to implement, but I'm not sure offhand whether having a mismatch between the font lists known by the chrome and content processes will lead to any issues. Worth investigating a little, I think.

Comment 19

[Florian Quèze [:florian]]

Here is a cold startup profile on a slow Windows7 machine where gfxPlatformFontList::InitFontList takes 49% of the time until first paint: https://perfht.ml/2D5gjIp

Comment 20

[Jonathan Kew [:jfkthame]]

I don't think there's much we can do about this case (aside from not using dwrite at all, but is that even an option these days?). Of the time in gfxPlatformFontList::InitFontList, 99% is inside the call to GetSystemFontCollection, which results in dwrite scanning all the installed fonts and gathering whatever metadata it requires. My understanding is that this is a cost that will basically be paid by whatever process first tries to use dwrite after the system is booted; so if nothing else has already initialized dwrite, then it'll hit our startup.

It doesn't look like this is particularly related to our bundled emoji font (as discussed in earlier comments), as the big time sink is getting the standard system collection; the profile shows only 17ms in CreateBundledFontsCollection (for the emoji font), which is pretty insignificant compared to the 2.8 seconds(!) it takes to call GetSystemFontCollection.

Comment 21

[Florian Quèze [:florian]]

(In reply to Jonathan Kew (:jfkthame) from comment #20)
> I don't think there's much we can do about this case (aside from not using
> dwrite at all, but is that even an option these days?). Of the time in
> gfxPlatformFontList::InitFontList, 99% is inside the call to
> GetSystemFontCollection, which results in dwrite scanning all the installed
> fonts and gathering whatever metadata it requires.

Is it possible to use dwrite without calling GetSystemFontCollection first? Could we use specific fonts instead of getting the whole collection?

Can GetSystemFontCollection be called off main thread?

Could we initialize this after first paint (even if that means no text displayed at first paint)?

Comment 22

[Florian Quèze [:florian]]

(In reply to Florian Quèze [:florian] from comment #19)
> Here is a cold startup profile on a slow Windows7 machine where
> gfxPlatformFontList::InitFontList takes 49% of the time until first paint:
> https://perfht.ml/2D5gjIp

This was not a one time thing, I just captured 2 other Win7 cold startup profiles and they both show the same thing: 
https://perfht.ml/2ASV2ve
https://perfht.ml/2ASLkJy

Comment 23

[Jonathan Kew [:jfkthame]]

(In reply to Florian Quèze [:florian] from comment #21)
> (In reply to Jonathan Kew (:jfkthame) from comment #20)
> > I don't think there's much we can do about this case (aside from not using
> > dwrite at all, but is that even an option these days?). Of the time in
> > gfxPlatformFontList::InitFontList, 99% is inside the call to
> > GetSystemFontCollection, which results in dwrite scanning all the installed
> > fonts and gathering whatever metadata it requires.
> 
> Is it possible to use dwrite without calling GetSystemFontCollection first?
> Could we use specific fonts instead of getting the whole collection?

Presumably it would be possible to use specific fonts by loading them directly by file path, though that sounds a bit fragile; I'd also be concerned that if we avoid GetSystemFontCollection, we may still end up paying the same dwrite initialization cost on our first dwrite call, whatever it is. I think jdaggett looked into that a bit, several years back; there might be more info in some of the old dwrite-related bugs.

This seems like it would be a significant amount of complexity to add/maintain just for this case, though.

> Can GetSystemFontCollection be called off main thread?

AFAIK, that's probably OK to do, though it's unclear to me how much benefit it would have if the main thread ends up waiting for it to complete before it can do anything involving text measurement/rendering.

> Could we initialize this after first paint (even if that means no text
> displayed at first paint)?

A first paint with no text doesn't sound like a good option to me. That would just look broken.

Maybe what we need here is an old-fashioned "splash screen" that just lets the user know something is happening (and doesn't depend on dwrite in any way!): "Initializing the browser..." displayed via GDI?


(In reply to Florian Quèze [:florian] from comment #22)
> This was not a one time thing, I just captured 2 other Win7 cold startup
> profiles and they both show the same thing: 
> https://perfht.ml/2ASV2ve
> https://perfht.ml/2ASLkJy

I'm not surprised -- AIUI, dwrite initialization on win7 is just really slow. (Are these fully-updated win7 systems? I thought there had been a Windows KB update at one time that was supposed to improve this, at least somewhat. But it may still be pretty slow.)

Out of interest, how does warm startup compare? Does telemetry tell us what proportion of startups are cold?

Comment 24

[Florian Quèze [:florian]]

(In reply to Jonathan Kew (:jfkthame) from comment #23)

> > Can GetSystemFontCollection be called off main thread?
> 
> AFAIK, that's probably OK to do, though it's unclear to me how much benefit
> it would have if the main thread ends up waiting for it to complete before
> it can do anything involving text measurement/rendering.

That means everything else that happens on the main thread between this and when we need to paint for the first time could happen in parallel.

> > Could we initialize this after first paint (even if that means no text
> > displayed at first paint)?
> 
> A first paint with no text doesn't sound like a good option to me. That
> would just look broken.
> 
> Maybe what we need here is an old-fashioned "splash screen" that just lets
> the user know something is happening (and doesn't depend on dwrite in any
> way!): "Initializing the browser..." displayed via GDI?

The current plan is to display as soon as possible a normal window loading about:blank, and then when we are ready, to replace about:blank with browser.xul. This is similar to what Chrome does. I don't think we need fonts for about:blank.

> Out of interest, how does warm startup compare?

Warm startup paints a very different picture. Here are some warm startup profiles on the same machine:
https://perfht.ml/2AUbjAd
https://perfht.ml/2D5fKye
https://perfht.ml/2D69HcM
https://perfht.ml/2AT7BXw

> Does telemetry tell us what proportion of startups are cold?

Unfortunately, no. Cold and warm startup are both relatively artificial, although cold startup is possible if people turn on their computer with the intent of looking for something on the web, and start Firefox right after the computer boots. Warm startup is an almost impossible case: it assumes the user starts Firefox, then closes it, then restarts it without doing anything else in the middle.

If we could find a reliable way to tell if a startup is cold startup, I would like to collect this info in telemetry.
It's likely that most startups are neither warm nor cold, ie. Firefox has been used a little while ago, but other applications have been used between when Firefox was closed and when it's started again, so it's no longer guaranteed that all the files Firefox cares about are in disk cache.

Comment 25

[Kris Maglione [:kmag]]

(In reply to Jonathan Kew (:jfkthame) from comment #23)
> AFAIK, that's probably OK to do, though it's unclear to me how much benefit
> it would have if the main thread ends up waiting for it to complete before
> it can do anything involving text measurement/rendering.

The benefit of doing it off-main-thread is that we can begin initializing it
early, and hope that it can do its IO-bound work while the main thread is
blocked on CPU-bound work, and vice versa, and finish before we first need it.

It doesn't always work out that way (i.e., sometimes we do more harm than good
by thrashing a platter drive to death with competing IO requests), but so far,
most of my experience with moving startup tasks to background threads has been
positive.

> A first paint with no text doesn't sound like a good option to me. That
> would just look broken.
> 
> Maybe what we need here is an old-fashioned "splash screen" that just lets
> the user know something is happening (and doesn't depend on dwrite in any
> way!): "Initializing the browser..." displayed via GDI?

We've talked about showing a wireframe window as soon as possible at startup
(bug 1336227), and then painting the real browser UI as soon as we can after
that. I've noticed that splash screens are making a comeback lately, but they're
still pretty unpopular.

And I think developers tend to use them as an excuse to be a bit more lax about
their startup perf. Or, at least, I have the completely unscientific impression
that startup time for apps tends to get slower after they introduce splash
screens.

Comment 26

[Jonathan Kew [:jfkthame]]

(In reply to Florian Quèze [:florian] from comment #24)
> > Out of interest, how does warm startup compare?
> 
> Warm startup paints a very different picture. Here are some warm startup
> profiles on the same machine:
> https://perfht.ml/2AUbjAd
> https://perfht.ml/2D5fKye
> https://perfht.ml/2D69HcM
> https://perfht.ml/2AT7BXw
> 
> > Does telemetry tell us what proportion of startups are cold?
> 
> Unfortunately, no. Cold and warm startup are both relatively artificial,
> although cold startup is possible if people turn on their computer with the
> intent of looking for something on the web, and start Firefox right after
> the computer boots. Warm startup is an almost impossible case: it assumes
> the user starts Firefox, then closes it, then restarts it without doing
> anything else in the middle.

As far as DirectWrite is concerned, I think (if my understanding is correct) that the extremely slow GetSystemFontCollection time should only occur if we're the first DW-using process launched after the system starts up. If the system has been running for some time, including perhaps previous Firefox sessions as well as other applications, then even if everything Firefox-related is gone from cache, we should still see a much better GetSystemFontCollection time because DW will have already initialized its font collection, and all we're asking for is a reference to it. So by a "warm" startup I didn't really mean an immediate restart of Firefox, where as much as possible will be in cache, but just a startup where DirectWrite has already been initialized (by some other process, potentially).