1616268 - 2.02 - 48.64% raptor-tp6-apple|-tumblr|-wikipedia|-wikipedia|-wikipedia-firefox-cold (linux64|macosx1014-64|windows10-64|windows7-32|-shippable) regression on push 780285749c804b0b84c3607d362cd9905c286512 (Mon February 17 2020)

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=780285749c804b0b84c3607d362cd9905c286512

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

49% raptor-tp6-tumblr-firefox-cold loadtime linux64-shippable opt 1,654.00 -> 2,458.58
48% raptor-tp6-tumblr-firefox-cold loadtime linux64-shippable opt 1,658.08 -> 2,457.42
46% raptor-tp6-tumblr-firefox-cold loadtime linux64-shippable-qr opt 1,719.83 -> 2,504.42
28% raptor-tp6-apple-firefox-cold loadtime windows7-32-shippable opt 1,090.08 -> 1,393.17
26% raptor-tp6-apple-firefox-cold loadtime windows10-64-shippable-qr opt 1,080.58 -> 1,364.00
25% raptor-tp6-apple-firefox-cold loadtime windows10-64-shippable opt 1,096.42 -> 1,371.92
18% raptor-tp6-tumblr-firefox-cold loadtime macosx1014-64-shippable opt 2,409.25 -> 2,834.17
15% raptor-tp6-tumblr-firefox-cold loadtime windows7-32-shippable opt 1,622.42 -> 1,871.42
15% raptor-tp6-tumblr-firefox-cold loadtime windows10-64-shippable-qr opt 1,608.33 -> 1,853.83
12% raptor-tp6-tumblr-firefox-cold linux64-shippable opt 977.17 -> 1,095.68
11% raptor-tp6-tumblr-firefox-cold loadtime windows10-64-shippable opt 1,635.96 -> 1,821.00
9% raptor-tp6-tumblr-firefox-cold linux64-shippable-qr opt 1,016.55 -> 1,108.23
6% raptor-tp6-apple-firefox-cold macosx1014-64-shippable opt 1,073.12 -> 1,140.27
6% raptor-tp6-apple-firefox-cold windows10-64-shippable-qr opt 579.64 -> 612.31
2% raptor-tp6-wikipedia-firefox-cold linux64-shippable opt 1,471.71 -> 1,505.88
2% raptor-tp6-wikipedia-firefox-cold loadtime linux64-shippable opt 1,537.88 -> 1,573.58
2% raptor-tp6-wikipedia-firefox-cold linux64-shippable-qr opt 1,575.57 -> 1,610.00
2% raptor-tp6-wikipedia-firefox-cold fcp linux64-shippable opt 1,454.75 -> 1,484.08

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=24975

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/TestEngineering/Performance/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

Comment 1

[Jonathan Kew [:jfkthame]]

That's unfortunate... I wouldn't have guessed it would have such an impact.

I suspect the issue is that when we need font metrics (e.g. as part of computing normal line-height, or for CSS units 'ch' or 'ex') we now (correctly, as required by the spec) check that the font supports the <space> character rather than just using the first font we find in the font-family list.

Unfortunately, this means that we have to load the font's character map, whereas previously we may not have needed it. And that can be expensive.

I have an idea that I think might help here, I'll push a try run and see what happens...

Comment 2

[Jonathan Kew [:jfkthame]]

Let's see how this goes... https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1d0fd43353d5f6707c2e9358f73867a69a58e7c

Comment 3

[Jonathan Kew [:jfkthame]]

(In reply to Jonathan Kew (:jfkthame) from comment #2)

Let's see how this goes... https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1d0fd43353d5f6707c2e9358f73867a69a58e7c

That seems to show several good talos wins, but also some significant losses. It's unclear to me at the moment whether it's actually useful, or how reliable those (seemingly-contradictory) results are.

It looks like the real issue here isn't related to loading character maps, it's more fundamental than that: the new (corrected) behavior results in loading font resources that previously weren't loaded. For example, looking into both the Apple and Tumblr testcases, they have elements using icon fonts that don't support a <space> character. This means that when we're computing line heights, we (now) need to go to the next font resource in the list because a font without <space> is not eligible to be the "first available font" in CSS terms when computing font-based dimensions.

I'm looking into whether we can avoid additional font loads in common cases, but it's possible that being fully spec-compliant here is inherently expensive, and we may want to push back against the spec's definition of "first available font", given that it can lead to downloading multiple font resources just to determine normal line-height for an empty element.

Comment 4

[Jonathan Kew [:jfkthame]]

For example, the CSS on tumblr includes this gem (excerpted and reformatted for clarity):

    .split .chrome.options::after,
    .option.selected::before {
        font-family: 'tumblr-icons', 'Blank';
    }
    @font-face {
        font-family: 'Blank';
        src: url("data:font/opentype;base64,T1RUTwAKAIAAA ....")
    }
    @font-face {
        font-family: "tumblr-icons";
        src: url("/fonts/tumblr-icons/tumblr-icons_e24c07a0f2ea4f19997f508adc0110e4.woff2") format("woff2")
    }

where 'tumblr-icons' is a webfont resource that contains a PUA-encoded icon font, but does not support a <space> character.

Previously, we'd load the 'tumblr-icons' font and use its ascent and descent as a basis for normal line height computation.

Since bug 1611617, when we need font-based line height metrics for these elements, we now see that 'tumblr-icons' is a font that doesn't support <space> and so we fall back to the next font family, 'Blank'. This is the Adobe Blank font, provided as an inline data: URL, which means we load it synchronously.

Unfortunately, this is a font that although it contains only 2K (empty) glyphs, has a character map that maps all 1 million-plus Unicode codepoints (as overlapping ranges) to the blank glyphs, and on my Linux machine, it seems FreeType takes an entire second(!) -- while blocking reflow -- to instantiate this font. And that results in the huge cold-page-load regression seen here.

(Warm page-load is essentially unaffected because the webfont cache means that we don't have to re-instantiate the font each time we reload the page.)

Comment 5

[Jonathan Kew [:jfkthame]]

Given the size of some of the regressions here (e.g. on tumblr due to their use of Adobe Blank), I think we should back out the behavior change from bug 1611617 for the time being.

Comment 6

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1616268 - Back out changeset 780285749c80 (bug 1611617) for cold-start regressions on various sites. r=heycam

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2b147a03edff
Back out changeset 780285749c80 (bug 1611617) for cold-start regressions on various sites. r=heycam

Comment 9

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/2b147a03edff

Comment 10

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #25027 (as of Thu, 20 Feb 2020 23:16:32 GMT) ==

Improvements:

32% raptor-tp6-tumblr-firefox-cold loadtime linux64-shippable opt 2,457.58 -> 1,661.08
31% raptor-tp6-tumblr-firefox-cold loadtime linux64-shippable-qr opt 2,506.33 -> 1,718.08
25% raptor-tp6-apple-firefox-cold loadtime linux64-shippable-qr opt 1,509.17 -> 1,128.83
24% raptor-tp6-apple-firefox-cold loadtime linux64-shippable opt 1,494.67 -> 1,136.33
21% raptor-tp6-apple-firefox-cold loadtime windows10-64-shippable opt 1,389.50 -> 1,093.33
21% raptor-tp6-apple-firefox-cold loadtime macosx1014-64-shippable opt 2,326.92 -> 1,834.83
21% raptor-tp6-apple-firefox-cold loadtime windows7-32-shippable opt 1,391.42 -> 1,105.75
20% raptor-tp6-apple-firefox-cold loadtime windows10-64-shippable-qr opt 1,362.42 -> 1,088.08
15% raptor-tp6-tumblr-firefox-cold loadtime macosx1014-64-shippable opt 2,819.67 -> 2,401.67
... and 10 more.

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25027