1627691 - 2.72 - 28.97% Base Content Explicit / Base Content Heap Unclassified / Base Content Resident Unique Memory / Explicit Memory / Heap Unclassified (macosx1014-64-shippable) regression on push 1a8269aa3f6e27978920d9ba7f40367548e7aee0 (Thu April 2 2020)

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, defect, P2)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

We have detected an awsy regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=bcbee71911302d581e7c274bdaf25e49fcae7863

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

29% Base Content Heap Unclassified macosx1014-64-shippable opt 9,258,244.00 -> 11,940,069.00
29% Base Content Heap Unclassified macosx1014-64-shippable opt 9,251,627.33 -> 11,899,182.00
18% Base Content Explicit macosx1014-64-shippable opt 16,158,122.67 -> 19,008,170.67
13% Base Content Resident Unique Memory macosx1014-64-shippable opt 22,251,093.33 -> 25,107,114.67
11% Heap Unclassified macosx1014-64-shippable opt 117,404,408.04 -> 130,332,315.67
10% Heap Unclassified macosx1014-64-shippable opt tp6 129,368,952.54 -> 141,896,516.86
3% Explicit Memory macosx1014-64-shippable opt tp6 525,321,811.61 -> 539,592,741.24

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=25545

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/AWSY/Tests

Comment 1

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Made a push to try with the 2 bugs in the revision and the results comparison revealed the regression culprit.

Comment 2

[Jonathan Kew [:jfkthame]]

To be precise, it's the first patch in bug 1619350 that triggers the regression, seen clearly in the graph here:
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=try,2036624,1,4&series=try,2036622,1,4&series=try,2036628,1,4&timerange=86400&zoom=1586275520644,1586278162182,2281662.90255988,32287033.729028642
(The first data point is the "base" revision from before the bug 1619350 + bug 1619349 push, then there's a data point as each patch is added.)

Looking into this, as I wasn't expecting that patch to have such an impact. Leaving needinfo for now while investigating....

Comment 3

[Jonathan Kew [:jfkthame]]

I think this is probably because "hidden" macOS system fonts are now being included in the list of families the font-preloader processes, pulling in extra font names etc. This is causing a bunch of additional font faces to be processed at startup.

If that's the cause, we should be able to fix it by simply skipping hidden fonts when collecting the list to be preloaded. I've pushed a try job to see if that helps the AWSY figures; if it looks promising, will post a patch for review.

Comment 4

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1627691 - Don't include 'hidden' system fonts when preloading metadata to support the font-selection algorithm. r=jwatt

Comment 5

[Jonathan Kew [:jfkthame]]

Comment on attachment 9139213 [details]
Bug 1627691 - Don't include 'hidden' system fonts when preloading metadata to support the font-selection algorithm. r=jwatt

Beta/Release Uplift Approval Request

• User impact if declined: Memory-usage regression on macOS
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Patch in bug 1619350 inadvertently caused us to preload data for a bunch of rarely-needed macOS system fonts; this patch just restores the previous behavior where these fonts are not preloaded.
• String changes made/needed:

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1fb48f64d69a
Don't include 'hidden' system fonts when preloading metadata to support the font-selection algorithm. r=jwatt

Comment 7

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/1fb48f64d69a

Comment 8

[Jonathan Kew [:jfkthame]]

(In reply to Jonathan Kew (:jfkthame) from comment #5)

• Has the fix been verified in Nightly?: No

This can now be considered "verified in Nightly", I think: the AWSY graphs at https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=mozilla-central,2073523,1,4&series=mozilla-central,2073525,1,4&series=mozilla-central,2073529,1,4&timerange=1209600 clearly show the regression happening on April 2, and then the fix on April 9.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Please nominate this for Beta approval when you get a chance.

Comment 10

[Jonathan Kew [:jfkthame]]

I did already! :)

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9139213 [details]
Bug 1627691 - Don't include 'hidden' system fonts when preloading metadata to support the font-selection algorithm. r=jwatt

Details, details! Approved for 76.0b4.

Comment 12

[Razvan Maries]

https://hg.mozilla.org/releases/mozilla-beta/rev/aeeb60934b1f

Comment 13

[Marian Raiciof [:marauder]]

== Change summary for alert #25615 (as of Tue, 14 Apr 2020 04:20:06 GMT) ==

Improvements:

26% Base Content Heap Unclassified macosx1014-64-shippable opt 11,704,874.00 -> 8,678,399.67
25% Base Content Heap Unclassified macosx1014-64-shippable opt 11,646,452.33 -> 8,742,178.33
16% Base Content Explicit macosx1014-64-shippable opt 18,702,165.33 -> 15,625,386.67
11% Heap Unclassified macosx1014-64-shippable opt 127,019,829.76 -> 112,722,364.35
11% Base Content Resident Unique Memory macosx1014-64-shippable opt 24,651,861.33 -> 22,053,205.33
10% Heap Unclassified macosx1014-64-shippable opt tp6 139,511,902.76 -> 125,355,245.24
4% Explicit Memory macosx1014-64-shippable opt 429,244,959.19 -> 411,588,484.59
3% Resident Memory macosx1014-64-shippable opt tp6 594,740,769.42 -> 575,620,632.71
3% Explicit Memory macosx1014-64-shippable opt tp6 537,888,952.47 -> 521,005,511.22

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25615