1743409 - 34.69 - 17.81% ts_paint / startup_about_home_paint + 18 more (OSX) regression when Firefox is built with the macOS 11 SDK

Status: RESOLVED WONTFIX

(Core :: Widget: Cocoa, defect, P2)

Description

[Acasandrei Beatrice (needinfo me)]

Perfherder has detected a talos performance regression from push 572b175efb0960eb15cc814d9c8f8f65afecae52. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
35%	ts_paint	macosx1015-64-shippable-qr	e10s stylo webrender-sw	535.92 -> 721.83
34%	sessionrestore_many_windows	macosx1015-64-shippable-qr	e10s stylo webrender-sw	560.96 -> 754.08
33%	ts_paint_webext	macosx1015-64-shippable-qr	e10s stylo webrender-sw	548.33 -> 728.25
31%	sessionrestore	macosx1015-64-shippable-qr	e10s stylo webrender-sw	575.17 -> 754.67
31%	sessionrestore_many_windows	macosx1015-64-shippable-qr	e10s stylo webrender-sw	574.33 -> 752.00
28%	sessionrestore_no_auto_restore	macosx1015-64-shippable-qr	e10s stylo webrender-sw	606.33 -> 774.50
24%	ts_paint	macosx1015-64-shippable-qr	e10s fission stylo webrender	755.50 -> 934.75
23%	ts_paint	macosx1015-64-shippable-qr	e10s stylo webrender	750.29 -> 923.67
23%	startup_about_home_paint	macosx1015-64-shippable-qr	e10s stylo webrender-sw	760.42 -> 935.17
23%	sessionrestore_many_windows	macosx1015-64-shippable-qr	e10s stylo webrender	784.08 -> 961.83
...	...	...	...	...
21%	ts_paint_webext	macosx1015-64-shippable-qr	e10s fission stylo webrender	760.17 -> 917.67
21%	sessionrestore	macosx1015-64-shippable-qr	e10s stylo webrender	789.62 -> 953.08
20%	startup_about_home_paint	macosx1015-64-shippable-qr	e10s stylo webrender	968.58 -> 1,165.83
18%	startup_about_home_paint_realworld_webextensions	macosx1015-64-shippable-qr	e10s fission stylo webrender	983.17 -> 1,161.83
18%	startup_about_home_paint	macosx1015-64-shippable-qr	e10s fission stylo webrender	977.75 -> 1,151.92

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Haik Aftandilian [:haik]]

(In reply to Acasandrei Beatrice from comment #0)

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

Let's take a few days to try and root cause this and potentially land a fix. If the fix is more involved, we will want to back out the fix before 96 closes.

I'm building central with each of the macOS 11 and 10.12 SDKs to do some comparisons and Markus indicated he would also take a look.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1696504

Comment 3

[Markus Stange [:mstange]]

I collected some ts_paint profiles on Try, using ./mach try fuzzy --geckoProfile.
try push with 10.12 SDK (before upgrade), try push with 11 SDK (after upgrade) (ts_paint is part of "T(o)", the "other" talos tests.)
before: https://share.firefox.dev/3GWPUZS https://share.firefox.dev/3qLepDG https://share.firefox.dev/3ry0wrP https://share.firefox.dev/3rFidGa
after: https://share.firefox.dev/3rxjGOC https://share.firefox.dev/3KrwgYk https://share.firefox.dev/3KtKNCC https://share.firefox.dev/3qHrIVJ

The difference seems to be related to fonts.
With the new SDK, two things have become slower:

• NSWindow creation takes 100ms instead of 7ms, because one of the window buttons inits its text cell, which triggers some font initialization: https://share.firefox.dev/3qJ6o2n (before SDK upgrade: https://share.firefox.dev/3IkjkS5)
• gfxPlatformFontList initialization takes 60ms instead of 0ms: https://share.firefox.dev/3nGypWu (before SDK upgrade: https://share.firefox.dev/3GO3PkY)

The build was run on macOS 10.15.7. It's unknown which SDK version exactly causes the runtime behavior change. We haven't tested any other SDKs between 10.12 and 11.0.

Jonathan, do you think there's anything we can do about this?

Comment 4

[Stephen A Pohl [:spohl]]

My machine seems to execute these tests much faster than on try and the delta is not nearly as significant as try reports. I ran these tests with Firefox built with each SDK between 10.12 and 12.3. The performance regression appears to have started with the 10.15 SDK.

Comment 5

[Markus Stange [:mstange]]

Thanks! Could you gather a profile with the 10.14 SDK and one with the 10.15 SDK, using MOZ_PROFILER_STARTUP=1? You'll have to capture the profile manually once startup is done, using the profiler panel as usual (or Ctrl+Shift+2).

Comment 6

[Markus Stange [:mstange]]

Oh, I mean a profile of just launching Firefox normally with ./mach run, not a profile of ts_paint.

Comment 7

[Stephen A Pohl [:spohl]]

Please let me know if this worked:

10.14: https://share.firefox.dev/3scOjce
10.15: https://share.firefox.dev/3GpLHgd

Like I mentioned in my last comment, the performance regression that I experienced locally was significantly less noteworthy than what was reported, so I can't be 100% confident that I was able to reproduce the same regression that was reported originally. But hopefully the performance profiles can shed some light on it.

Comment 8

[Markus Stange [:mstange]]

Thank you!

(In reply to Stephen A Pohl [:spohl] from comment #7)

Please let me know if this worked:

Hmm, I'm not sure. If you switch to the parent process and search for "toolbarwindow init" (like this), nothing font-related appears. Or maybe it does just run much faster on your machine, like you said. Or maybe the slowdown is more pronounced when running on macOS 10.15.

Just to make sure, could you get another startup profile of a build with the 11 SDK?

Comment 9

[Stephen A Pohl [:spohl]]

(In reply to Markus Stange [:mstange] from comment #8)

Just to make sure, could you get another startup profile of a build with the 11 SDK?

Here you go: https://share.firefox.dev/3gua5TK

Comment 10

[Markus Stange [:mstange]]

Thanks, this does show some font-stuff under window initialization! https://share.firefox.dev/3srKAIh
Unfortunately, XUL symbols were missing in this profile, I'm not sure why.

But it suggests that the problem starts with the macOS 11 SDK. Or maybe we just got (un)lucky with the mutex timing.

Comment 11

[Stephen A Pohl [:spohl]]

(In reply to Markus Stange [:mstange] from comment #10)

Thanks, this does show some font-stuff under window initialization! https://share.firefox.dev/3srKAIh
Unfortunately, XUL symbols were missing in this profile, I'm not sure why.

But it suggests that the problem starts with the macOS 11 SDK. Or maybe we just got (un)lucky with the mutex timing.

Hah, you were too quick. I just edited my previous comment with a profile that seems to be properly symbolicated.

Comment 12

[Stephen A Pohl [:spohl]]

Performance profiles, this time captured on macOS 10.15:

10.14 SDK: https://share.firefox.dev/3GxYgGz
10.15 SDK: https://share.firefox.dev/3uwi4HX
11 SDK: https://share.firefox.dev/34tcNpQ

Comment 13

[Markus Stange [:mstange]]

Thanks!

The 10.15 SDK profile doesn't look comparable because it's doing some SQL bookmarks maintenance during startup so I'm ignoring it.
The 10.14 SDK and 11 SDK profiles look quite comparable!
Here's the two in the profiler comparison view: https://share.firefox.dev/39U3bqK

The only difference that stands out is that the 11 SDK profile shows asynchronous CMAP loading directly after startup, whereas the 10.14 SDK profile does not. But I think that's due to the different web pages that were loaded during startup, and not because of the SDK.

Comment 14

[Markus Stange [:mstange]]

Given that we cannot reproduce a performance regression locally, and given that the test machines which show the regression are running an old version of macOS, I am wontfixing this bug. The SDK switch is unavoidable. If there are startup improvements we can make, we should choose what to optimize based on profiling results from a modern versions of macOS.