Closed Bug 1743409 Opened 6 months ago Closed 17 days ago

34.69 - 17.81% ts_paint / startup_about_home_paint + 18 more (OSX) regression when Firefox is built with the macOS 11 SDK

Categories

(Core :: Widget: Cocoa, defect, P2)

defect

Tracking

()

RESOLVED WONTFIX
Tracking Status
firefox-esr91 --- unaffected
firefox94 --- unaffected
firefox95 --- unaffected
firefox96 --- disabled

People

(Reporter: beatrice.acasandrei, Unassigned, NeedInfo)

References

Details

(Keywords: perf, perf-alert, Whiteboard: [mac:integration] )

Perfherder has detected a talos performance regression from push 572b175efb0960eb15cc814d9c8f8f65afecae52. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio Test Platform Options Absolute values (old vs new)
35% ts_paint macosx1015-64-shippable-qr e10s stylo webrender-sw 535.92 -> 721.83
34% sessionrestore_many_windows macosx1015-64-shippable-qr e10s stylo webrender-sw 560.96 -> 754.08
33% ts_paint_webext macosx1015-64-shippable-qr e10s stylo webrender-sw 548.33 -> 728.25
31% sessionrestore macosx1015-64-shippable-qr e10s stylo webrender-sw 575.17 -> 754.67
31% sessionrestore_many_windows macosx1015-64-shippable-qr e10s stylo webrender-sw 574.33 -> 752.00
28% sessionrestore_no_auto_restore macosx1015-64-shippable-qr e10s stylo webrender-sw 606.33 -> 774.50
24% ts_paint macosx1015-64-shippable-qr e10s fission stylo webrender 755.50 -> 934.75
23% ts_paint macosx1015-64-shippable-qr e10s stylo webrender 750.29 -> 923.67
23% startup_about_home_paint macosx1015-64-shippable-qr e10s stylo webrender-sw 760.42 -> 935.17
23% sessionrestore_many_windows macosx1015-64-shippable-qr e10s stylo webrender 784.08 -> 961.83
... ... ... ... ...
21% ts_paint_webext macosx1015-64-shippable-qr e10s fission stylo webrender 760.17 -> 917.67
21% sessionrestore macosx1015-64-shippable-qr e10s stylo webrender 789.62 -> 953.08
20% startup_about_home_paint macosx1015-64-shippable-qr e10s stylo webrender 968.58 -> 1,165.83
18% startup_about_home_paint_realworld_webextensions macosx1015-64-shippable-qr e10s fission stylo webrender 983.17 -> 1,161.83
18% startup_about_home_paint macosx1015-64-shippable-qr e10s fission stylo webrender 977.75 -> 1,151.92

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Flags: needinfo?(mh+mozilla)
Flags: needinfo?(mh+mozilla) → needinfo?(haftandilian)

(In reply to Acasandrei Beatrice from comment #0)

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

Let's take a few days to try and root cause this and potentially land a fix. If the fix is more involved, we will want to back out the fix before 96 closes.

I'm building central with each of the macOS 11 and 10.12 SDKs to do some comparisons and Markus indicated he would also take a look.

Set release status flags based on info from the regressing bug 1696504

Blocks: 1696504
Flags: needinfo?(haftandilian)
Summary: 34.69 - 17.81% ts_paint / startup_about_home_paint + 18 more (OSX) regression on Thu November 25 2021 → 34.69 - 17.81% ts_paint / startup_about_home_paint + 18 more (OSX) regression when Firefox is built with the macOS 11 SDK
Priority: -- → P2
Whiteboard: [mac:integration]

I collected some ts_paint profiles on Try, using ./mach try fuzzy --geckoProfile.
try push with 10.12 SDK (before upgrade), try push with 11 SDK (after upgrade) (ts_paint is part of "T(o)", the "other" talos tests.)
before: https://share.firefox.dev/3GWPUZS https://share.firefox.dev/3qLepDG https://share.firefox.dev/3ry0wrP https://share.firefox.dev/3rFidGa
after: https://share.firefox.dev/3rxjGOC https://share.firefox.dev/3KrwgYk https://share.firefox.dev/3KtKNCC https://share.firefox.dev/3qHrIVJ

The difference seems to be related to fonts.
With the new SDK, two things have become slower:

The build was run on macOS 10.15.7. It's unknown which SDK version exactly causes the runtime behavior change. We haven't tested any other SDKs between 10.12 and 11.0.

Jonathan, do you think there's anything we can do about this?

Flags: needinfo?(jfkthame)

My machine seems to execute these tests much faster than on try and the delta is not nearly as significant as try reports. I ran these tests with Firefox built with each SDK between 10.12 and 12.3. The performance regression appears to have started with the 10.15 SDK.

Thanks! Could you gather a profile with the 10.14 SDK and one with the 10.15 SDK, using MOZ_PROFILER_STARTUP=1? You'll have to capture the profile manually once startup is done, using the profiler panel as usual (or Ctrl+Shift+2).

Oh, I mean a profile of just launching Firefox normally with ./mach run, not a profile of ts_paint.

Please let me know if this worked:

10.14: https://share.firefox.dev/3scOjce
10.15: https://share.firefox.dev/3GpLHgd

Like I mentioned in my last comment, the performance regression that I experienced locally was significantly less noteworthy than what was reported, so I can't be 100% confident that I was able to reproduce the same regression that was reported originally. But hopefully the performance profiles can shed some light on it.

Thank you!

(In reply to Stephen A Pohl [:spohl] from comment #7)

Please let me know if this worked:

Hmm, I'm not sure. If you switch to the parent process and search for "toolbarwindow init" (like this), nothing font-related appears. Or maybe it does just run much faster on your machine, like you said. Or maybe the slowdown is more pronounced when running on macOS 10.15.

Just to make sure, could you get another startup profile of a build with the 11 SDK?

Flags: needinfo?(spohl.mozilla.bugs)

(In reply to Markus Stange [:mstange] from comment #8)

Just to make sure, could you get another startup profile of a build with the 11 SDK?

Here you go: https://share.firefox.dev/3gua5TK

Flags: needinfo?(spohl.mozilla.bugs)

Thanks, this does show some font-stuff under window initialization! https://share.firefox.dev/3srKAIh
Unfortunately, XUL symbols were missing in this profile, I'm not sure why.

But it suggests that the problem starts with the macOS 11 SDK. Or maybe we just got (un)lucky with the mutex timing.

(In reply to Markus Stange [:mstange] from comment #10)

Thanks, this does show some font-stuff under window initialization! https://share.firefox.dev/3srKAIh
Unfortunately, XUL symbols were missing in this profile, I'm not sure why.

But it suggests that the problem starts with the macOS 11 SDK. Or maybe we just got (un)lucky with the mutex timing.

Hah, you were too quick. I just edited my previous comment with a profile that seems to be properly symbolicated.

Performance profiles, this time captured on macOS 10.15:

10.14 SDK: https://share.firefox.dev/3GxYgGz
10.15 SDK: https://share.firefox.dev/3uwi4HX
11 SDK: https://share.firefox.dev/34tcNpQ

Thanks!

The 10.15 SDK profile doesn't look comparable because it's doing some SQL bookmarks maintenance during startup so I'm ignoring it.
The 10.14 SDK and 11 SDK profiles look quite comparable!
Here's the two in the profiler comparison view: https://share.firefox.dev/39U3bqK

The only difference that stands out is that the 11 SDK profile shows asynchronous CMAP loading directly after startup, whereas the 10.14 SDK profile does not. But I think that's due to the different web pages that were loaded during startup, and not because of the SDK.

Given that we cannot reproduce a performance regression locally, and given that the test machines which show the regression are running an old version of macOS, I am wontfixing this bug. The SDK switch is unavoidable. If there are startup improvements we can make, we should choose what to optimize based on profiling results from a modern versions of macOS.

Status: NEW → RESOLVED
Closed: 17 days ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.