1938328 - Talos reports 50% slower startup of Firefox on MacOS 14.70 compared to MacOS 10.15

Status: NEW

(Core :: Performance Engineering, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

While running my investigation for slow Firefox startup times on MacOS workers in our CI especially on the 10.15 workers (which was caused by the Metal Shading Language compiler cache I as well detected a difference with the new MacOS 14.70 workers. Talos jobs running now on the upgraded workers report a startup time of Firefox that is 50% higher (~1000ms vs ~1500ms) on Perfherder. Note that there is one single entry of a MacOS 10.15 job in that list (dark blue) which shows the original startup time of Firefox without the Metal cache influence.

Not sure if that is a change that is expected by the upgrade to MacOS 14.70 on the workers, but that's a huge change that should be investigated. Note that current Firefox builds on cleaned-up 10.15 workers still have the ~1000ms startup time!

Comment 3

[Bas Schouten (:bas.schouten)]

ni?mstange for OS X expertise, I have no idea if this is expected in some way.

Comment 4

[Markus Stange [:mstange]]

It's not expected. Is this on the same hardware? Do you happen to have Instruments profiles of this?

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

(In reply to Markus Stange [:mstange] from comment #4)

It's not expected. Is this on the same hardware? Do you happen to have Instruments profiles of this?

Yes it's the same hardware in CI. See bug 1922355 which is the meta bug for upgrading CI jobs from 10.15 to 14.70.

I assume that we would need Instruments profiles for both the 10.15 and 14.70 versions? We still have some Mac Mini with 10.15 left in CI so it probably should be possible to create.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

For now I've triggered two profiler runs for Talos other:

• Mac OSX 10.15 (profile): 1218ms for ts_paint opt e10s fission gecko-profile stylo webrender
• MacOS 14.70 (profile): 1645ms for ts_paint opt e10s fission gecko-profile stylo webrender

Maybe those include the information that you need? If not there a way to trigger a job with configured parameters?

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

A quick glance shows that with the final-ui observer notification we already have a difference of 200ms which then doubles until startup is done.

Checking the flame graph of what happens before shows that nsAppStartup::CreateHiddenWindow takes around twice that time on 14.70. Not all the frames in the 10.15 profile are symbolicated but it looks like that something under nsNativeThemeCocoa::nsNativeThemeCocoa takes way longer. Focusing on this sub tree I see that on 10.15 XTCopyFontsWithProperties is used which takes only 40ms while on 14.70 XTCopyFontWithName is used which takes 218ms.

For the delay after final-ui it's harder to see differences. So maybe Markus can take a look.

Comment 8

[Markus Stange [:mstange]]

Thanks for these profiles. I've filed bug 1946247 and bug 1946248 for the two things that stand out in the macOS 14.7 profiles.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

Please note that since I filed this bug we have again a lot of noise in those measurements. While the perf test reported quite constant numbers when the underlying cache issue got fixed on Mac, the values started to spread again significantly as this graph from the last 90 days is showing.

Something else - probably beside the above mentioned bugs - is still causing an extra delay during the startup of Firefox. I saw a lot of Wdspec tests this morning (as reported by OF) that were timing out due to long startup times of about 4-6s for opt builds, which usually should just be 1.5s.