1690855 - Child process launch is 10x slower on Mac

Status: NEW

(Core :: IPC, defect, P3)

Description

[Neha Kochar [:neha]]

This was found during Fission Nightly experiment data review and we noticed that child_process_launch_ms is 10x slower on Mac compared to Linux. And the same is 3x slower on Windows.
~20ms on Linux
~60ms on Windows
~200ms on Mac

Fission is within the noise level of non-Fission. But these ranges on default Firefox (without Fission) should be looked into to see why Mac is so slow for child process launch time.

See https://protosaur.dev/fission-experiment-monitoring-dashboard/dashboard/dashboard.html for those graphs.

Comment 1

[Markus Stange [:mstange]]

(In reply to Neha Kochar [:neha] from comment #0)

~200ms on Mac

So these are the numbers for CHILD_PROCESS_LAUNCH_MS, and more specifically for the mean.
I checked the GLAM dashboard for this metric and I can't really find those 200ms there, but I think that's because GLAM is showing the median (and various percentiles) as opposed to the mean.

Furthermore, there are aspects of content process initialization that are not captured in the CHILD_PROCESS_LAUNCH_MS metric (see bug 1691471), so in reality the problem might be even worse than what we see in the graphs. Bug 1690877 is a recent regression that is slowing down content process initialization on some macOS versions, but it is not captured in the 200ms numbers that this bug was filed about.

Comment 2

[Markus Stange [:mstange]]

Profile: https://share.firefox.dev/2LCs5PW

I can see the following sources of macOS-specific startup delay in this profile:

• 2 seconds during gfxPlatform::Init() (bug 1690877)
• 13-17ms in StartMacSandbox
• 15-27ms in ContentChild::SetProcessName and in establishing the connection to the window server
• 15-17ms in nsAppShell::Init

Bug 1467758 will, at some point, let us avoid the connection to the window server and the native event loop, so it should hopefully eliminate the last two items on this list.

Comment 3

[Olli Pettay [:smaug][[e-mail]]]

This is tracked as fission-perf and this is anyhow more like a meta bug, so P3/S3 should be fine here.

Comment 4

[Neha Kochar [:neha]]

This was found during Fission experiment data review but I'd like to clarify that this report is not for Fission concerns. It is instead for improving the drastic difference seen in the CHILD_PROCESS_LAUNCH_MS for Firefox (e10s) on Mac vs Linux which is 10x slower, and even Windows which is 3x slower.

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

Looking at Telemetry Evolution: https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=Median&cumulative=1&end_date=2021-02-03&include_spill=1&keys=!__none__!__none__&max_channel_version=nightly%252F88&measure=CHILD_PROCESS_LAUNCH_MS&min_channel_version=nightly%252F85&os=Darwin&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2020-12-15&trim=1&use_submission_date=0
I see Mac ~5x Linux (maybe 4x), and windows is ~2x Linux, looking at medians. I note I don't see much of a drop on the Feb 22 landing of Mac non-native-theming; perhaps the impact is indirect. There is a drop from around 60-65 to 50-55ms. Windows may have dropping when NNT landed for that, but from ~24 to ~22 - hard to say given noise and a small amount of time since the landing.

It's odd that the fission experiment dashboard values for CONTENT_PROCESS_LAUNCH_MS are so different than I see in Telemetry/evolution. Corey, any thoughts on that?

Comment 6

[Corey Dow-Hygelund [:ccd]]

This is a mean versus median effect, due to skewed distributions. The Fission dashboard uses means, which is very close to the mean values that comes from TMO.

OSX: https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=Median!Mean&cumulative=1&end_date=2021-02-03&include_spill=1&keys=!__none__!__none__&max_channel_version=nightly%252F88&measure=CHILD_PROCESS_LAUNCH_MS&min_channel_version=nightly%252F85&os=Darwin&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2020-12-15&trim=1&use_submission_date=0

Linux: https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=Median!Mean&cumulative=1&end_date=2021-02-03&include_spill=1&keys=!__none__!__none__&max_channel_version=nightly%252F88&measure=CHILD_PROCESS_LAUNCH_MS&min_channel_version=nightly%252F85&os=Linux&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2020-12-15&trim=1&use_submission_date=0

Comment 7

[Chris Peterson [:cpeterson]]

While macOS, Linux, and Windows users on Nightly probably have very different hardware, the differences between the mean and median on each platform are pretty consistent in those telemetry graphs:

OS	Median	Mean
Linux	~12 ms	~17 ms
Windows	~22 ms	~60 ms
macOS	~60 ms	~160 ms

Comment 8

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Child process launch on Mac is a little different from other platforms: we block (on a background thread) waiting for the child process to exchange Mach ports with the parent before GeckoChildProcessHost considers the process launched; on other Unix platforms that just covers the fork or clone, and on Windows if I recall correctly it's CreateProcess + some sandboxing setup (which needs to happen after the process is created but before it starts running).

If you measure from the start of launch until some later point, like when the HELLO_MESSAGE_TYPE message is received, it would probably be more similar across platforms (give or take hardware differences).

This could still be causing performance impact, if there are things the parent process could be doing in parallel with the child process's early startup, but it's also possible that it doesn't matter in practice.

We might be able to make the Mach port setup async, now that attachments are handled by the IPC channel instead of at serialization time; Nika would be the person to ask about that. But, if this isn't causing real performance problems, that wouldn't be a high priority to work on.