1599632 - Firefox no longer starts on older versions of macOS - XPCOMGlueLoad error for Metal

Status: VERIFIED FIXED

(Core :: Graphics: WebGPU, defect)

Description

[:Felipe Gomes (needinfo for replies!)]

Nightly no longer opens on macOS Yosemite (10.10.5). I did a mozregression bisect and it pointed to https://hg.mozilla.org/integration/autoland/rev/4d16c3d62cfc

The error is:

XPCOMGlueLoad error for file /Applications/Firefox Nightly.app/Contents/MacOS/XUL:
dlopen(/Applications/Firefox Nightly.app/Contents/MacOS/XUL, 265): Library not loaded: /System/Library/Frameworks/Metal.framework/Versions/A/Metal
  Referenced from: /Applications/Firefox Nightly.app/Contents/MacOS/XUL
  Reason: image not found
Couldn't load XPCOM.

Comment 1

[Matthew N. [:MattN]]

[Tracking Requested - why for this release]: Nightly no longer opens on macOS Yosemite (10.10.5)

Comment 2

[Dzmitry Malyshau [:kvark]]

The issue is caused by this line: https://searchfox.org/mozilla-central/rev/cce8b90aece0f42e5025e45282de16066eeaa662/toolkit/library/moz.build#311
Do we have a baseline described/tracked somewhere, i.e. the minimum macOS version we want to be running on?

Not sure how to solve this properly yet. On Windows, we are explicitly loading D3D11.dll, D3D12.dll, and friends, and we are able to handle that gracefully. If it's possible to do on macOS as well, we'll do it.

Comment 3

[Dzmitry Malyshau [:kvark]]

Attachment: Switch Metal framework to link weakly

Comment 4

[Dzmitry Malyshau [:kvark]]

Posted a potential fix. We need somebody to grab the build from https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e2d7d1f4c8192da066a3030dd9dafe5be6ca842 and try to run it on macOS 10.10 or earlier before merging. Felipe, would you be able to test this? (thanks!)

Comment 5

[Dzmitry Malyshau [:kvark]]

To clarify, the exact artifact url is https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZnB8XVrORcK2MUr1UA-l1g/runs/0/artifacts/public/build/target.dmg

Comment 6

[Julien Cristau [:jcristau]]

(In reply to Dzmitry Malyshau [:kvark] from comment #2)

Do we have a baseline described/tracked somewhere, i.e. the minimum macOS version we want to be running on?

The documented minimum version is currently 10.9. E.g. https://www.mozilla.org/en-US/firefox/70.0.1/system-requirements/

Comment 7

[Timea Cernea [:tbabos][inactive]]

Reproduced the issue on affected latest Nightly on MacOS 10.10.5.

The artifact - provided in Comment 5 - is also crashing the Nightly Debug build. Attached screenshot and crash log as well.

Comment 8

[Timea Cernea [:tbabos][inactive]]

Attachment: CrashLog Debug build

Comment 9

[Timea Cernea [:tbabos][inactive]]

Attachment: Crash on debug build

Comment 10

[Dzmitry Malyshau [:kvark]]

Thanks Timea! Interestingly, the function that crashed doesn't seem instantly related to WebGPU/Metal: "mac_plugin_interposing_child_OnShowCursor"

Comment 11

[Dzmitry Malyshau [:kvark]]

Timea,
I got my hands on a macOS 10.10.5 machine, and I was able to reproduce the original issue on latest Nightly. However, the artifact from Comment 5 does fix it for me. On that note, I'm going to proceed with the patch. Could you double-check if you tested it correctly?

Also re-adding ni? for Felipe to test this artifact.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3238ac73b187
Switch Metal framework to link weakly r=jrmuizel

Comment 13

[:Felipe Gomes (needinfo for replies!)]

(In reply to Dzmitry Malyshau [:kvark] from comment #11)

Timea,
I got my hands on a macOS 10.10.5 machine, and I was able to reproduce the original issue on latest Nightly. However, the artifact from Comment 5 does fix it for me. On that note, I'm going to proceed with the patch. Could you double-check if you tested it correctly?

Also re-adding ni? for Felipe to test this artifact.

Yep, that one worked (it is a debug build though, so not sure if in opt that would make a difference)

Comment 14

[Markus Stange [:mstange]]

Attachment: symbolicated crash report

I resymbolicated the crash report using one of my scripts.
It shows that the debug try build was crashing on a null pointer dereference in ~ServiceWorkerShutdownBlocker() during a shutdown cycle collection. This looks completely unrelated to Metal.

Comment 15

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/3238ac73b187

Comment 16

[Timea Cernea [:tbabos][inactive]]

Hey Dzmitry,
Just updated to the latest Nightly and checked it out once more. Nightly does launch now without any issues.
There might've been some problem with the debug build on my Mac 10.10.5.
Also marking this as verified due to the above results.

Comment 17

[Firefox Bug Husbandry Bot]

Please specify a root cause for this bug. See :tmaity for more information.

Comment 18

[Dzmitry Malyshau [:kvark]]

RCA "Test Configuration not covered (OS, hardware)"