1664843 - Crash in [@ abort | glrAppleSyncState] (WebGL crashes on 10.12/10.13 on Intel HD 3000, affects Google Maps)

Status: VERIFIED FIXED

(Core :: Graphics, defect)

Description

[[:philipp]]

Crash report: https://crash-stats.mozilla.org/report/index/ea8d92b8-3f3e-4e0e-8188-4a6840200914

Top 10 frames of crashing thread:

0 libsystem_kernel.dylib mach_msg_trap 
1 XUL google_breakpad::ReceivePort::WaitForMessage toolkit/crashreporter/google-breakpad/src/common/mac/MachIPC.mm:249
2 XUL google_breakpad::CrashGenerationClient::RequestDumpForException toolkit/crashreporter/breakpad-client/mac/crash_generation/crash_generation_client.cc:70
3 XUL google_breakpad::ExceptionHandler::WriteMinidumpWithException toolkit/crashreporter/breakpad-client/mac/handler/exception_handler.cc:403
4 XUL google_breakpad::ExceptionHandler::SignalHandler toolkit/crashreporter/breakpad-client/mac/handler/exception_handler.cc:650
5 libsystem_platform.dylib _sigtramp 
6  @0x7ffee7400f3f 
7  @0x7ffee7400a67 
8 libsystem_c.dylib abort 
9 AppleIntelHD3000GraphicsGLDriver glrAppleSyncState 

these crash reports are starting to show up from nightly and devedition users on osx 10.12 and 10.13 since firefox 80 - they are not visible in beta or release builds at this time.

url correlations and comments indicate that these crashes are all happening on google maps:

• bp-8dc196d4-f678-4d07-8891-2201a0200910: "Google Maps is crashing 100% on macOS High Sierra 10.13.6 on a MacBook Pro 13 Late 2011 model, with Intel HD 3000 graphics"
• bp-fac6e39c-99c7-48c8-a53e-8c2a70200805: "I went to Google Maps, hit Directions, then the tab crashed."
• bp-2edb7bc4-c872-4ccf-8f9e-87c670200831: "tab crashes almost instantly upon loading maps.google.com"

Comment 1

[Gabriele Svelto [:gsvelto]]

Interestingly it seems the first few frames - with the exception handler and crash reporter machinery - aren't stripped from the crash. They should be so I'll file a bug for that.

Comment 2

[[:philipp]]

actually this is happening on release as well with multiple different signatures and seems to be a reproducible and fairly obnoxious crash for affected user judging by the comments in reports and threads like https://support.mozilla.org/en-US/questions/1304314

Comment 3

[[:philipp]]

80.0a1 build 20200705214003 seems to have been the first nightly build affected by the problem. this would have been the patches landing the day before: https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2020-07-04&tochange=6087e976924f95018479c6f5881878c95b8bd8e2

those 4 bugs look related to macos and/or gfx in particular: https://bugzilla.mozilla.org/buglist.cgi?bug_id=1649490,1647565,1647186,1324591

Comment 4

[Timothy Nikkel (:tnikkel)]

Unless in addition to loading google maps all of these users just happened to be changing the system pref for overlay scrollbars at the same time I don't think bug 1647565 is responsible. If it's one of those 4 bugs I would guess bug 1649490.

Comment 5

[Daniel Veditz [:dveditz]]

If the Apple driver is explicitly calling Abort(), isn't this more a stability problem than a potential vulnerability?

Comment 6

[Kelsey Gilbert [:jgilbert]]

Yep!

Comment 7

[Jim Mathies [:jimm]]

Pretty bad webgl related crash in an apple drive related to google maps. Looks like it regressed in 80. Needs a priority.

Comment 8

[Gian-Carlo Pascutto [:gcp]]

Although it's triggered by WebGL use cases, the (suspected) regressor was in device enumeration code (for Telemetry, to support WebRenderer) and the crashing code isn't ours (but if it's on old macOS, we can't even ask Apple to fix).

Jeff, any idea how to approach this?

Comment 9

[Jim Mathies [:jimm]]

Likely regressed by the telemetry changes we made in bug 1649490

Comment 10

[Jim Mathies [:jimm]]

Graphics team triaged this, we felt the telemetry work wasn't related.

Comment 11

[Gian-Carlo Pascutto [:gcp]]

I really don't see anything else in this list that would cause us to suddenly hit a bug in the HD3000 driver on older macOS:
https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2020-07-04&tochange=6087e976924f95018479c6f5881878c95b8bd8e2

We have multiple bugs filed on this, let's see if any of the users can use mozregression?

Comment 12

[Steven Michaud [:smichaud] (Retired)]

(In reply to [:philipp] from comment #3)

80.0a1 build 20200705214003 seems to have been the first nightly build affected by the problem. this would have been the patches landing the day before: https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2020-07-04&tochange=6087e976924f95018479c6f5881878c95b8bd8e2

those 4 bugs look related to macos and/or gfx in particular: https://bugzilla.mozilla.org/buglist.cgi?bug_id=1649490,1647565,1647186,1324591

2020-07-05 was when Mozilla started pushing Firefox users on older versions of OS X (pre 10.12) to the ESR branch. Crashes on the release branch are throttled (at about the rate of 1 in 10), but those on ESR releases aren't. So there are many bugs that show a substantial increase in crash numbers starting on that date. (All the other ones I know about are security bugs, so I won't list them here.)

Looking at the crash stats in "Crash Data" now, I notice a large increase starting just after 2020-08-25. That, too, may be an artifact. But until we know otherwise, it's a better place to look for possible triggers for this bug. The bug itself is almost certainly an Apple bug, in one or more of their graphics drivers. But if we can find the trigger, it may be possible to work around it.

Comment 13

[Jim Mathies [:jimm]]

Update - we think we have this hardware in Toronto, we're trying to get access to it so we can do some debugging.

Comment 14

[Markus Stange [:mstange]]

I have requested the machines from the office.

Comment 15

[[:philipp]]

an affected user at https://support.mozilla.org/en-US/questions/1305543#answer-1353867 came up with this range, so it looks like it's caused by bug 1649490 after all:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=1d15ac60&tochange=3eea2c35

Comment 16

[Jeff Muizelaar [:jrmuizel]]

Attachment: Bug 1664843. Revert "Bug 1649490 - detect all Mac GPUs"

This reverts commit f559b920572b19d1f943744416c38baf82780b98
for causing crashes on Google maps on Intel HD 3000 hardware

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c14528f13392
Revert "Bug 1649490 - detect all Mac GPUs" r=miko

Comment 18

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 9178205 [details]
Bug 1664843. Revert "Bug 1649490 - detect all Mac GPUs"

Beta/Release Uplift Approval Request

• User impact if declined: Google maps crashes on MacOS for users with Intel gen6 hardware. It seems like the crash might be restricted to 10.12/10.13
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This should be relatively low risk. It reverts a change that allowed us to detect multiple GPUs on mac systems.
• String changes made/needed:

Comment 19

[Julien Cristau [:jcristau]]

Comment on attachment 9178205 [details]
Bug 1664843. Revert "Bug 1649490 - detect all Mac GPUs"

approved for 82.0b5

Comment 20

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/1e79912f4650

Comment 21

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/c14528f13392

Comment 22

[Markus Stange [:mstange]]

I can reproduce this now. I installed macOS 10.12 on a machine with an Intel HD 3000; going to Google Maps in an affected Firefox build crashes the tab.

Comment 23

[Steven Michaud [:smichaud] (Retired)]

Interesting. I don't crash on 10.12, on a mid 2015 MacBook Pro with Intel Graphics hardware (Iris Pro) that uses the AppleIntelHD5000Graphics kernel extension. I tested with FF 81 and yesterday's mozilla-central nightly (which doesn't yet have this bug's patch), using plain-vanilla settings.

Comment 24

[Steven Michaud [:smichaud] (Retired)]

Not surprising, though: On 10.12, the only hardware-specific Intel driver that contains a "glrAppleSyncState" function is AppleIntelHD3000GraphicsGLDriver (not AppleIntelHD4000GraphicsGLDriver or AppleIntelHD5000GraphicsGLDriver).

This is the first graphics driver bug I've seen that is so very specific.

Comment 25

[Markus Stange [:mstange]]

I've been comparing a bad build and a good build. It looks like the telemetry gfx environment is the exact same, and matches the one from this crash report. This machine only has a single GPU.

However, the information displayed on https://webglreport.com/ has a lot of differences! For example:

	Bad build	Good build
Max Varying Vectors	32	15
Max Combined Texture Image Units	48	16
Aliased Line Width Range	[1, 7]	[1, 1]
Advertises support for EXT_float_blend	Yes	No

I wonder if there's some code in Firefox that would be limiting those queryable numbers to something smaller, and if the different way of querying GPUs broke it.

Comment 26

[Markus Stange [:mstange]]

Maybe this is related to sandboxing. Does the GfxInfo code run in the content process? There's a list of allow-listed IOKit property names here: https://searchfox.org/mozilla-central/rev/dfd9c0f72f9765bd4a187444e0c1e19e8834a506/security/sandbox/mac/SandboxPolicyContent.h#194-209

Comment 27

[Markus Stange [:mstange]]

Notably, the allowlist does not include the property name "class-code" which bug 1649490 queries.

The sandbox seems to be it: After setting security.sandbox.content.level to 0 and restarting, webglreport in the bad build reports the same (lower) numbers as the good build, and Google Maps no longer crashes.

Comment 28

[Markus Stange [:mstange]]

So in the "bad build", the sandbox in the content process makes it so that GfxInfo computes an empty vendor ID + device ID.

jgilbert supplied the missing piece of the puzzle: We have code in gfxPlatform.cpp which specifically works around an Intel HD 3000 crash by disabling Core Profile contexts based on the vendor ID + device ID. This code is now ineffective and the crash is back.

Comment 30

[Markus Stange [:mstange]]

I've filed bug 1668145 for the second attempt.

Comment 31

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9178205 [details]
Bug 1664843. Revert "Bug 1649490 - detect all Mac GPUs"

Approved for 81.0.1.

Comment 32

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/e6d54c3f03a2

Comment 33

[Daniel Bodea [:danibodea]]

I have reproduced this issue in older versions and verified in Nightly v83.0a1 from 2020-09-29 and Beta v82.0b5 on Mac Mini (mid2011) with Mac OS 10.13.6. Waiting for Firefox Release v80.0.1 to verify there as well. Leaving NI on me.

Comment 34

[Daniel Bodea [:danibodea]]

This fix was also verified on Firefox Release v81.0.1. Thank you.

Comment 35

[piotr.skowronek]

I experience very similar behavior but on HD4000 - reported a bug here #1739870, could anyone take a look on it and suggest how to debug it further?