Closed Bug 1738289 Opened 3 years ago Closed 2 years ago

macOS 10.15.7 build 19H1519 crashes at gpusGenerateCrashLog with graphics kernel error 0xfffffffb/-5

Categories

(Core :: Graphics, defect)

Product:
Core
Component:
Graphics
Platform:
Unspecified
macOS
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr91 --- fixed
firefox93 --- wontfix
firefox94 + wontfix
firefox95 + fixed
firefox96 + fixed
firefox97 + fixed

People

(Reporter: aryx, Unassigned)

References

Details

(Keywords: crash, regression)

Crash Data

Crash rate for the signature went from 10-20/day for Firefox 93 to 300+/day. Increase started on 2021-10-25 and really kicked in on the 26th. Only reported for macOS 1015 and 11.

Crash report: https://crash-stats.mozilla.org/report/index/7a05de0f-f620-4238-923a-31e430211028

mac crash info:

{
  "num_records": 2,
  "records": [
    {
      "message": "abort() called",
      "module": "/usr/lib/system/libsystem_c.dylib"
    },
    {
      "module": "/System/Library/PrivateFrameworks/GPUSupport.framework/Versions/A/Libraries/libGPUSupportMercury.dylib",
      "signature_string": "Graphics kernel error: 0xfffffffb\n"
    }
  ]
}

Reason: EXC_SOFTWARE / SIGABRT

Top 10 frames of crashing thread:

0 libsystem_kernel.dylib __pthread_kill 
1 libsystem_pthread.dylib pthread_kill 
2 libsystem_c.dylib abort 
3 libGPUSupportMercury.dylib gpusGenerateCrashLog.cold.1 
4 libGPUSupportMercury.dylib gpusGenerateCrashLog 
5 AppleIntelHD4000GraphicsGLDriver gpusKillClientExt 
6 libGPUSupportMercury.dylib gpusSubmitDataBuffers 
7 AppleIntelHD4000GraphicsGLDriver IntelCommandBuffer::getNew 
8 AppleIntelHD4000GraphicsGLDriver intelSubmitCommands 
9 XUL mozilla::gl::GLContext::fFlush gfx/gl/GLContext.h:1137
Blocks: gfx-triage

gsvelto mentioned this aligns with updates release by Apple.

This is mostly on Ivybridge = 0x0166 and 10.15.7 19H1519. It would be interesting to to tease this out to see if it was caused by users switching to that version.

Jeff do you have someone at Apple that you could reach out to about this?

Flags: needinfo?(jgilbert)

I will reach out to them!

Flags: needinfo?(jgilbert)

This is known to Apple, and Chrome and Safari both also experience it, but "we don’t yet have a reliable repro".

It may or may not have to do with bad internal driver sync of tex data uploads, per Google investigations.

We did increase the number of glFlushes that we do in bug 1527181, but the timeline really match. The spike began on oct26, but Firefox 93 came out on 2021-10-05. I think in most cases, we end up not doing any extra flushes in bug 1527181. (no extra flushes are done for normal raf(drawing) flows) It's perhaps possible that things might improve this disabled, so we should probably try that, as a potential high-upside-low-downside.

No longer blocks: gfx-triage

macOS 10.15.7 build 19H1519 was released on 2021/10/25, just at the start of the burst:

https://en.wikipedia.org/wiki/MacOS_Catalina#Release_history

The error messages associated with the 0xfffffffb/-5 context error in the IOAcceleratorFamily2 kernel extension are "stream error found. token_id was 0x%04x token_size was 0x%04x", "unrecognized token id", and "unexpected token size". So they all have to do with "tokens", as described at bug 1713230 comment #3.

Summary: macOS 10.15 + 11 Crash in Crashes at gpusGenerateCrashLog with graphics kernel error '0xfffffffb\n' → macOS 10.15 + 11 Crash in Crashes at gpusGenerateCrashLog with graphics kernel error 0xfffffffb/-5

FYI, a few hours ago I've reported bug #1739870 (which has been found as a duplicate of this one) where you can see repro steps (simply open Google Maps and switch to Street View or just open Google Earth site). If you had a FF version with a fix I can check whether it solves the problem.

BTW, on Chrome Google Maps Street View works, but Google Earth fails 'WebGL rendering context lost') - what is nice in Chrome tho is that it at least behaves better in such a case, only one tab requires reload, whereas Firefox crashes the whole browser :/

These crashes are very hardware-specific. They happen on macOS 10.15.7 build 19H1519, and almost exclusively on Intel graphics hardware ("vendor id" 0x8086) with the "device id" 0x0166. There's no point trying to use your STR, Piotr, except on this hardware. I tried and failed on "Intel UHD Graphics 630" hardware (device id 0x3e9b) and "Intel Iris Pro" hardware (device id 0x0d26).

Apple, though, should surely be able to scare up a machine with the "right" kind of graphics hardware that runs the "right" version of macOS. They'll probably be able to use it to reproduce these crashes using Piotr's STR. Jeff, could you pass this information along to Apple?

https://crash-stats.mozilla.org/search/?signature=~gpusGenerateCrashLog&platform_version=~10.15.7%2019H1519&mac_crash_info=~Graphics%20kernel%20error%3A%200xfffffffb&platform=Mac%20OS%20X&date=%3E%3D2021-10-24T16%3A17%3A00.000Z&date=%3C2021-11-07T16%3A17%3A00.000Z&_facets=signature&_facets=version&_facets=adapter_device_id&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-adapter_device_id

Flags: needinfo?(jgilbert)
Summary: macOS 10.15 + 11 Crash in Crashes at gpusGenerateCrashLog with graphics kernel error 0xfffffffb/-5 → macOS 10.15 build 19H1519 Crashes at gpusGenerateCrashLog with graphics kernel error 0xfffffffb/-5

(In reply to Jeff Gilbert [:jgilbert] from comment #5)

We did increase the number of glFlushes that we do in bug 1527181, but the timeline really match. The spike began on oct26, but Firefox 93 came out on 2021-10-05. I think in most cases, we end up not doing any extra flushes in bug 1527181. (no extra flushes are done for normal raf(drawing) flows) It's perhaps possible that things might improve this disabled, so we should probably try that, as a potential high-upside-low-downside.

We're seeing the spike on ESR91 as well, fwiw. Is SW-WR for affected chipsets a viable short-term option worth trying?

Summary: macOS 10.15 build 19H1519 Crashes at gpusGenerateCrashLog with graphics kernel error 0xfffffffb/-5 → macOS 10.15.7 build 19H1519 Crashes at gpusGenerateCrashLog with graphics kernel error 0xfffffffb/-5

The adapter device id 0x0166 corresponds to Intel HD Graphics 4000 hardware (https://en.wikipedia.org/wiki/List_of_Intel_graphics_processing_units#Gen7).

You can find which Mac models contain this graphics hardware by searching through this list on "HD Graphics 4000".

To people seeing these crashes (on macOS 10.15.7 build 19H1519, with Intel graphics hardware):

If you have a MacBook Pro, it may contain more than one kind of graphics hardware -- builtin "HD Graphics 4000" hardware plus something from a different vendor (perhaps AMD). If so, you may be able to avoid these crashes by forcing programs you run to use the other ("discrete") kind. One way to do this is to use gfxCardStatus.

Apple AMD drivers also have lots of bugs, and they can trigger crashes with signatures that are similar (though not identical) to those of this bug. But Apple's Intel drivers on macOS 10.15.7 build 19H1519 seem to be the worst of the worst. So you'll likely be better off if you can avoid using them.

I see 95.X "won't fix" :/

The thing is that recently even Facebook started to kill FF - probably there are some ads triggering that (even when uBlock is engaged).
Any hope for putting some pressure on Apple to fix drivers? Or at least making this crash less impactful? Chrome and Opera are more stable on the same hardware - just one tab is being crashed, whereas FF is being crashed totally :/ I know that I've got probably kinda old macbook (that still does its job perfectly from performance perspective) and Apple won't be in hurry to fix drivers, but similar bugs may happen even on newest hardware and crashing whole FF is quite annoying.

FYI, I did reach out to another one of our contacts at Apple and am waiting to hear back. I know this is frustrating but right now it doesn't seem like there's much in the way of options for addressing this on our end.

status-firefox96: affected → wontfix
status-firefox97: --- → affected
tracking-firefox97: --- → +

My contact asked if there's a Feedback Assistant ticket filed for this yet. Does anybody have a ticket # for that if yes?

Received a Catalina 10.15.7 update over this past weekend -- it's crashed my Firefox only once so far

Nope, imho 10.15.7 doesn't fix the problem. I've checked with Google Maps and Street view - and yes, it doesn't crash FF anymore - I think google did some workarounds for Firefox (and they I guess they disabled 3D globe on FF?). On Chrome and Safari 3D view is available but ... still it kills the tab.

I have one page that still kills FF no problem - try to visit this page (it is real estate developer page and it has a lot of animations): https://zakatekdabie.pl/inwestycja - can you verify if your FF stands for more that a minute on this page?

(In reply to adrie from comment #20)

Received a Catalina 10.15.7 update over this past weekend -- it's crashed my Firefox only once so far

CLARIFICATION: I should have mentioned that my previous Catalina update caused my Firefox browser to crash while using Facebook exclusively and constantly. Since my current Catalina update, there has been only one Facebook crash.

Hi, I finally understand that not the only one with this problem! :(

I've a MacMini 2012 whit Intel HD Graphics 4000!

Hope it can be fixed!

A site that generates the block for me is (I point it out if it can be useful):
https://www.treedom.net/it/user/shooterhatesyou/event/breakingforest

People who see these crashes on macOS 10.15.7 builds 19H1519 or 19H1615:

Please open a bug with Apple, here: https://feedbackassistant.apple.com/

Be sure to include steps to reproduce, and the kind of Mac you're using. And reference this bug.

Please also post your feedback number here, so it can be passed back to Apple with regard to this bug (as per comment #19).

Summary: macOS 10.15.7 build 19H1519 Crashes at gpusGenerateCrashLog with graphics kernel error 0xfffffffb/-5 → macOS 10.15.7 builds 19H1519 and 19H1615 crashes at gpusGenerateCrashLog with graphics kernel error 0xfffffffb/-5

(In reply to Steven Michaud [:smichaud] (Retired) from comment #24)

People who see these crashes on macOS 10.15.7 builds 19H1519 or 19H1615:

Please open a bug with Apple, here: https://feedbackassistant.apple.com/

Be sure to include steps to reproduce, and the kind of Mac you're using. And reference this bug.

Please also post your feedback number here, so it can be passed back to Apple with regard to this bug (as per comment #19).

Done, https://feedbackassistant.apple.com/feedback/9819579 - I hope it is clear enough.

Judging by Mozilla's crash statistics for the last two weeks, Apple does seem to have fixed these crashes in macOS 10.15.7 build 19H1615 (which was released on 2021-12-13):

https://crash-stats.mozilla.org/search/?mac_crash_info=~Graphics%20kernel%20error%3A%200xfffffffb&platform_version=~10.15.7&adapter_device_id=0x0166&platform=Mac%20OS%20X&date=%3E%3D2021-12-21T17%3A06%3A00.000Z&date=%3C2022-01-04T17%3A06%3A00.000Z&_facets=signature&_facets=platform_version&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-platform_version

I'm going to resolve this bug WORKSFORME. Whatever problems people still see with macOS 10.15.7 build 19H1615 on "HD Graphics 4000" hardware are probably other issue(s), for which you'd need to open new bug(s).

Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(jgilbert)
Resolution: --- → WORKSFORME
Summary: macOS 10.15.7 builds 19H1519 and 19H1615 crashes at gpusGenerateCrashLog with graphics kernel error 0xfffffffb/-5 → macOS 10.15.7 build 19H1519 crashes at gpusGenerateCrashLog with graphics kernel error 0xfffffffb/-5

(In reply to Steven Michaud [:smichaud] (Retired) from comment #26)

Judging by Mozilla's crash statistics for the last two weeks, Apple does seem to have fixed these crashes in macOS 10.15.7 build 19H1615 (which was released on 2021-12-13):

Nope, they didn't fix it.

I'm going to resolve this bug WORKSFORME. Whatever problems people still see with macOS 10.15.7 build 19H1615 on "HD Graphics 4000" hardware are probably other issue(s), for which you'd need to open new bug(s).

With all due respect, I don't agree with this decision. The problem is still there - visit this sties and FF will crash on them:
https://www.treedom.net/it/user/shooterhatesyou/event/breakingforest, https://zakatekdabie.pl/inwestycja - I've just reproduced this on 19H1615.

The fall in stats you see because apparently Google Maps (Street view) and Facebook have introduced fixes into their websites or simply changes the code for some reason. THE PROBLEM IS STILL THERE. Kindly please re-open this one so Apple as stated above based on https://feedbackassistant.apple.com/feedback/9819579 could see that the problem is still there.

(In reply to piotr.skowronek from comment #27)

Kindly please re-open this one so Apple as stated above based on https://feedbackassistant.apple.com/feedback/9819579 could see that the problem is still there.

FYI, I did pass this link along to our contact and was told it would be passed along internally too :-)

Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---

Piotr, if you'll look at my crash statistics from comment #26 you'll see that there are no crashes over the last two weeks in build 19H1615 that match this bug's specific criteria. Apple may still have other bugs in its graphics drivers (even specific to "HD Graphics 4000" hardware). But to deal with it (or them) you'll need to open one or more new bugs.

With all due respect, I don't agree with this decision. The problem is still there - visit this sties and FF will crash on them:
https://www.treedom.net/it/user/shooterhatesyou/event/breakingforest, https://zakatekdabie.pl/inwestycja - I've just reproduced this on 19H1615.

https://crash-stats.mozilla.org/report/index/70bc81db-93aa-4396-9f84-b26030220104
https://crash-stats.mozilla.org/report/index/477e5581-4815-4673-8209-e2d830220104

For both tries.

(In reply to Steven Michaud [:smichaud] (Retired) from comment #29)

Piotr, if you'll look at my crash statistics from comment #26 you'll see that there are no crashes over the last two weeks in build 19H1615 that match this bug's specific criteria. Apple may still have other bugs in its graphics drivers (even specific to "HD Graphics 4000" hardware). But to deal with it (or them) you'll need to open one or more new bugs.

Can you take a look above in my last comment and check if the specific criteria have changed?

Sigh.

Piotr, your crash reports indicate that you're still using build 19H1519. Try upgrading to macOS 10.15.7 build 19H1615. That's the latest available build for macOS 10.15.7.

BTW, CVE-2021-30977 & CVE-2021-30981 from the 19H1615 release notes look possibly-relevant:

Description: A buffer overflow was addressed with improved bounds checking.

(In reply to Steven Michaud [:smichaud] (Retired) from comment #32)

Sigh.

Piotr, your crash reports indicate that you're still using build 19H1519. Try upgrading to macOS 10.15.7 build 19H1615. That's the latest available build for macOS 10.15.7.

Honestly, I don't know how it has happened - there was no notification on my mac about this update whatsoever. I had to 'manually' check and there it was. I've just installed 19H1615 and I can confirm that THIS PROBLEM IS FIXED!

Apologies for confusion. I will also update the feedback item: https://feedbackassistant.apple.com/feedback/9819579

(In reply to Ryan VanderMeulen [:RyanVM] from comment #33)

BTW, CVE-2021-30977 & CVE-2021-30981 from the 19H1615 release notes look possibly-relevant:

Description: A buffer overflow was addressed with improved bounds checking.

Interesting and kinda scary....

Alright, let's tentatively call this fixed again then! :-)

Status: REOPENED → RESOLVED
Closed: 2 years ago → 2 years ago
status-firefox95: wontfix → fixed
status-firefox96: wontfix → fixed
status-firefox97: affected → fixed
status-firefox-esr91: affected → fixed
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.