Open Bug 1533143 Opened 7 years ago Updated 3 years ago

Crash in [@ IOAccelContextAddResource]

Categories

(Core :: Graphics: Layers, defect, P3)

Product:
Core
Component:
Graphics: Layers
Platform:
Unspecified
macOS
Type:
defect

Tracking

()

Tracking Flags:
Tracking Status
firefox67 --- affected
firefox68 --- affected
firefox69 --- affected

People

(Reporter: dholbert, Unassigned, NeedInfo)

References

Details

(Keywords: crash, Whiteboard: [gfx-noted][tbird crash])

Crash Data

This bug is for crash report bp-cddef803-9036-4f5e-a556-9b67d0190306.

Top 10 frames of crashing thread:

0 IOAccelerator IOAccelContextAddResource 
1 AppleIntelHD4000GraphicsGLDriver AppleIntelHD4000GraphicsGLDriver@0x65de 
2 AppleIntelHD4000GraphicsGLDriver AppleIntelHD4000GraphicsGLDriver@0x4b5a87 
3 AppleIntelHD4000GraphicsGLDriver AppleIntelHD4000GraphicsGLDriver@0x501c 
4 GLEngine glClear_Exec 
5 XUL mozilla::layers::CompositorOGL::BeginFrame gfx/gl/GLContext.h:933
6 XUL mozilla::layers::LayerManagerComposite::Render gfx/layers/composite/LayerManagerComposite.cpp:922
7 XUL mozilla::layers::LayerManagerComposite::UpdateAndRender gfx/layers/composite/LayerManagerComposite.cpp:538
8 XUL mozilla::layers::LayerManager::Log gfx/layers/Layers.cpp:76
9 XUL mozilla::layers::LayerManagerComposite::EndTransaction gfx/layers/composite/LayerManagerComposite.cpp:464

This was a crash report from the "Ambient 13" live display in the Mountain View office.

jgilbert, looks like this is in GLContext.h calling into GL Driver code -- do you have any thoughts about what might be going on here? (E.g. does this look like a GL driver bug, perhaps?)

Flags: needinfo?(jgilbert)

The crash came from:

Mac mini (Late 2012)
Processor: 2.3 GHz Intel Core i7
Memory: 16 GB 1600 MHz DDR3
Graphics: Intel HD Graphics 4000 1536 MB

Also, the crash report ID in comment 0 was for Firefox 63.0.1 (a bit out of date), but we have a trickle of other crash reports
with the same signature and with more recent versions. e.g. here's one in Firefox 66 beta: bp-35175b3f-2eac-41e4-a0ce-102b50190228

Its backtrace is a bit different from the backtrace in comment 0, and it seems to be the more common backtrace that we get for this [IOAccelContextAddResource] signature:

Top 10 frames of crashing thread:

0 IOAccelerator IOAccelContextAddResource 
1 AppleIntelHD5000GraphicsGLDriver AppleIntelHD5000GraphicsGLDriver@0x11310 
2 AppleIntelHD5000GraphicsGLDriver AppleIntelHD5000GraphicsGLDriver@0x10599 
3 GLEngine glDrawArrays_ACC_Exec 
4 XUL mozilla::layers::CompositorOGL::BindAndDrawQuads gfx/gl/GLContext.h:1067
5 XUL void mozilla::layers::CompositorOGL::DrawGeometry<mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> > gfx/layers/opengl/CompositorOGL.cpp:1654
6 XUL mozilla::layers::CompositorOGL::DrawQuad gfx/layers/opengl/CompositorOGL.cpp:1087
7 XUL mozilla::layers::TiledContentHost::RenderLayerBuffer gfx/layers/composite/TiledContentHost.cpp:513
8 XUL mozilla::layers::TiledContentHost::Composite gfx/layers/composite/TiledContentHost.cpp:461
9 XUL mozilla::layers::PaintedLayerComposite::RenderLayer gfx/layers/composite/PaintedLayerComposite.cpp:106

The common pattern (different from comment 0 but common among most reports for this signature that I've looked at) seems to have: CompositorOGL::BindAndDrawQuads, calling into glDrawArrays_ACC_Exec, calling into AppleIntelHD4000GraphicsGLDriver, calling into IOAccelContextAddResource

OS: Unspecified → macOS
Priority: -- → P3
Whiteboard: [gfx-noted]

Likely driver bug, low volume.

Flags: needinfo?(jgilbert)
Priority: P3 → P4
Severity: critical → minor

FWIW, the same machine (Mountain View "Ambient 13", specs in comment 1) hit this same crash again today:
bp-c96a10af-0581-4602-9d8e-91b640190307

It's got a slightly different backtrace from yesterday's crash (comment 0) -- this one matches the more common "glDrawArrays_ACC_Exec" pattern noted in comment 2.

Priority: P4 → P3

Change the status for beta to have the same as nightly and release.
For more information, please visit auto_nag documentation.

status-firefox68: --- → affected

This bug might benefit from an analysis using a HookCase hook library (https://github.com/steven-michaud/HookCase).

I've started using HookCase to look into this bug. By examining the lifecycle of the objects involved in this bug's crashes, I've discovered one way to emulate them (bp-afe1d54b-b0c8-4f77-ba8c-ab7020200430). But it's not the only way, and I'm not sure it's the "right" way. The distinguishing characteristic of this particular way to crash is that it happens as you open a new window (though not a new tab). But I don't have permission to look at the comments on this bug's crashes at https://crash-stats.mozilla.org/. Someone who does, please look through them for things like "crashes when opening a window". I'd also like to hear about any other specific and possibly useful comments.

Daniel, I'm NI-ing you because you reported this bug. Please pass my request along to someone else if you'd prefer.

Assignee: nobody → smichaud
Status: NEW → ASSIGNED
Flags: needinfo?(dholbert)

Good to hear from you, Steven! Thanks for taking a look.

I'm bouncing your request to mccr8 (Andrew) because I don't seem to have privileges to see crash-report comments, either (I used to, but perhaps they're considered extra-private like minidumps now). The crash report "comments" page for this bug's signature says "Forbidden" for me (and I am logged in with my MoCo LDAP credentials). I'm pretty sure Andrew has access, though.

Andrew, would you mind seeing if there are any trends in the comments for this bug's crash reports? (particularly if there are mentions of opening a new window, per comment 7)

Flags: needinfo?(dholbert) → needinfo?(continuation)

The vaguely useful comments I see are:
"playing an ad that kicked from youtube"
"Twice today, Mozilla has crashed all of a sudden, without warning. The only prior indication that something was not right that I am aware of is that as of yesterday, when switching between tabs, the display glitched a few times, leaving black and white sections over the webpages."
"Since the last version was installed, constant crashes are occurring."
"I was listening to Andrew Cuomo on a segment of The Daily Show & then !poof! gone. I was on FB at the time"
"I was browsing in Twitter"
"Crashed on its own."

So, nothing very useful.

No particular pattern to the URLs that I can see. Twitter, YouTube, random other pages.

Flags: needinfo?(continuation)

Thanks for the info. Looks like I'll just have to follow my nose :-)

This is probably an Apple bug. With luck I'll be able to find it, and then find a way for Firefox to work around it. With HookCase and a good disassembler (Hopper Disassembler) I can dig deep into macOS system code.

By the way, I made some progress on this in the first couple of weeks, but now I'm stuck. I've identified a change Apple could make to its (user-mode) driver code, which would almost certainly fix these crashes. But I don't know why they're happening, so I probably wouldn't be able to make the case to Apple to change its code.

For now, I'm spending a lot of time trying to reverse engineer what Apple does with the "sideband buffer" and "command buffers" (what gets "submitted" by gpusSubmitDataBuffers()) -- on both the user side and the kernel side. But the code is fiercely complex, and my progress has been slow. I've been working on a kernel extension dedicated to this task, with the same functionality as HookCase. I also recently added watchpoint support to both HookCase and this new kernel extension. But I have no idea how long it will be before my work begins to bear fruit.

Not sure whether this helps or not but I've been seeing a lot of crashes in Firefox lately (started with the release prior to 80.0.0). I'm now running 80.0.1 and continue to see these crashes. Examining my crash submissions led me to this bug but I don't know whether it's actually the same issue or not since I didn't have crashes until recently. For me the crash is always triggered when scrolling. Whatever is going on something has changed relatively recently to make this much more frequent than it ever was in the past.

(In reply to comment #12)

Please post links to a few of your crash reports.

Thanks.

Yup, they look a lot like the other crashes for this bug. So yes, you're seeing the same bug.

Tell me what you can about your Mac: The model, how much free hard disk space, how much RAM, how much VRAM, whether you have any external monitors (and how many), and anything else you can think of that might be relevant.

Also, do you have even semi-reliable steps you can take to reproduce this bug? If so, please describe them very precisely. (By semi-reliable I mean steps that "work" (cause crashes) at least 50% of the time.)

This is an older MacBook Pro (13-inch, Mid 2012) model, 16 GB RAM with Intel HD Graphics 4000 1536 MB VRAM running Mac OS High Sierra (10.13.6). There's one external 32" WQHD monitor connected. Firefox is always positioned on the external monitor. Since I've been working from home the last while the external monitor has been pretty much permanently connected so I'm not sure whether the issue manifests without the external monitor.

I don't have semi-reliable steps to reproduce at this point. The only thing I can say is I use the two-finger scrolling on the trackpad and it generally seems I've just lifted my fingers off the trackpad when the crash occurs so seemingly towards the end of the scroll as the scroll momentum is decreasing.

Thanks again.

And since you say your crash frequency has increased recently: What changes have you made recently to your setup? For example, did you just add your external monitor, or change the one you were using?

I've made no changes that I'm aware of. I had been running with this same external monitor setup for about 6 months and hadn't been seeing crashes until the last month. I checked /Library/Receipts/InstallHistory.plist and it doesn't look like I've installed any system updates over the last couple months either.

Actually I checked the next oldest crash report and it appears identical to the others:

So I guess the same issue has existed for some time but didn't occur nearly as frequently.

One more question, which I forgot to ask earlier:

Do you keep your Mac running for a long time between reboots? (Not counting shutting its cover and reopening it -- this only puts your Mac to sleep.) If so, do the crashes tend to happen more often the longer your Mac has been "up" (since the last reboot)?

Yes, reboots are infrequent. The current uptime is 33 days. I'll have to experiment with rebooting to see whether that seems to "resolve" the crashes for a period of time.

I'll have to experiment with rebooting to see whether that seems to "resolve" the crashes for a period of time.

Do try that. There's another bug (another Apple graphics driver bug) where it made a difference -- bug 1576767.

Lately I haven't had much time to spend on this bug. But even with lots of time I'd very likely still be in the situation I described in comment #11 -- I'd probably still be stuck.

If rebooting more often doesn't help, and if you're feeling adventurous, take a look at the documentation for my HookCase debugging tool. If you're willing to try it out, I'd probably be able to write a HookCase hook library that would "fix" the Apple bug I refer to in comment #11. Start with the Readme and skim through everything. Look particularly at the "Building", "Installing" and "Using" sections. I don't blame you if all this looks too scary and/or complicated. But let me know if you'd be willing to try a HookCase hook library. If it works (if it gets rid of this bug's crashes), my next step would be to try to figure out how to patch Firefox to work around Apple's bug. I'd also (of course) ask you to try test builds with these patches.

Note that, at the very least, you'd need sufficient free hard disk space to install Apple's XCode, if you don't already have it. I'd recommend XCode 9.4.1, which is the last version targeted at your version of OS X (10.13). It takes up about 10GB.

I'm still encountering this bug on macOS Big Sur (actually 11.2.3, but the version doesn't matter), running on Mac Mini 2014.
I don't think it's related to reboot frequency, as I shut down the computer once a day. I've noticed, instead, that Firefox crashes especially when I'm using Google Meet and YouTube, but it sporadically crashes also in other circumstances.
Here are some link to recent crash reports:

For what it's worth, I found a similar bug report (a crash in IOAccelContextAddResource) for Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=707445

Some useful details from that bug report:

  • It's hardware-specific (it depended on specific GPU)
  • The code that was crashing is in the MacOS Intel driver code, and it was judged to be a bug in that code.

This seems to be the case here, too -- the backtrace from comment 32 (e.g. bp-f762f6c6-8d50-4f14-b63f-244570210303) shows AppleIntelHD5000GraphicsGLDriver in the backtrace as the thing calling into the code that crashes. So this likely seems to be an apple bug (maybe not the exact same bug discussed in that Chrome report, but a similar one).

So unfortunately I suspect there's not a lot we can do here, I think, aside from potentially deny-listing the particular driver or driver-version, perhaps.

Lorenzo: one workaround you could try is disabling webrender. (Your backtrace shows that we're using WebRender -- our GPU-accelerated rendering mode -- on your machine, which is part of why we end up triggering this graphics driver code that happens to be crashy.) You can make Firefox less-aggressively use your graphics driver by visiting about:config, searching for gfx.webrender.force-disabled, and clicking the button at the right side of that row so that the value shows true.

See Also: → https://bugs.chromium.org/p/chromium/issues/detail?id=707445

(In reply to Daniel Holbert [:dholbert] from comment #24)

For what it's worth, I found a similar bug report (a crash in IOAccelContextAddResource) for Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=707445

Some useful details from that bug report:

  • It's hardware-specific (it depended on specific GPU)
  • The code that was crashing is in the MacOS Intel driver code, and it was judged to be a bug in that code.

This seems to be the case here, too -- the backtrace from comment 32 (e.g. bp-f762f6c6-8d50-4f14-b63f-244570210303) shows AppleIntelHD5000GraphicsGLDriver in the backtrace as the thing calling into the code that crashes. So this likely seems to be an apple bug (maybe not the exact same bug discussed in that Chrome report, but a similar one).

So unfortunately I suspect there's not a lot we can do here, I think, aside from potentially deny-listing the particular driver or driver-version, perhaps.

Lorenzo: one workaround you could try is disabling webrender. (Your backtrace shows that we're using WebRender -- our GPU-accelerated rendering mode -- on your machine, which is part of why we end up triggering this graphics driver code that happens to be crashy.) You can make Firefox less-aggressively use your graphics driver by visiting about:config, searching for gfx.webrender.force-disabled, and clicking the button at the right side of that row so that the value shows true.

Thanks for your prompt reply.
I've enabled this workaround and I'm gonna use the browser normally for the next days. I'll write an update here with a new crash link if it still crashes, even with WebRender disabled.

Thanks! If that helps, we may want to automatically change our rendering behavior for users with your particular GPU and/or driver. I don't know how to do that, but I know that's something that we sometimes do. :)

jnicol, do you know what we should do to get this driver-version deny-listed or worked around (or whatever's appropriate) for WebRender? (See crash reports in comment 23, which seem potentially like webrender calling into a buggy graphics driver.)

Flags: needinfo?(jnicol)

I'm going to pass the needinfo along to Jeff or Markus as they'll know better about Mac. I believe our plan was to ship webrender to 100% there, but yeah if we cannot workaround this bug the driver may need to be blocklisted.

Flags: needinfo?(mstange.moz)
Flags: needinfo?(jnicol)
Flags: needinfo?(jmuizelaar)
Whiteboard: [gfx-noted] → [gfx-noted][tbird crash]

I no longer think I'm going to be able to work around this Apple bug.

Assignee: smichaud → nobody
Status: ASSIGNED → NEW
Severity: minor → S4

I'm not sure what to do here either. A GPU process would help with stability, maybe.

Flags: needinfo?(mstange.moz)

(Dropping "on Mountain View office ambient display" since that's no longer relevant; we're just tracking the crash-signature in the wild at this point.)

Summary: Crash in [@ IOAccelContextAddResource] on Mountain View office ambient display → Crash in [@ IOAccelContextAddResource]
You need to log in before you can comment on or make changes to this bug.