1381354 - [10.13] Crash in AppleIntelBDWGraphicsMTLDriver@0x273d1 and many other signatures, all with crash address 0x20

Status: RESOLVED WORKSFORME

(Core :: Graphics, defect, P3)

Description

[Nicholas Nethercote [inactive]]

This bug was filed from the Socorro interface and is 
report bp-5ed02629-78cc-48f2-8b7e-d0c400170715.
=============================================================

We are seeing many Mac crashes either with this signature, or with one of many unhelpful signatures that are just memory addresses, e.g. @0x7fff83e6b9f7. When you combine all the signatures it's easily the #1 Mac topcrash in Nightly.

It's happening across multiple Firefox versions, from 52 to 56. The one thing they all have in common is that the crash address is 0x20, which is why I'm grouping them all into this bug report.

They appear to go back to at least July 7, but the rate has been increasing since then.

mstange, any ideas?

Comment 1

[Marcia Knous [:marcia]]

These are all 10.13 crashes with the latest Beta 3 seed - we have done outreach to Apple regarding this and are waiting to hear back. This was discussed at the Channel meeting on Thursday.

I also had filed bug 1379750 to track this initially and it was linked to the meta bug.

Comment 3

[(not currently active) Ted Mielczarek]

FYI: my script that fetches OS X system symbols does so by downloading Apple system updates. This is why we don't have any symbols for these system libraries--these are pre-release versions. If someone at Mozilla (marcia?) has access to machines with 10.13 beta installed, I'd be happy to talk you through running similar scripts to dump symbols out of the system libraries on that machine and I could upload them to the symbol store for you:
https://github.com/luser/breakpad-scrape-system-symbols/

Comment 4

[alex_mayorga]

¡Hola Marcia!

Ended up here from bp-e18f009c-b348-487a-8cb4-3f5451170718 which is happening on Release for user asking SuMo question https://support.mozilla.org/es/questions/1168163/

Have you found any workarounds to suggest, please?

¡Gracias!
Alex

Comment 5

[Marcia Knous [:marcia]]

(In reply to alex_mayorga from comment #4)
> ¡Hola Marcia!
> 
> Ended up here from bp-e18f009c-b348-487a-8cb4-3f5451170718 which is
> happening on Release for user asking SuMo question
> https://support.mozilla.org/es/questions/1168163/
> 
> Have you found any workarounds to suggest, please?
> 
> ¡Gracias!
> Alex

Maybe Bug 1379843#c12 can be of some help.

Comment 6

[Evan Kinney]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #3)
> FYI: my script that fetches OS X system symbols does so by downloading Apple
> system updates. This is why we don't have any symbols for these system
> libraries--these are pre-release versions. If someone at Mozilla (marcia?)
> has access to machines with 10.13 beta installed, I'd be happy to talk you
> through running similar scripts to dump symbols out of the system libraries
> on that machine and I could upload them to the symbol store for you:
> https://github.com/luser/breakpad-scrape-system-symbols/

I couldn't get the upload script to work (maybe my API token doesn't have access?), but here is the resultant archive from 10.13b3 (build 17A306f):

https://www.dropbox.com/s/r3pt0ymcw9ukh8g/symbols.zip?dl=0

Comment 7

[Marcia Knous [:marcia]]

ni on Ted for Comment 6.

Comment 8

[Evan Kinney]

It doesn't seem like the symbols for the AppleGVA framework made it in there (maybe since it's in PrivateFrameworks?), and I suspect it's going to be relevant, so:

---
MODULE mac x86 908E59BBE627334397B9C7B07B39B9B10 AppleGVA
PUBLIC 16add 0 AVF_CreateMediaAcceleratorInterface
PUBLIC 44303 0 AVFQTX_CreateMediaAcceleratorInterface
PUBLIC 443eb 0 AVFQTX_GetMaxParsedStructureVersion
PUBLIC 4444b 0 AVFQTX_IsHWDRMPreferred
PUBLIC 44661 0 AVFQTX_GetFreeDRMInstanceCount
PUBLIC 5e212 0 AVFQTXENC_IsAvailable
PUBLIC 60b21 0 AVFQTXENC_CreateMediaAcceleratorInterface
PUBLIC 60bd7 0 AVFQTXENC_GetEncodeAvailability
PUBLIC 619e5 0 AVFQTXENC_IsHWScalerAvailable
PUBLIC a1af5 0 AVFJPG_CreateMediaAcceleratorInterface
PUBLIC a1b8f 0 AVFJPG_GetJPEGDecodeAvailability
PUBLIC a41a2 0 AVFCODEC_CreateMediaAcceleratorInterface
PUBLIC a434a 0 AVFPixelProcess_Create
PUBLIC a49b4 0 AVFPixelProcess_SetSourceChromaLocation
PUBLIC a4a02 0 AVFPixelProcess_SetSourceYCbCrMatrix
PUBLIC a4b4c 0 AVFPixelProcess_SetSourceParametricCurves
PUBLIC a4cae 0 AVFPixelProcess_SetScaling
PUBLIC a4e28 0 AVFPixelProcess_SetMainMatrix
PUBLIC a4f1e 0 AVFPixelProcess_SetDestParametricCurves
PUBLIC a5080 0 AVFPixelProcess_SetDestYCbCrMatrix
PUBLIC a51ca 0 AVFPixelProcess_SetDestChromaLocation
PUBLIC a521b 0 AVFPixelProcess_SetPriority
PUBLIC a53d8 0 AVFPixelProcess_PrepareToTransferImages
PUBLIC a5c3b 0 AVFPixelProcess_TransferImageSync
PUBLIC a5cb9 0 AVFPixelProcess_TransferImageAsync
PUBLIC a5d1d 0 AVFPixelProcess_WaitForAsyncTransfers
---

Comment 9

[(not currently active) Ted Mielczarek]

(In reply to Evan Kinney from comment #6)
> I couldn't get the upload script to work (maybe my API token doesn't have
> access?), but here is the resultant archive from 10.13b3 (build 17A306f):
> 
> https://www.dropbox.com/s/r3pt0ymcw9ukh8g/symbols.zip?dl=0

Thanks! Yes, you need an admin to grant symbol upload permission to your account in order to do that. I'm uploading them right now, anyone with the right permissions should be able to reprocess some of these crashes.


(In reply to Evan Kinney from comment #8)
> It doesn't seem like the symbols for the AppleGVA framework made it in there
> (maybe since it's in PrivateFrameworks?), and I suspect it's going to be
> relevant, so:

Oops, I didn't get this one. My gathersymbols script just tries to scrape the right places, but maybe it could be improved.

Comment 10

[(not currently active) Ted Mielczarek]

I reprocessed two of the crashes linked here:
https://crash-stats.mozilla.com/report/index/5ed02629-78cc-48f2-8b7e-d0c400170715
https://crash-stats.mozilla.com/report/index/e18f009c-b348-487a-8cb4-3f5451170718

The top frame (in the graphics driver) is still missing symbols, but the rest of the stacks have symbols now.

Comment 11

[emkinney]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> Oops, I didn't get this one. My gathersymbols script just tries to scrape
> the right places, but maybe it could be improved.

Ah, yeah, now I see; I didn't actually look at the script before I ran it, haha.

I added /System/Library/{PrivateFrameworks,Extensions} and re-ran. Even more symbols, and these got the AppleIntel*Graphics* libraries as well (they live in Extensions) so that should take care of everything else if you'd like to upload them to the symbol store:

https://www.dropbox.com/s/lfm2m4zgnbspvvv/even-moar-symbols-17A306f.zip?dl=0

Comment 12

[Marcia Knous [:marcia]]

(In reply to emkinney from comment #11)
> (In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> > Oops, I didn't get this one. My gathersymbols script just tries to scrape
> > the right places, but maybe it could be improved.
> 
> Ah, yeah, now I see; I didn't actually look at the script before I ran it,
> haha.
> 
> I added /System/Library/{PrivateFrameworks,Extensions} and re-ran. Even more
> symbols, and these got the AppleIntel*Graphics* libraries as well (they live
> in Extensions) so that should take care of everything else if you'd like to
> upload them to the symbol store:
> 
> https://www.dropbox.com/s/lfm2m4zgnbspvvv/even-moar-symbols-17A306f.zip?dl=0

Thanks, I think these will be useful since some of these issues seem graphics-related.

Comment 13

[emkinney]

Interestingly enough, this apparently isn't sufficient to map symbols in my own crashes of this nature:

https://crash-stats.mozilla.com/report/index/dba6f177-8614-402c-999a-1c7540170721

Any idea why?

Also, just simply bringing up amazon.com's homepage is enough to reliably trigger this crash within a few seconds if anyone's looking to reproduce.

Comment 14

[Nicholas Nethercote [inactive]]

[Tracking Requested - why for this release]:

Bad Mac crash.

Comment 15

[Marcia Knous [:marcia]]

We can track this for 56, but these crashes are affecting all versions of Firefox running the latest 10.13 beta seed.

Next steps to move this bug forward is to get the rest of the symbols uploaded in Comment 11, and then try to see if we can ascertain the different sets of crashes that are happening in this seed. The most concerning crashes are ones that cause a kernel panic.

Besides amazon, there are other sites that are causing an immediate tab crash when loading, including zillow.com.

Comment 16

[Marcia Knous [:marcia]]

(In reply to Marcia Knous [:marcia - use ni] from comment #15)
> We can track this for 56, but these crashes are affecting all versions of
> Firefox running the latest 10.13 beta seed.
> 
> Next steps to move this bug forward is to get the rest of the symbols
> uploaded in Comment 11, and then try to see if we can ascertain the
> different sets of crashes that are happening in this seed. The most
> concerning crashes are ones that cause a kernel panic.
> 
> Besides amazon, there are other sites that are causing an immediate tab
> crash when loading, including zillow.com.

Now that the new seed is out and the crashes and kernel panics seem to be gone, I don't think we have to track this for 56 or any other branch. I tested various sites where I was having kernel panics or crashing instantly and can no longer reproduce those crashes.

Comment 17

[(not currently active) Ted Mielczarek]

(In reply to emkinney from comment #11)
> (In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> > Oops, I didn't get this one. My gathersymbols script just tries to scrape
> > the right places, but maybe it could be improved.
> 
> Ah, yeah, now I see; I didn't actually look at the script before I ran it,
> haha.
> 
> I added /System/Library/{PrivateFrameworks,Extensions} and re-ran. Even more
> symbols, and these got the AppleIntel*Graphics* libraries as well (they live
> in Extensions) so that should take care of everything else if you'd like to
> upload them to the symbol store:
> 
> https://www.dropbox.com/s/lfm2m4zgnbspvvv/even-moar-symbols-17A306f.zip?dl=0

Thanks, I uploaded these symbols as well.

Comment 18

[(not currently active) Ted Mielczarek]

(In reply to emkinney from comment #13)
> Interestingly enough, this apparently isn't sufficient to map symbols in my
> own crashes of this nature:
> 
> https://crash-stats.mozilla.com/report/index/dba6f177-8614-402c-999a-
> 1c7540170721
> 
> Any idea why?

The frames in that crashing stack don't have a module listed, meaning that we don't even know what binary that address maps to. It's possible that your crash involves a shared library being unloaded while still in use.

Also, I reprocessed the two crashes in comment 10 after uploading that second set of symbols, and they now have symbols for the top frame and show up as [@ IGAccelComputeCommandEncoder::setKernel ].

Comment 19

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #18)
> (In reply to emkinney from comment #13)
> > Interestingly enough, this apparently isn't sufficient to map symbols in my
> > own crashes of this nature:
> > 
> > https://crash-stats.mozilla.com/report/index/dba6f177-8614-402c-999a-
> > 1c7540170721
> > 
> > Any idea why?

For the record: I downloaded this minidump and ran it through my dump-lookup tool, which looks at everything on the stack to see if there's anything that looks like a function. It didn't find anything, which means that either the stack pointer is pointing to a totally incorrect memory region, or there was a shared library loaded in that address range that got unloaded. The latter seems more likely, since if the stack pointer was incorrect I would not expect the stack addresses to look so sensible. Breakpad is able to unwind with frame pointers down a full 9 frames, which seems unlikely to happen by accident.

Comment 20

[Milan Sreckovic [:milan] (needinfo for best results)]

Can anybody else see any of these crashes anymore?  I can't, but my crash-stats skills are probably lacking.

Comment 21

[u279076]

I don't see these showing up beyond July 29th.