Firefox 33.0.1+ startup crash with signature [@ @0x0 | AllocateCB(void*, _D3DDDICB_ALLOCATE*) ]

VERIFIED FIXED in Firefox 33

Status

()

defect
--
critical
VERIFIED FIXED
5 years ago
3 years ago

People

(Reporter: KenSaunders, Assigned: bjacob)

Tracking

({crash, regression})

33 Branch
mozilla36
x86
Windows 7
Points:
---
Bug Flags:
qe-verify +

Firefox Tracking Flags

(firefox33+ verified, firefox34 verified, firefox35 verified, firefox36 verified)

Details

(crash signature)

Attachments

(4 attachments, 1 obsolete attachment)

Reporter

Description

5 years ago
After updating to 33.0.1 > restart, Firefox crashes when launching.
It will load properly in Safe Mode.
I rolled back to 33.0, it works fine.

I have a lot enabled add-ons, if a list is helpful, I'll provide one.
I also use the profile manager.
"C:\Program Files\Mozilla Firefox Release\firefox.exe" -no-remote -p


https://crash-stats.mozilla.com/report/index/10196c31-68ef-4cbf-a9fa-860aa2141026
Disable the hardware acceleration in "options/advanced/general/[]Use hardware acceleration" as workaround.

Please post the graphic section from about:support because this is related to your Nvidia Graphic card driver
Severity: blocker → critical
Crash Signature: [@ @0x0 | AllocateCB(void*, _D3DDDICB_ALLOCATE*) ]
Component: General → Graphics
Flags: needinfo?(KenSaunders)
Keywords: crash, regression
Product: Firefox → Core
Version: unspecified → 33 Branch
Reporter

Comment 2

5 years ago
Posted file Graphics
Flags: needinfo?(KenSaunders)
Reporter

Comment 3

5 years ago
Hardware acceleration was/has been disabled.

This is a backup PC that I'm using for now and I'm not sure when the drivers were last updated.

Comment 4

5 years ago
The last drivers for your Nvidia card are http://www.nvidia.com/download/driverResults.aspx/57491/en-us
Summary: Firefox 33.0.1 won't start - crashing on start → Firefox 33.0.1+ startup crash with signature [@ @0x0 | AllocateCB(void*, _D3DDDICB_ALLOCATE*) ]
The most likely reason for this D3D11 crash is that in bug 1083071 we landed a piece of code to run some D3D11 commands on startup to check if D3D11 actually works before using it, and we (accidentally) did that even if D3D11 is blacklisted. So we ended up tickling D3D11 bugs on systems with old drivers where we already knew that we shouldn't. The fix for this issue landed today. So if the issue persists in builds from tomorrow on, I'd be interested to know.

The next problem is that this crash does not only occur on old drivers that we already have blacklisted. It also happens in some newer drivers, particularly on NVIDIA cards. This hints that our driver blacklist may need to be updated substantially for D3D11. In other words, just because a driver version was good enough for D3D10 and claims to also support D3D11, does not imply that it actually supports D3D11 usefully.

Some basic facts about this crash signature:

* It only started with builds from Oct 24, which means almost certainly that it really is caused by the above-mentioned bug 1083071 patch.

* All crash reports are before Layers initialization (The AppNotes don't have "Layers?" substring). This further confirms the above theory.

* It affects all channels that have this patch, from release 33.0.1 to Nightly.

* It only affects Windows 7 and Windows 7 SP1.

To dig deeper into this crash, let's look at the GPU vendor breakdown:

$ cat 20141025-pub-crashdata.csv | grep -F 'AllocateCB(void*, _D3DDDICB_ALLOCATE*)'  | sed 's/.*AdapterVendorID\:\ \(0x\)\?\([0-9a-f]\+\).*/with vendor: 0x\2/g' | sort | uniq -c | sort -rn
   1691 with vendor: 0x10de        # 10de is NVIDIA
    328 with vendor: 0x8086        # 8086 is Intel
    319 with vendor: 0x1002        # 1002 is AMD
     55 with vendor: 0x15ad        # 15ad is VMWare
      1 with vendor: 0x1ab8        # 1ab8 is Parallels

So we see that the majority of crashes with this sig have NVIDIA cards (overall, only 20% of crashes have NVIDIA cards, so this is very significant).

But at the same time, AMD and Intel each account for more than 10% of the crashes, so the crash is NOT completely NVIDIA specific.
Bas, all of the crashes for which we have useful stacks, that I have seen, all seem to crash in this CreateTexture2D call:

http://hg.mozilla.org/releases/mozilla-release/annotate/bf6c1a1aa45b/gfx/thebes/gfxWindowsPlatform.cpp#l1511

This is not only true for NVIDIA crashes, but also for crashes with other GPU vendors. For example, here is a VMWare crash also crashing right there:

https://crash-stats.mozilla.com/report/index/4f60513c-a7a8-44c2-8491-b6c392141024

Now, this CreateTexture2D call is the first nontrivial D3D11 call that we make, so it could simply be that we crash on whatever is our first D3D11 call, but just in case, could you please proofread this code and check that we're not doing something really wonky in this CreateTexture2D call?
Bas: question for you in comment 6 ^
Flags: needinfo?(bas)
Going in another direction now - let's investigate whether our NVIDIA driver blacklist needs to be updated, i.e. if some NVIDIA driver versions might be just bad for D3D11.

See attached text file.

The commands generating this file:

$ cat 20141025-pub-crashdata.csv | grep Windows.NT.6.1 | grep 'AdapterVendorID: 0x10de' | grep 'AdapterDriverVersion\:\ [0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+' |sed 's/.*AdapterDriverVersion\:\ [0-9]\+\.[0-9]\+\.\([0-9]\+\)\.\([0-9]\+\).*/\1 \2/g' | awk '{$2 = sprintf("%04d", $2); print}' | sed 's/\([0-9]\+\)\ \([0-9]\+\)/\1.\2/g' | sort -n | uniq -c > nvidia-win7-driver-versions

$ cat 20141025-pub-crashdata.csv | grep -F 'AllocateCB(void*, _D3DDDICB_ALLOCATE*)' | grep Windows.NT.6.1 | grep 'AdapterVendorID: 0x10de' | grep 'AdapterDriverVersion\:\ [0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+' | sed 's/.*AdapterDriverVersion\:\ [0-9]\+\.[0-9]\+\.\([0-9]\+\)\.\([0-9]\+\).*/\1 \2/g' | awk '{$2 = sprintf("%04d", $2); print}' | sed 's/\([0-9]\+\)\ \([0-9]\+\)/\1.\2/g' | sort -n | uniq -c > nvidia-win7-driver-versions-thiscrash

$ cat nvidia-win7-driver-versions-thiscrash | while read l; do v=`echo "$l" | sed 's/.*\ //'`; crashes=`echo "$l" | sed 's/\ .*//'`; total=`grep -F "$v" nvidia-win7-driver-versions | sed 's/[ ]*\(.*\)\ .*/\1/'`; echo "version $v : $crashes crashes with this sig / $total total = `echo "$crashes $total" | awk '{x = 100 * $1 / $2; printf "%.1f",x}'` % of crashes with this version"; done > crash-breakdown-by-driver-version

We can see from this file that at least the driver version cutoff should probably be upped from its current 11.82xx position (NVIDIA 182.xx series) to at least include the super crashy versions in the 11.85xx range (NVIDIA 185.xx series).

But then it gets ugly. We have some super recent driver versions (13.xxx, i.e. NVIDIA 3xx series) that show high crashiness with this signature... this suggests that an approach based solely on driver versions might not be appropriate. We might need to factor in the device IDs, or maybe something else.
Attachment #8511763 - Attachment mime type: application/x-fluid → text/plain

Updated

5 years ago
Status: UNCONFIRMED → NEW
Ever confirmed: true

Comment 9

5 years ago
(In reply to Ken Saunders from comment #2)
> Created attachment 8511699 [details]
> Graphics

As LOIC said, the latest drivers for 6150SE are 285.62 and this should help you. Please update your drivers and report back.

Comment 10

5 years ago
(In reply to Ken Saunders from comment #2)
> Created attachment 8511699 [details]
> Graphics

Sorry for the churm, it appears 307.83 (Feb 26th, 2013) are the last driver version supported for 6150SE.
http://www.geforce.com/drivers/results/57492

Comment 11

5 years ago
See also bug 1083071 comment #154 and following that have some insight to this crash - follow-up discussion probably makes sense to have in here and not there, though.
Benoit+Bas+Jeff are working on this.
Assignee: nobody → bjacob
This patch should make sense regardless of crash data: on feature level 9, we should not be trying use resource sharing anyway (no d2d, no fast webgl) so it is pointless testing to see if it works.

What I don't know yet (depends on further crash analysis) is how high its chances are to fix the present crash. That depends on whether the present crash is concentrated on feature level 9 devices.
Attachment #8512078 - Flags: review?(jmuizelaar)
Attachment #8512078 - Flags: review?(bas)
Updated to address both call sites.

You might ask why not just put this in DoesD3D11DeviceSupportResourceSharing. The answer is that that would make this function name a lie, and a correct function name would then be, DoesD3D11DeviceSupportResourceSharingOrFeatureLevelLessThan10 or something else. If you feel like r-'ing for this, please make a name suggestion.
Attachment #8512078 - Attachment is obsolete: true
Attachment #8512078 - Flags: review?(jmuizelaar)
Attachment #8512078 - Flags: review?(bas)
Attachment #8512084 - Flags: review?(jmuizelaar)
Attachment #8512084 - Flags: review?(bas)
Actually, the logic is so different between the two call sites, I'm not sure that trying to put that feature level test inside the function is workable at all.
Comment on attachment 8512084 [details] [diff] [review]
Only test resource sharing on d3d feature level >= 10

Review of attachment 8512084 [details] [diff] [review]:
-----------------------------------------------------------------

A grudging r+ because of the logic duplication but no good suggestion on how to improve it.
Attachment #8512084 - Flags: review?(jmuizelaar) → review+
Comment on attachment 8512084 [details] [diff] [review]
Only test resource sharing on d3d feature level >= 10

Approval Request Comment
[Feature/regressing bug #]: patch in bug 1083071 caused this
[User impact if declined]: lots of crashes
[Describe test coverage new/current, TBPL]: n/a
[Risks and why]: i think this is really low risk. Only 1 line of code, that can't really crash by itself, and the logic there is just saying "dont try to do the fancy stuff on old hardware", which should be safe.
[String/UUID change made/needed]: none
Attachment #8512084 - Flags: review?(bas)
Attachment #8512084 - Flags: approval-mozilla-release?
Attachment #8512084 - Flags: approval-mozilla-beta?
Attachment #8512084 - Flags: approval-mozilla-aurora?
Can we have a try before the uplift? thanks :)
NVIDIA device breakdown done with this command:

$ zcat 20141025-pub-crashdata.csv.gz | grep -F 'AllocateCB(void*, _D3DDDICB_ALLOCATE*)'  | grep -F 'AdapterVendorID: 0x10de' | sed 's/.*AdapterDeviceID\:\ \(0x\)\?\([0-9a-f]\+\).*/with device: 0x\2/g' | sort | uniq -c | sort -rn | tee nvidia-devices-breakdown

Result, annotated by googling device ids (most old ids found here: http://www.nvidia.ca/object/IO_32667.html)

    394 with device: 0x03d0 // GeForce 6150SE nForce 430
    351 with device: 0x03d6 // GeForce 7025 / nForce 630a
     99 with device: 0x01d3 // GeForce 7300 SE/7200 GS
     62 with device: 0x01d7 // GeForce Go 7300
     60 with device: 0x03d1 // GeForce 6100 nForce 405
     57 with device: 0x0393 // GeForce 7300 GT
     48 with device: 0x01d1 // GeForce 7300 LE
     41 with device: 0x07e1 // GeForce 7100 / nForce 630i
     40 with device: 0x0392 // GeForce 7600 GS
     39 with device: 0x01df // GeForce 7300 GS
     36 with device: 0x0391 // GeForce 7600 GT
     29 with device: 0x0531 // GeForce 7150M / nForce 630M
     23 with device: 0x053b // GeForce 7050 PV / nForce 630a
     22 with device: 0x07e5 // GeForce 7050 / nForce 620i
     22 with device: 0x07e3 // GeForce 7050 / nForce 610i
     20 with device: 0x0398 // GeForce Go 7600
     19 with device: 0x03d2 // GeForce 6100 nForce 400
     19 with device: 0x0141 // GeForce 6600
     17 with device: 0x1040 // GeForce GT520
     16 with device: 0x0427 // GeForce 8400M GS
     15 with device: 0x0866 // GeForce 9400M G
     15 with device: 0x053e // GeForce 7025 / nForce 630a
     13 with device: 0x00cd // Quadro FX 3450/4000 SDI
     12 with device: 0x042b // Quadro NVS 135M
     12 with device: 0x0390 // GeForce 7650 GS
     12 with device: 0x0295 // GeForce 7950 GT
     11 with device: 0x0649 // GeForce 9600M GT
     10 with device: 0x0428 // GeForce 8400M G
      8 with device: 0x01dd // GeForce 7500 LE
      7 with device: 0x0402 // GeForce 8600 GT
      7 with device: 0x029d // Quadro FX 3500
      7 with device: 0x01d8 // GeForce Go 7400
      7 with device: 0x0091 // GeForce 7800 GTX
      6 with device: 0x0845 // GeForce 8200M G
      6 with device: 0x06ef // GeForce G 103M
      6 with device: 0x06ec // GeForce G 105M
      6 with device: 0x0426 // GeForce 8400M GT
      5 with device: 0x0844 // GeForce 9100M G
      5 with device: 0x06e8 // GeForce 9200M GS
      5 with device: 0x0407 // GeForce 8600M GT
      5 with device: 0x009d // Quadro FX 4500
(plus a long tail)

We see that the top devices (the ones most frequently found in this crash signature) are GeForce 6 and 7 series, from 8 to 10 years ago, definitely not Direct3D 10 capable, so they will definitely be saved from crashing by the patch here.

[Incidentally, the device of the original bug reporter is also a GeForce 6 series.]

But further down the list, we have a bunch of *mobile* (i.e. laptop) GeForce 8 and 9 series. On the desktop, GeForce 8 and 9 series are Direct3D 10.0 capable. Not sure about status on laptops, and not sure why the crashes seem totally concentrated on laptop devices. It might be for whatever reason that these laptop devices or drivers don't support Direct3D 10.0 feature level as well as their desktop counterparts do.

Then, there is one very mysterious entry: the GeForce GT520 is a recent-ish (2010) device, it's the big outlier here, no idea how to interprete that. For now let's treat it as noise / accept that we can't fix all bugs at once.
(In reply to Benoit Jacob [:bjacob] from comment #20)
> NVIDIA device breakdown done with this command:
> 
> $ zcat 20141025-pub-crashdata.csv.gz | grep -F 'AllocateCB(void*,
> _D3DDDICB_ALLOCATE*)'  | grep -F 'AdapterVendorID: 0x10de' | sed
> 's/.*AdapterDeviceID\:\ \(0x\)\?\([0-9a-f]\+\).*/with device: 0x\2/g' | sort
> | uniq -c | sort -rn | tee nvidia-devices-breakdown
> 

I wonder how many of these crashes are this crash (that happens during the driver test) vs other possibly unrelated crashes in 'AllocateCB'
(In reply to Jeff Muizelaar [:jrmuizel] from comment #21)
> (In reply to Benoit Jacob [:bjacob] from comment #20)
> > NVIDIA device breakdown done with this command:
> > 
> > $ zcat 20141025-pub-crashdata.csv.gz | grep -F 'AllocateCB(void*,
> > _D3DDDICB_ALLOCATE*)'  | grep -F 'AdapterVendorID: 0x10de' | sed
> > 's/.*AdapterDeviceID\:\ \(0x\)\?\([0-9a-f]\+\).*/with device: 0x\2/g' | sort
> > | uniq -c | sort -rn | tee nvidia-devices-breakdown
> > 
> 
> I wonder how many of these crashes are this crash (that happens during the
> driver test) vs other possibly unrelated crashes in 'AllocateCB'

Either all or nearly all of these crashes are this crash. This is recognizable by not having 'Layers?' in the AppNotes, which indicates crashing before compositor initialization. There aren't a lot of places where we do d3d11 work before compositor initialization...

Also, crashes started with the builds from last friday, which had that patch.
Reporter

Comment 23

5 years ago
Sorry for being slow.
I updated the driver.

https://crash-stats.mozilla.com/report/index/b5b2149e-6fa5-45fd-98ce-858742141027
Reporter

Comment 24

5 years ago
Posted file about-graphics.txt
Reporter

Comment 25

5 years ago
Actually, it looks like the same driver version.
The update check showed no new updates.
Thanks Ken for the update. As explained in comment 22, we expect that the new patch, which avoids running this code on non-Direct3D-10-capable hardware, will fix this crash for you, because your hardware (a GeForce 6 series GPU) is definitely non-Direct3D-10-capable.
Breakdown for Intel:

$ zcat 20141025-pub-crashdata.csv.gz | grep -F 'AllocateCB(void*, _D3DDDICB_ALLOCATE*)'  | grep -F 'AdapterVendorID: 0x8086' | sed 's/.*AdapterDeviceID\:\ \(0x\)\?\([0-9a-f]\+\).*/with device: 0x\2/g' | sort | uniq -c | sort -rn | tee intel-devices-breakdown

result:

    253 with device: 0x0be1 // Intel Graphics Media Accelerator 3600 Series
     74 with device: 0x0be2 // Intel Graphics Media Accelerator 3650 Series
      1 with device: 0x0166 // 3rd Generation Intel® HD Graphics 4000

Aside from the unique crash report with hd4000, which I can't interprete, all other reports were with the Intel GMA 3600 series devices, which are PowerVR based and support only Direct3D 9. So they are definitely taken care of by the present patch.
Breakdown for AMD:

$ zcat 20141025-pub-crashdata.csv.gz | grep -F 'AllocateCB(void*, _D3DDDICB_ALLOCATE*)'  | grep -F 'AdapterVendorID: 0x1002' | sed 's/.*AdapterDeviceID\:\ \(0x\)\?\([0-9a-f]\+\).*/with device: 0x\2/g' | sort | uniq -c | sort -rn | tee amd-devices-breakdown

result:

     31 with device: 0x7146 // ATI RADEON X1300 / X1550 Series  Direct3D 9
     30 with device: 0x7196 // ATI MOBILITY /ATI RADEON X1350   Direct3D 9
     29 with device: 0x71c7 // ATI RADEON X1650 Series          Direct3D 9
     24 with device: 0x7142 // ATI RADEON X1300/X1550 Series    Direct3D 9
     19 with device: 0x71c6 // ATI RADEON X1650 Series          Direct3D 9
     18 with device: 0x71c2 // ATI RADEON X1600 Series          Direct3D 9
     14 with device: 0x7183 // ATI RADEON X1300/X1550 Series    Direct3D 9
     12 with device: 0x7145 // ATI MOBILITY /ATI RADEON X1400   Direct3D 9
     11 with device: 0x7188 // ATI MOBILITY /ATI RADEON X2300   Direct3D 9
      9 with device: 0x9591 // ATI Mobility Radeon HD 3650      Direct3D 10.1
      9 with device: 0x954f // ATI Radeon HD 4350               Direct3D 10.1
      9 with device: 0x7280 // ATI RADEON X1950 Series          Direct3D 9
      9 with device: 0x7143 // ATI RADEON X1300/ X1550 Series   Direct3D 9
      7 with device: 0x9501 // ATI Radeon HD 3870               Direct3D 10.1
      7 with device: 0x94c3 // ATI Radeon HD 2400 PRO PCI-E     Direct3D 10.0
      6 with device: 0x7240 // ATI RADEON X1950 Series          Direct3D 9
      6 with device: 0x715f // ATI RADEON X1550 64-bit          Direct3D 9
      5 with device: 0x95c4 // ATI Mobility Radeon HD 3450      Direct3D 10.1
      5 with device: 0x71c5 // ATI MOBILITY /ATI RADEON X1600   Direct3D 9
      5 with device: 0x71c1 // ATI RADEON X1650 Series          Direct3D 9
      5 with device: 0x718a // ATI MOBILITY /ATI RADEON X2300   Direct3D 9
      5 with device: 0x7147 // ATI RADEON X1550 64-bit          Direct3D 9
(plus long tail)

So again we see that the top devices for this crash signature are Direct3D 9. It's worrying though to find a substantial amount of Direct3D 10.1 devices further down the list.
So at this point, it seems that the patch here will fix the vast majority of these crashes, but not all of them. Above we have established that for each of the 3 major desktop GPU vendoes. IMHO we should take this patch in Firefox 33.0.2, and I expect based on the above analysis that it will bring crashiness levels down to acceptable levels. Then we should be in a better position to study the remaining crashes.
Comment on attachment 8512084 [details] [diff] [review]
Only test resource sharing on d3d feature level >= 10

Could you land that in the branch GECKO330_2014101104_RELBRANCH?
Thanks
Attachment #8512084 - Flags: approval-mozilla-release?
Attachment #8512084 - Flags: approval-mozilla-release+
Attachment #8512084 - Flags: approval-mozilla-beta?
Attachment #8512084 - Flags: approval-mozilla-beta+
Attachment #8512084 - Flags: approval-mozilla-aurora?
Attachment #8512084 - Flags: approval-mozilla-aurora+

Comment 31

5 years ago
(In reply to Ken Saunders from comment #25)
> Actually, it looks like the same driver version.
> The update check showed no new updates.

Not sure what you mean by update check. Did you grab them from the Geforce link I provided? You aren't trying to get updated drivers via Windows Update, are you?
https://hg.mozilla.org/mozilla-central/rev/14ec535fd1c0
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla36
Flags: qe-verify+
Duplicate of this bug: 1093116
No crashes for either signature on beta builds after 20141027152126.
Crash Signature: [@ @0x0 | AllocateCB(void*, _D3DDDICB_ALLOCATE*) ] → [@ @0x0 | AllocateCB(void*, _D3DDDICB_ALLOCATE*) ] [@ @0x0 | NDXGI::CDevice::AllocateCB(void*, _D3DDDICB_ALLOCATE*)]
There are no new crashes in Firefox 33, 34, 35 or 36 after the fix arrived based on Socorro data.

33.1
https://crash-stats.mozilla.com/query/?product=Firefox&version=Firefox%3A33.1&range_value=1&range_unit=weeks&date=11%2F11%2F2014+14%3A00%3A00&query_search=signature&query_type=contains&query=D3DDDICB_ALLOCATE&reason=&release_channels=&build_id=&process_type=any&hang_type=any

34 beta
No more crashes after beta 5
https://crash-stats.mozilla.com/report/list?signature=%400x0+%7C+AllocateCB%28void%2A%2C+_D3DDDICB_ALLOCATE%2A%29&product=Firefox&query_type=contains&range_unit=weeks&process_type=any&version=Firefox%3A34.0b&hang_type=any&date=2014-11-11+14%3A00%3A00&range_value=1#tab-reports

35.0a2 
No more crashes after the fix
https://crash-stats.mozilla.com/report/list?signature=%400x0+%7C+AllocateCB%28void%2A%2C+_D3DDDICB_ALLOCATE%2A%29&product=Firefox&query_type=contains&range_unit=weeks&process_type=any&version=Firefox%3A35.0a2&hang_type=any&date=2014-11-11+14%3A00%3A00&range_value=1#tab-reports

36.0a1
No more crashes after the fix
https://crash-stats.mozilla.com/report/list?signature=%400x0+%7C+AllocateCB%28void%2A%2C+_D3DDDICB_ALLOCATE%2A%29&product=Firefox&query_type=contains&range_unit=weeks&process_type=any&version=Firefox%3A36.0a1&hang_type=any&date=2014-11-11+14%3A00%3A00&range_value=1#tab-reports
and
https://crash-stats.mozilla.com/report/list?signature=%400x0+%7C+NDXGI%3A%3ACDevice%3A%3AAllocateCB%28void%2A%2C+_D3DDDICB_ALLOCATE%2A%29&product=Firefox&query_type=contains&range_unit=weeks&process_type=any&version=Firefox%3A36.0a1&hang_type=any&date=2014-11-11+14%3A00%3A00&range_value=1#tab-reports
Flags: needinfo?(bas)
You need to log in before you can comment on or make changes to this bug.