Closed Bug 1266220 Opened 4 years ago Closed 3 years ago

crash in @0x0 | igdumdim32.dll@0x227d8c

Categories

(Core :: Audio/Video: Playback, defect, P1, critical)

45 Branch
x86
Windows
defect

Tracking

()

RESOLVED DUPLICATE of bug 1277626
Tracking Status
firefox45 --- wontfix
firefox46 + wontfix
firefox47 + wontfix
firefox48 --- affected

People

(Reporter: ashughes, Unassigned)

References

(Depends on 1 open bug)

Details

(Keywords: crash, Whiteboard: [gfx-noted])

Crash Data

This bug was filed from the Socorro interface and is 
report bp-af696fd0-d495-45dd-b4a6-97d902160420.
=============================================================
0 		@0x0 	
Ø 1 	igdumdim32.dll 	igdumdim32.dll@0x227d8c 	
Ø 2 	igdumdim32.dll 	igdumdim32.dll@0x1a4ead 	
Ø 3 	igdumdim32.dll 	igdumdim32.dll@0x17ecd8 	
Ø 4 	igdumdim32.dll 	igdumdim32.dll@0x168bb3 	
5 	d3d9.dll 	CreateDecodeDeviceLH 	
6 	d3d9.dll 	CDxva2Container::CreateDecodeDevice(_GUID const*, _DXVA2_VideoDesc const*, _DXVA2_ConfigPictureDecode const*, IDirect3DSurface9**, unsigned int, IDirect3DDecodeDevice9**) 	
7 	dxva2.dll 	CVideoDecoderDevice::CVideoDecoderDevice(CVideoAccelerationService*, _GUID const&, _DXVA2_VideoDesc const*, _DXVA2_ConfigPictureDecode const*, IDirect3DSurface9**, unsigned int, long*) 	
8 	dxva2.dll 	CVideoAccelerationService::CreateVideoDecoder(_GUID const&, _DXVA2_VideoDesc const*, _DXVA2_ConfigPictureDecode const*, IDirect3DSurface9**, unsigned int, IDirectXVideoDecoder**) 	
9 	xul.dll 	mozilla::D3D9DXVA2Manager::SupportsConfig(IMFMediaType*, float) 	dom/media/platforms/wmf/DXVA2Manager.cpp
10 	xul.dll 	mozilla::WMFVideoMFTManager::CanUseDXVA(IMFMediaType*) 	dom/media/platforms/wmf/WMFVideoMFTManager.cpp
11 	xul.dll 	mozilla::WMFVideoMFTManager::ConfigureVideoFrameGeometry() 	dom/media/platforms/wmf/WMFVideoMFTManager.cpp
12 	xul.dll 	mozilla::WMFMediaDataDecoder::ProcessOutput() 	dom/media/platforms/wmf/WMFMediaDataDecoder.cpp
13 	xul.dll 	mozilla::WMFMediaDataDecoder::ProcessDecode(mozilla::MediaRawData*) 	dom/media/platforms/wmf/WMFMediaDataDecoder.cpp
14 	xul.dll 	nsRunnableMethodImpl<void ( mozilla::WMFMediaDataDecoder::*)(mozilla::MediaRawData*), 1, RefPtr<mozilla::MediaRawData> >::Run() 	xpcom/glue/nsThreadUtils.h
15 	xul.dll 	mozilla::TaskQueue::Runner::Run() 	xpcom/threads/TaskQueue.cpp
16 	xul.dll 	nsThreadPool::Run() 	xpcom/threads/nsThreadPool.cpp
17 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp
18 	xul.dll 	NS_ProcessNextEvent(nsIThread*, bool) 	xpcom/glue/nsThreadUtils.cpp
19 	xul.dll 	mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) 	ipc/glue/MessagePump.cpp
20 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc
21 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc
22 	xul.dll 	nsThread::ThreadFunc(void*) 	xpcom/threads/nsThread.cpp
23 	nss3.dll 	_PR_NativeRunThread 	nsprpub/pr/src/threads/combined/pruthr.c
24 	nss3.dll 	pr_root 	nsprpub/pr/src/md/windows/w95thred.c
25 	msvcr120.dll 	_callthreadstartex 	f:\dd\vctools\crt\crtw32\startup\threadex.c:376
26 	msvcr120.dll 	msvcr120.dll@0x2c000 	
27 	kernel32.dll 	BaseThreadInitThunk 	
28 	ntdll.dll 	__RtlUserThreadStart 	
29 	ntdll.dll 	_RtlUserThreadStart 	
=============================================================
More reports: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=28&signature=%400x0+%7C+igdumdim32.dll%400x227d8c#tab-reports
[Tracking Requested - why for this release]:
This is #59 with 1167 crashes in Firefox 45.0.2 and #98 with 184 crashes in Firefox 46.0b*, but strangely does not show up in 47.0a2 or 48.0a1. There are crashes going back to earlier than 45 but there seems to be an escalation recently.

This is strongly correlated to Broadwell integrated chipsets (91.16%) with the 10.18.10.3947 driver (99.67%). For 30% of users this manifests as a startup crash.

Milan, can this be helped with a block given the strong device/driver correlation?
Flags: needinfo?(milan)
Whiteboard: [gfx-noted]
Note, 88% of the crashes happen on Windows 7 with the Intel HD 5500 using the 10.18.10.3947 driver. Apparently there's an  HP Pavilion 14T and Lenovo X1 Carbon in Toronto that has this chipset. Might be worth giving that a test.
[Tracking Requested - why for this release]:
45 is EOL in a few days, tracking request moved to 46 & 47
We are about to ship 46 -- I'll track this in case we want to blocklist this particular configuration.
Matt, thoughts about blocking accelerated video on this config?
Component: Graphics → Audio/Video
Flags: needinfo?(milan) → needinfo?(matt.woodrow)
Yeah, a blacklist entry sounds like the right thing to do for this.

We also don't have a crash guard for this call site, might be nice to have.
Flags: needinfo?(matt.woodrow)
Anthony, does comment 1 still hold?  Are we still only seeing this in 45 & 46?  In that case we'd want a downloadable block in place, just for those versions.
Flags: needinfo?(anthony.s.hughes)
Actually, I see a few in 47 (e.g., https://crash-stats.mozilla.com/report/index/1ef96de5-d59e-49fb-b5bd-134f32160423), so no need to add the version based block.  If we fix this, we can add that part.

Anthony, you can check if any other driver version shows up with a high percentage, and if it perhaps happens to be the latest version available?

This would be the downloadable blocklist entry for the single device, single driver version.

<gfxBlacklistEntry>
  <os>WINNT 6.1</os>
  <vendor>0x8086</vendor>
  <devices>
    <device>0x1616</device>
  </devices>
  <feature>FEATURE_HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

We could also add devices 0x0a16 and 0x1606.  I'm not sure if the driver above is a bad one, or if it just happens to be the latest one and if we should block beyond this.
This is still mainly an issue on Release and in Beta. I suspect numbers for 47 to climb once we release our first Beta and a significant percentage of users install the update. At this point there are only 3 crashes with Firefox 47.0a2 but all of those are with Intel driver 10.18.10.3947, this compares to 99.65% in 45.0.2.
Flags: needinfo?(anthony.s.hughes)
Furthermore, the top device (~88%) is the Intel 0x1616 (HD 5500) has had several driver updates since version 10.18.10.3947 was released last June. The most recent driver version available was released less than two months ago.
Component: Audio/Video → Audio/Video: Playback
Jeff, can you review this downloadable blocklist entry?

<gfxBlacklistEntry>
  <os>WINNT 6.1</os>
  <vendor>0x8086</vendor>
  <devices>
    <device>0x1616</device>
    <device>0x0a16</device>
    <device>0x1606</device>
  </devices>
  <feature>FEATURE_HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>
Flags: needinfo?(jmuizelaar)
(In reply to Milan Sreckovic [:milan] from comment #11)
> Jeff, can you review this downloadable blocklist entry?
> 
> <gfxBlacklistEntry>
>   <os>WINNT 6.1</os>
>   <vendor>0x8086</vendor>
>   <devices>
>     <device>0x1616</device>
>     <device>0x0a16</device>
>     <device>0x1606</device>
>   </devices>
>   <feature>FEATURE_HARDWARE_VIDEO_DECODING</feature>
>   <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
>   <driverVersion>10.18.10.3947</driverVersion>   
>   <driverVersionComparator>EQUAL</driverVersionComparator>
> </gfxBlacklistEntry>

Is there any reason to believe that we should just block this driver on all haswell+?
Flags: needinfo?(jmuizelaar) → needinfo?(milan)
0x0a16 is a haswell, the other two are broadwells, right?  We could block this driver on all devices?
Flags: needinfo?(milan) → needinfo?(jmuizelaar)
(In reply to Milan Sreckovic [:milan] from comment #13)
> 0x0a16 is a haswell, the other two are broadwells, right?  We could block
> this driver on all devices?

Yeah, I think that's worth considering. These are the devices that have that version from crashstats:

1 	0x1616 	1524 	89.70 %
2 	0x0a16 	112 	6.59 %
3 	0x1606 	33 	1.94 %
4 	0x0a06 	9 	0.53 %
5 	0x0416 	8 	0.47 %
6 	0x041e 	8 	0.47 %
7 	0x0412 	3 	0.18 %
8 	0x0402 	2 	0.12 %

They are all haswell or broadwell so I think we should just disable hardware video decoding on this driver completely.
Flags: needinfo?(jmuizelaar)
Jorge, would you mind adding this downloadable blocklist entry?

<gfxBlacklistEntry>
  <os>All</os>
  <vendor>0x8086</vendor>
  <feature>FEATURE_HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>
Flags: needinfo?(jorge)
Andrew is handling most blocks these days.
Flags: needinfo?(jorge) → needinfo?(awilliamson)
what would be a suitable name for this block?  "Intel 5500HD"?
(In reply to Andrew Williamson [:eviljeff] from comment #17)
> what would be a suitable name for this block?  "Intel 5500HD"?

No. This block is specific to a driver version (10.18.10.3947). It blocks a variety of hardware.
Added.

https://addons.mozilla.org/en-US/admin/models/blocklist/blocklistdetail/1208/
Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(awilliamson)
Resolution: --- → FIXED
Milan, can you verify this is right?

(In reply to Milan Sreckovic [:milan] from comment #15)
>   <feature>FEATURE_HARDWARE_VIDEO_DECODING</feature>

Shouldn't it be HARDWARE_VIDEO_DECODING instead? This came up while testing a new version of the blocklist.
Flags: needinfo?(milan)
Yes, I'm sorry, you're correct.  It should be:

<gfxBlacklistEntry>
  <os>All</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>
Flags: needinfo?(milan)
Okay, I updated the block.
I have reasons to believe that igdumdim32.dll@0x21d559 in 10.18.10.3960 is probably the same crash
(In reply to Jeff Muizelaar [:jrmuizel] from comment #23)
> I have reasons to believe that igdumdim32.dll@0x21d559 in 10.18.10.3960 is
> probably the same crash

So it appears to be crashing in the same place. However there are some differences:
1. The crash addresses are non-zero
2. The stack further up is different.
Depends on: 1270279
There is a problem with a bug in our code that's making the blocklist entry in comment 21 too aggressive.  So, the entry g1208 needs to be removed.  I will put in the replacements.

Andrew, can you remove it?
Status: RESOLVED → REOPENED
Flags: needinfo?(awilliamson)
Resolution: FIXED → ---
(In reply to Milan Sreckovic [:milan] from comment #25)
> There is a problem with a bug in our code that's making the blocklist entry
> in comment 21 too aggressive.  So, the entry g1208 needs to be removed.  I
> will put in the replacements.
> 
> Andrew, can you remove it?

Just deleted?  You don't want the 'os' changed to something other than 'all' instead?
Flags: needinfo?(awilliamson)
Hi Andrew - yes, it will have to be replaced with multiple entries, one for each version of Windows.  We're sorting out exactly which ones, so if you could delete this one, we'll put in a request for the "real" ones after we more carefully review them.  Thanks!
Flags: needinfo?(awilliamson)
okay, deleted #1208.
Flags: needinfo?(awilliamson)
Jeff, it looks like we need all of these, if we want to cover all window versions, and can't use "All" without a specific device.  Do we want that, or do we want to instead explicitly list each of the devices?

Then, there is your comment 23, with another driver version, and there is also a recent crash that shouldn't have had us in accelerated setup:

https://crash-stats.mozilla.com/report/index/1b9bad97-1607-498d-95a2-bf6252160519

Anyway, these are the 7 entries for all supported Windows versions, not device specific:

<gfxBlacklistEntry>
  <os>WINNT 5.1</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 5.2</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 6.0</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 6.1</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 6.2</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 6.3</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 10.0</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>
Flags: needinfo?(jmuizelaar)
By the way, the blocklist was effective - I could only find 20 crashes in the past week, so we do want to put something back, just making sure it's the right thing.
It's fine to add these back for now. We can try to do better in the future.
Flags: needinfo?(jmuizelaar)
(In reply to Andrew Williamson [:eviljeff] from comment #28)
> okay, deleted #1208.

Comment 29 has the new entries, but I'll repeat them here:

<gfxBlacklistEntry>
  <os>WINNT 5.1</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 5.2</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 6.0</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 6.1</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 6.2</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 6.3</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>

<gfxBlacklistEntry>
  <os>WINNT 10.0</os>
  <vendor>0x8086</vendor>
  <feature>HARDWARE_VIDEO_DECODING</feature>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>10.18.10.3947</driverVersion>   
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>
Flags: needinfo?(awilliamson)
Is D3D11 DXVA available for this driver and is it also blocked?
(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #33)
> Is D3D11 DXVA available for this driver and is it also blocked?

I don't know about availability, but igdumdim32.dll 10.18.10.3947 is currently *not* blacklisted for D3D11.
Milan - what needs to happen to close this bug?
Flags: needinfo?(milan)
Just verify the crashes have gone down due to the blocklist update.  I see https://crash-stats.mozilla.com/report/index/97c8018f-b7d1-4dd3-97aa-38ea62160531 from today, on 44.0.2, but I don't know how to tell if we're blocking or not (blocklist takes a couple of runs to get updated, so it may not kick in on the first run.)  Overall though, ~60 crashes which compares favourably to the original count of thousands.  So, we can probably close this bug.

However :)  I think this is just one example, and I'm guessing we actually have these crashes on other driver versions.  They would show up with a different offset, and thus as a different crash, but consider:
https://crash-stats.mozilla.com/report/index/71e3d46e-1b56-4dd4-9106-a9b072160525 on 9.17.10.3517 and
https://crash-stats.mozilla.com/report/index/fd28ead1-1373-4dfe-ac01-c7ef02160524 on 8.15.10.2219.

Those are still in the hundreds, and are likely the same underlying cause.  Perhaps this was just a special case of the D3D11 DXVA bug you've tracked down?  Do we have another bug to track the combined calls, even across vendors?  E.g., 

https://crash-stats.mozilla.com/search/?product=Firefox&proto_signature=~CreateDecodeDeviceLH&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature
Flags: needinfo?(milan)
Flags: needinfo?(cpearce)
Flags: needinfo?(ajones)
If the issue is present in 46 then it isn't related to D3D11 DXVA. If we want to stop the crashes then we either need to blacklist everything that crashes or try to make things crash resistant.
^ blacklist everything that crashes
(In reply to Milan Sreckovic [:milan] from comment #37)
> Just verify the crashes have gone down due to the blocklist update.  I see
> https://crash-stats.mozilla.com/report/index/97c8018f-b7d1-4dd3-97aa-
> 38ea62160531 from today, on 44.0.2, but I don't know how to tell if we're
> blocking or not (blocklist takes a couple of runs to get updated, so it may
> not kick in on the first run.)  Overall though, ~60 crashes which compares
> favourably to the original count of thousands.  So, we can probably close
> this bug.

Note: this crash report is using D3D9 DXVA; you can see d3d9.dll on the crashing thread's stack.



> Do we have another bug to track the combined calls, even across
> vendors?  E.g., 

We blacklisted a bunch of crashes in D3D11 DXVA in the Intel drivers in bug 1274132.

Note; we're using our own black list here, which is controlled by a pref. We should really be using the same blocklist that you're using for blacklisting, so that it can be pushed out out-of-band, rather than requiring a hot-fix like our blacklist does.
Flags: needinfo?(cpearce)
I'm guessing this is a special case of bug 1277626.
(In reply to Milan Sreckovic [:milan] from comment #41)
> I'm guessing this is a special case of bug 1277626.

Seems to be fixed in 49.
Status: REOPENED → RESOLVED
Closed: 4 years ago3 years ago
Flags: needinfo?(ajones)
Resolution: --- → DUPLICATE
Duplicate of bug: 1277626
You need to log in before you can comment on or make changes to this bug.