crash spike in igd10umd32.dll@0x280a1 with 39.0 beta (Specific to Intel Driver 8.15.10.2086)

RESOLVED FIXED

Status

()

Core
Graphics
--
critical
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: Robert Kaiser, Assigned: milan)

Tracking

({crash})

Trunk
x86
Windows NT
crash
Points:
---
Bug Flags:
qe-verify +

Firefox Tracking Flags

(firefox39+ fixed, firefox40+ fixed, firefox41+ fixed, firefox42- wontfix, firefox43+ wontfix, firefox44+ wontfix, firefox45- affected)

Details

(crash signature)

Attachments

(3 attachments, 1 obsolete attachment)

(Reporter)

Description

2 years ago
[Tracking Requested - why for this release]:

This bug was filed from the Socorro interface and is 
report bp-e6cd12c2-6ead-4a2e-8cb3-94da52150525.
=============================================================

This signature is the #1 in Top Crash Scores in 39.0b1 right now because about half of those crashes happen within the first 60 seconds after startup and overall, this is over 5% of all crashes in this beta.

Graphic Adapter split is as follows:

Intel Corporation 	Core Processor Integrated Graphics Controller	2794 	63.084 %
Intel Corporation 	Mobile 4 Series Chipset Integrated Graphics Controller	1486 	33.552 %
Intel Corporation 	Core Processor Integrated Graphics Controller	118 	2.664 %
Intel Corporation 	4 Series Chipset Integrated Graphics Controller	30 	0.677 %
Intel Corporation 	4 Series Chipset Integrated Graphics Controller	1 	0.023 %

The crashes seem to be related to video, as some comments point to things like:
"i cannot open youtube"
"crashes before videos begin."
"i can,t wach any vido"
"when i want to see videos in youtube i'm getting this thing"
"i was on rapgodfather playing a video then the pluggin container crash everytime i play a video"
etc.

Bas, I thought we already blocked stuff related to this in bug 600152 and http://hg.mozilla.org/mozilla-central/rev/b07c0925efe5 but we are still seeing a spike in the same signature. Can you take a look?
Flags: needinfo?(bas)
Sounds like video issue from the comments. Matt, any idea?

I wouldn't read too much into the other similar bug, it's probably just a similar driver issue.
Component: Graphics → Video/Audio
Flags: needinfo?(bas) → needinfo?(matt.woodrow)
(Reporter)

Comment 2

2 years ago
Anthony, given this is a mostly-startup crash that comments say involves video, can you take a look if we need to do something on the media side here?
Flags: needinfo?(ajones)
Comment hidden (obsolete)
Better stack:

igd10umd32+0x280a1
d3d11!CResource<ID3D11Texture3D>::CLS::FinalConstruct
d3d11!TCLSWrappers<CTexture2D>::CLSFinalConstructFn
d3d11!CLayeredObjectWithCLS<CTexture2D>::FinalConstruct
d3d11!CLayeredObjectWithCLS<CTexture2D>::CreateInstance
d3d11!CDevice::CreateLayeredChild
d3d11!CBridgeImpl<ID3D11LayeredDevice,ID3D11LayeredDevice,CLayeredObject<CDevice> >::CreateLayeredChild
d3d11!CD3D11LayeredChild<ID3D11DeviceChild,NDXGI::CDevice,64>::FinalConstruct
d3d11!NDXGI::CResource::FinalConstruct
d3d11!NDXGI::CDevice::CreateLayeredChild
d3d11!CBridgeImpl<ID3D11LayeredDevice,ID3D11LayeredDevice,CLayeredObject<NDXGI::CDevice> >::CreateLayeredChild
d3d11!NOutermost::CDeviceChild::FinalConstruct
d3d11!CUseCountedObject<NOutermost::CDeviceChild>::CUseCountedObject<NOutermost::CDeviceChild>
d3d11!CUseCountedObject<NOutermost::CDeviceChild>::CreateInstance
d3d11!NOutermost::CDevice::CreateLayeredChild
d3d11!CDevice::CreateAndRecreateLayeredChild<SD3D11LayeredTexture2DCreationArgs>
d3d11!CDevice::CreateTexture2D_Worker
d3d11!CDevice::OpenSharedResourceInternal_Worker
d3d11!CDevice::OpenSharedResource
xul!mozilla::layers::DXGIYCbCrTextureHostD3D11::OpenSharedHandle
xul!mozilla::layers::DXGIYCbCrTextureHostD3D11::Lock
xul!mozilla::layers::ImageHost::Lock
xul!mozilla::layers::AutoLockCompositableHost::AutoLockCompositableHost
xul!mozilla::layers::ImageHost::Composite
xul!mozilla::layers::ImageLayerComposite::RenderLayer
xul!mozilla::layers::RenderLayers<mozilla::layers::ContainerLayerComposite>
xul!mozilla::layers::ContainerRender<mozilla::layers::ContainerLayerComposite>
xul!mozilla::layers::ContainerLayerComposite::RenderLayer
xul!mozilla::layers::RenderLayers<mozilla::layers::ContainerLayerComposite>
xul!mozilla::layers::ContainerRender<mozilla::layers::ContainerLayerComposite>
xul!mozilla::layers::ContainerLayerComposite::RenderLayer
xul!mozilla::layers::LayerManagerComposite::Render
xul!mozilla::layers::LayerManagerComposite::EndTransaction
xul!mozilla::layers::LayerManagerComposite::EndEmptyTransaction
xul!mozilla::layers::CompositorParent::CompositeToTarget
xul!mozilla::layers::CompositorParent::CompositeCallback
xul!RunnableMethod<mozilla::layers::CompositorParent,void (__thiscall mozilla::layers::CompositorParent::*)(mozilla::TimeStamp),Tuple1<mozilla::TimeStamp> >::Run
xul!MessageLoop::DoWork
xul!base::MessagePumpForUI::DoRunLoop
xul!base::MessagePumpWin::RunWithDispatcher
xul!base::MessagePumpWin::Run
xul!MessageLoop::RunHandler
xul!MessageLoop::Run
xul!base::Thread::ThreadMain
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart
Rank 	Adapter driver version 	Count 	%
1 	8.15.10.2086 	6755 	100.00 %

Rank 	Adapter device id 	Count 	%
1 	0x0046 	4028 	59.63 %
2 	0x2a42 	2473 	36.61 %
3 	0x0042 	197 	2.92 %
4 	0x2e12 	49 	0.73 %
5 	0x2e22 	8 	0.12 %

Updated

2 years ago
Duplicate of this bug: 1154771

Updated

2 years ago
Crash Signature: [@ igd10umd32.dll@0x280a1] → [@ igd10umd32.dll@0x280a1] [@ igd10umd64.dll@0x3045b]
It may be triggered by video but I don't see any media code on the stack. Does comment 4 help?
Flags: needinfo?(bas)
(Assignee)

Updated

2 years ago
Flags: needinfo?(milan)
(Assignee)

Comment 8

2 years ago
This has been identified by "stability" as the top issue in 39, it really should get tracking +.
Flags: needinfo?(milan) → needinfo?(sledru)
(In reply to David Major [:dmajor] from comment #7)
> It may be triggered by video but I don't see any media code on the stack.
> Does comment 4 help?

DXGIYCbCrTextureHostD3D11 is mediacode essentially, it just lives inside layers.
Flags: needinfo?(bas)
Note that this code was introduced in https://hg.mozilla.org/mozilla-central/rev/d6242b24bc47 as part of a big media landing.
(Assignee)

Comment 11

2 years ago
This is turning into a needinfo hog :)

Some of the video guys are offline - while we're waiting, Sotaro, can you take a look at the stack in comment 4, and let us know if you have any ideas as to where we should start?  I know it isn't a mobile platform, but it is video so you may come up with something we didn't think of.
Flags: needinfo?(sotaro.ikeda.g)
Given that this is isolated to a single driver version (8.15.10.2086), is there something we could turn off to avoid this codepath?
Liz is running 39. Transferring the n-i to make sure she is aware of this.
Flags: needinfo?(sledru)
(Assignee)

Comment 14

2 years ago
FEATURE_HARDWARE_VIDEO_DECODING feature is available for separate blocklisting as of 36, so if this path is due to that particular feature, yes, we can do it.  We can get a quick exploratory patch with that fix, and see how it affects the crash results.

When is the next beta build?
Flags: needinfo?(milan)
According to the calendar thing, today is go to build for b3, and Monday for b4.
(In reply to Sylvestre Ledru [:sylvestre] from comment #13)
> Liz is running 39. Transferring the n-i to make sure she is aware of this.
Flags: needinfo?(lhenry)
Topcrash, startupcrash, tracking for 39+.

Milan I am about to go to build a little bit later this morning. If you can get me a patch as soon as possible then I can get it into Beta 3.
tracking-firefox39: ? → +
tracking-firefox40: --- → +
tracking-firefox41: --- → +
Flags: needinfo?(lhenry)
If the crash could happen driver independent ways, delete DXGIYCbCrTextureClientD3D11 before DXGIYCbCrTextureHostD3D11::OpenSharedHandle() might cause this problem. DXGIYCbCrTextureClientD3D11 allocates the buffer and send it to Host side.

nical, is there such possibility?
By the way, only on b2g, life time of ImageLayer's TextureClient is ensured by ImageClient::RemoveTextureWithTracker().
  https://dxr.mozilla.org/mozilla-central/source/gfx/layers/client/ImageClient.cpp#81
Flags: needinfo?(sotaro.ikeda.g)
(Assignee)

Comment 20

2 years ago
To continue on the blocklisting - looking at the code, I would expect that bug 1151721 has already blocklisted DXVA for these cards.  So, Liz, probably not a quick patch at this point, need to sort this out.
Flags: needinfo?(nical.bugzilla)
(Assignee)

Comment 21

2 years ago
Jet, there is some confusion over whether these crashes are on DXVA systems or not - can you find somebody with video experience to tell us from the crash reports/stacks if this is going through the DXVA path (in which case we have a blocklist problem (comment 20) or if we have a crash without video hardware acceleration, in which case we don't have a quick blocklisting solution.
Flags: needinfo?(milan) → needinfo?(bugs)
This is definitely the *non* DXVA case.

We can either add a new blacklist type for this (uploading YUV in the content process), or we could just hardcoded a check for this specific driver when setting up mD3D11ImageBridgeDevice.
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(bugs)
(In reply to Sotaro Ikeda [:sotaro] from comment #18)
> If the crash could happen driver independent ways, delete
> DXGIYCbCrTextureClientD3D11 before
> DXGIYCbCrTextureHostD3D11::OpenSharedHandle() might cause this problem.
> DXGIYCbCrTextureClientD3D11 allocates the buffer and send it to Host side.
> 
> nical, is there such possibility?

The KeepUntilFullDeallocation of YCbCrKeeAliveD3D11 in the destructor is supposed to prevent this problem from happening.
Flags: needinfo?(nical.bugzilla)
Created attachment 8615962 [details] [diff] [review]
some extra error checks

This will probably not fix the issue but still, let's add a bit of paranoiac error checks around the code that initializes the textures on the client side, if only to discard the doubt that something bad happened there.
Attachment #8615962 - Flags: review?(bas)
Following up on sotaro's idea that the OpenSharedHandle could be called after the destruction of the object itself, perhaps it could be that the device was lost before OpenSharedHandle (rather than the texture being destroyed)?
Most D3D calls would just return DXGI_ERROR_DEVICE_REMOVED rather than crash but who knows...
Created attachment 8615977 [details] [diff] [review]
some extra error checks

Sorry, missed a return true in the previous patch
Attachment #8615962 - Attachment is obsolete: true
Attachment #8615962 - Flags: review?(bas)
Attachment #8615977 - Flags: review?(bas)
Attachment #8615977 - Flags: review?(bas) → review+

Comment 27

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/17467c9adf1b

Updated

2 years ago
Whiteboard: [leave-open]

Comment 28

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/0f448c718d9f
So this driver version has always been blacklisted for D2D. That means we've never used texture sharing here, likely texture sharing is just bad on this driver. Let's not do it. We might even simply want to leach this off of D2D enabled (i.e. for an initial patch simply do if D2D is for any reason not used don't use texture sharing)

Comment 30

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/c986df6d8b8c
(Assignee)

Comment 31

2 years ago
(In reply to Bas Schouten (:bas.schouten) from comment #29)
> So this driver version has always been blacklisted for D2D. That means we've
> never used texture sharing here, likely texture sharing is just bad on this
> driver. Let's not do it. We might even simply want to leach this off of D2D
> enabled (i.e. for an initial patch simply do if D2D is for any reason not
> used don't use texture sharing)

I like this - let's get a patch for that?  Blacklisted D2D means no texture sharing anywhere, for anything.  Or more to the point, if we decide that we don't want texture sharing for compositor, we should decide the same everywhere.

Even if this disqualifies some systems that would otherwise work, we really don't need the complexity of one more code path.

We don't have one of the devices above, but we do have another GMAX4500HD, we'll see if we can reproduce the problem with this 8.15.10.2086 driver.
Flags: needinfo?(ajones)
https://hg.mozilla.org/mozilla-central/rev/17467c9adf1b
https://hg.mozilla.org/mozilla-central/rev/0f448c718d9f
https://hg.mozilla.org/mozilla-central/rev/c986df6d8b8c
I did an initial patch to just block this feature for this specific driver - https://hg.mozilla.org/try/rev/0715be474eeb

I prefer Bas' idea though, that sounds simpler and might catch other issues in the future.

I'm about to get on a plane, but will write a patch for this on Monday if nobody gets to it before then.
OK, up to you all.   Sounds good!
I'm aiming to get everything on mozilla-beta by 9am PDT on Monday. 
Otherwise this may have to wait for Beta 5.
This has a super high crash score on https://crash-analysis.mozilla.com/rkaiser/crash-report-tools/score/?version=39.0b3&limit=30.

I don't think we should go another beta release with this unfixed. Any chance to get comment 31 on beta Monday morning? Otherwise we should see if Liz is willing to grant an extension...
Flags: needinfo?(milan)
Created attachment 8616512 [details] [diff] [review]
sharing-crash
Attachment #8616512 - Flags: review?(bas)
I'm aiming to go to build by around 2pm PDT which would mean we need to merge this into beta (and allow 3 or so hours for the tests to run) before 11am.   If that helps!  If we need to wait longer, we can do that, but it means some delay before QE can test things.
Attachment #8616512 - Flags: review?(bas) → review+
Comment on attachment 8616512 [details] [diff] [review]
sharing-crash

Approval Request Comment
[Feature/regressing bug #]: Bug 1138967
[User impact if declined]: Crashes for users with specific driver
[Describe test coverage new/current, TreeHerder]: None, but very simple change to fallback to existing code path.
[Risks and why]: Low risk.
[String/UUID change made/needed]: None
Attachment #8616512 - Flags: approval-mozilla-beta?
Attachment #8616512 - Flags: approval-mozilla-aurora?

Comment 39

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/73dda324715f
Comment on attachment 8616512 [details] [diff] [review]
sharing-crash

Approved for uplift to aurora and beta. After discussion with Ryan we want to put this into beta 4 now but with the previous build prepared as a fallback in case this doesn't pass tests on the merge to beta.
Attachment #8616512 - Flags: approval-mozilla-beta?
Attachment #8616512 - Flags: approval-mozilla-beta+
Attachment #8616512 - Flags: approval-mozilla-aurora?
Attachment #8616512 - Flags: approval-mozilla-aurora+
I can reproduce this crash.
https://hg.mozilla.org/releases/mozilla-beta/rev/4241def0561b
status-firefox39: affected → fixed
I've confirmed that sharing alpha textures is broken in this driver.
There's a test program here: https://github.com/jrmuizel/d3d-tests/blob/master/alpha-texture-sharing.cc

The test program crashes on 8.15.10.2086 but not on 8.15.10.2302 or 8.15.10.2869
(Assignee)

Comment 45

2 years ago
(In reply to Jeff Muizelaar [:jrmuizel] from comment #44)
> There's a test program here:
> https://github.com/jrmuizel/d3d-tests/blob/master/alpha-texture-sharing.cc
> 
> The test program crashes on 8.15.10.2086 but not on 8.15.10.2302 or
> 8.15.10.2869

We may be able to use this information to relax the current blocklisting eventually.
Flags: needinfo?(milan)
https://hg.mozilla.org/mozilla-central/rev/73dda324715f
https://hg.mozilla.org/releases/mozilla-aurora/rev/58d4afb8473a
status-firefox40: affected → fixed
(Assignee)

Updated

2 years ago
Assignee: nobody → matt.woodrow
Using an R8 texture instead of an A8 texture does not crash.
(Reporter)

Comment 49

2 years ago
The crash signature is significantly lower on 39.0b5 with this patch but it is still significant enough to be a concern, esp. as it looks like the remaining crashes are almost all on startup now.
Can you link to some of them? I don't know how to find them in Soccoro.
(Reporter)

Comment 51

2 years ago
(In reply to Jeff Muizelaar [:jrmuizel] from comment #50)
> Can you link to some of them? I don't know how to find them in Soccoro.

https://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox:39.0b4&signature=igd10umd32.dll%400x280a1#tab-reports
(In reply to Robert Kaiser (:kairo@mozilla.com) - on vacation or slow to reply until the end of June from comment #51)
> (In reply to Jeff Muizelaar [:jrmuizel] from comment #50)
> > Can you link to some of them? I don't know how to find them in Soccoro.
> 
> https://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox:
> 39.0b4&signature=igd10umd32.dll%400x280a1#tab-reports

How did you get this list?
(Reporter)

Comment 53

2 years ago
(In reply to Jeff Muizelaar [:jrmuizel] from comment #52)
> (In reply to Robert Kaiser (:kairo@mozilla.com) - on vacation or slow to
> reply until the end of June from comment #51)
> > (In reply to Jeff Muizelaar [:jrmuizel] from comment #50)
> > > Can you link to some of them? I don't know how to find them in Soccoro.
> > 
> > https://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox:39.0b4&signature=igd10umd32.dll%400x280a1#tab-reports
> 
> How did you get this list?

I went into a top crash report for 39.0b4 on crash-stats (by selecting that from the dropdowns), and clicked on this signature (and then redacted the date fields out of the URL so when you go there you'll always get the last 7 days).
Created attachment 8621646 [details] [diff] [review]
Sharing crash

Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=558bbb000bea

It appears that not all affected devices on all versions of windows were actually blacklisted for direct2d with this driver version, so the existing patch didn't catch all affected machines.

We found a machine in the Toronto office which can reproduce this and tested with a variety of different driver version. Both the driver release before (2082) and after (2092) do not crash, so it appears it is literally just this version that has the bug.

The new patch just checks for this specific version and disables using the broken functionality.
Attachment #8621646 - Flags: review?(jmuizelaar)
Attachment #8621646 - Flags: review?(jmuizelaar) → review+

Comment 55

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/5a82441265b8
Comment on attachment 8621646 [details] [diff] [review]
Sharing crash

Approval Request Comment
[Feature/regressing bug #]: Bug 1138967
[User impact if declined]: Crashes for users with specific driver
[Describe test coverage new/current, TreeHerder]: None, but very simple change to fallback to existing code path.
[Risks and why]: Low risk.
[String/UUID change made/needed]: None

This is even more targeted than the previous attempt, and we've confirmed locally that only this exact driver version is affected.
Attachment #8621646 - Flags: approval-mozilla-beta?
Attachment #8621646 - Flags: approval-mozilla-aurora?
https://hg.mozilla.org/mozilla-central/rev/5a82441265b8
Comment on attachment 8621646 [details] [diff] [review]
Sharing crash

Approved for uplift to aurora and beta. Crash fix.
Attachment #8621646 - Flags: approval-mozilla-beta?
Attachment #8621646 - Flags: approval-mozilla-beta+
Attachment #8621646 - Flags: approval-mozilla-aurora?
Attachment #8621646 - Flags: approval-mozilla-aurora+
Still the #3 topcrash for 39.0b5.
I just mixed this up with another bug. This is actually #69 crasher in 39.0b5. So that is a big improvement.
I guess we missed beta 6 as well. These flags should have been reset to alert the uplift queries.

41 is looking good.
status-firefox39: fixed → affected
status-firefox40: fixed → affected
status-firefox41: affected → fixed
https://hg.mozilla.org/releases/mozilla-aurora/rev/47d4d88efa1d
status-firefox40: affected → fixed
https://hg.mozilla.org/releases/mozilla-beta/rev/f398d95abaa9
status-firefox39: affected → fixed
Flags: qe-verify+
(Assignee)

Comment 64

2 years ago
No crashes since this landed that I can see.
Looking in the crash-stats (https://crash-stats.mozilla.com/report/list?product=Firefox&range_unit=days&range_value=28&signature=igd10umd32.dll%400x280a1#tab-reports) I can see still a few crashes with Firefox 39 RC build 4 (20150624153222) [1] and with Firefox 39 RC build 5 (20150626112833) [2] and only one crash with Aurora 40.0a2 (20150619004003) [3]. All crashes happened after the fix landed.

igd10umd64.dll@0x3045b signature (https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=28&signature=igd10umd64.dll%400x3045b#tab-reports) recorded two crashes after the fix, one on Aurora 40.0a2 (20150621004005) [4] and one on Nightly 41.0a1 (20150621030204) [5].

[1] https://crash-stats.mozilla.com/report/index/bab1d979-92d0-4bfa-a91d-14d1b2150627
    https://crash-stats.mozilla.com/report/index/e6d7ec1f-d7bf-48e7-9d66-cd9132150629
[2] https://crash-stats.mozilla.com/report/index/94066fb6-253c-4f92-b5da-f51b22150628
    https://crash-stats.mozilla.com/report/index/5fbb2eb5-a019-4747-afdb-03c272150628 
[3] https://crash-stats.mozilla.com/report/index/ed0e1d9c-4f0f-426d-8db6-b31392150624
[4] https://crash-stats.mozilla.com/report/index/8c921103-abcc-48fe-9222-452352150622
[5] https://crash-stats.mozilla.com/report/index/49dcad9d-1cbd-423c-8384-0dc812150630
Component: Audio/Video → Audio/Video: Playback
Component: Audio/Video: Playback → Graphics
I see 909 crashes with Firefox 41.0.2 over the last week in igd10umd32.dll@0x280a1 but I'm not clear if they're what's tracked in this bug report. Milan, does this need to be tracked?
Flags: needinfo?(milan)
(Assignee)

Comment 67

2 years ago
We should track it because it came back, probably just a few days later with bug 1173983?  Matt, Jeff, we still have these crashes (e.g., see https://crash-stats.mozilla.com/report/index/7d40dc02-7498-4477-93ae-921e42151103), and it seems that driver is bad, but it passes whatever test we want things to pass.  Or am I reading the code wrong?

Or, is this a different crash, which just happens to have the same top address (heck, could be "assert" for all I know :)
Flags: needinfo?(milan)
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(jmuizelaar)
We can reproduce this crash locally.
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(jmuizelaar)
Sorry wrong bug.
Flags: needinfo?(matt.woodrow)
[Tracking Requested - why for this release]: nominating to track based on comment 67.
tracking-firefox42: --- → ?
Does anyone know how dmajor got a decent stack trace from the original report? I assume windbg, which I can try out next week (don't have my windows machine with me this week).

We reproduced this in the Toronto office previously (though it doesn't appear we noted which machine), so I assume we can do that again.

It would be nice to check if the alpha texture sharing check added for this specific driver is actually working.

If it is then we might have a more general crash with this driver, so maybe we should blacklist it entirely.
Flags: needinfo?(jmuizelaar)
Flags: needinfo?(matt.woodrow)
(Reporter)

Comment 72

2 years ago
(In reply to Matt Woodrow (:mattwoodrow) from comment #71)
> Does anyone know how dmajor got a decent stack trace from the original
> report? I assume windbg, which I can try out next week (don't have my
> windows machine with me this week).

AFAIK, yes, he loaded the minidump into WinDbg, with the symbol server set to the Mozilla one so it would find those, and that usually produces better stack traces than what the minidump stackwalker can do on the server.

Updated

2 years ago
Summary: crash spike in igd10umd32.dll@0x280a1 with 39.0 beta → crash spike in igd10umd32.dll@0x280a1 with 39.0 beta (Specific to Intel Driver 8.15.10.2086)
[Tracking Requested - why for this release]:
Too late for 42 but we could add that for 43.
status-firefox42: --- → wontfix
status-firefox43: --- → ?
tracking-firefox42: ? → -
tracking-firefox43: --- → ?
Tracking for 43+ since this crash signature is still showing up, with 138 crashes on 43.0b1 in the last week.
status-firefox43: ? → affected
status-firefox44: --- → affected
status-firefox45: --- → affected
tracking-firefox43: ? → +
tracking-firefox44: --- → +
tracking-firefox45: --- → +
I tried reproducing some kind of crash on a 0x2a42 with 8.15.10.2086 and could not reproduce any problems on youtube with DXVA on and with it off. I'm not sure what to do next. How severe is this crash?
Flags: needinfo?(jmuizelaar)
(In reply to Jeff Muizelaar [:jrmuizel] from comment #75)
> How severe is this crash?

Here is some data I pulled out of Socorro for Firefox 42:
* 615 crashes in igd10umd32.dll@0x280a1, 0 in igd10umd64.dll@0x3045b, 100% report driver 8.15.10.2086
* This crash is #46 for all Fx42 users but #1 for Fx42 users on 8.15.10.2086
* This crash is 49% of the total crash volume for users with Intel 8.15.10.2086
* Users with Intel 8.15.10.2086 account for 0.66% of the total crash volume for all Intel chipset users

My best assessment would be that this certainly isn't our most severe issue but it's likely severe for those hundreds of people using this Intel driver. Could this be worked around with a blocklist?
I've just reproduced this locally. I did by running video for a long time. I'll try to do it again.
Any luck here? Or should we be wontfixing this for 43?
Flags: needinfo?(jmuizelaar)
(Assignee)

Comment 79

2 years ago
Let's won't fix for 43.
Flags: needinfo?(jmuizelaar)
I saw only about 10 instances of this crash on FF44.0a2 based on 28 days of crash data. If this doesn't get fixed in the next 2-3 weeks, I will be inclined to wontfix for 44.
(Assignee)

Comment 81

a year ago
We're going to try and find out if the number went down because we lost all the users with this configuration, or because something got better.
status-firefox43: affected → wontfix
igd10umd32.dll@0x7f99 shows up as new in the Firefox Release explosiveness report and ranks #48 in 43.0.1 with 512 crashes (0.25%): https://crash-stats.mozilla.com/report/list?signature=igd10umd32.dll%400x7f99

Note the following driver breakdown:
1 	8.15.10.1892 	540 	39.82 %
2 	8.15.10.1883 	377 	27.80 %
3 	8.15.10.1855 	171 	12.61 %
4 	8.15.10.1872 	150 	11.06 %
5 	8.15.10.1994 	87 	6.42 %
6 	8.15.10.1851 	28 	2.06 %
7 	8.15.10.2020 	1 	0.07 %
8 	8.652.0.0 	1 	0.07 %
9 	8.672.4.0 	1 	0.07 %

Is this another manifestation of the issue in this bug report?
Johnny, Jet: This is a crash that has been getting wontfix'd for too many releases. Copied below is the crash data (for the first signature) for 7 days. This is not a top crasher (ranked #39 atm). Would you be able to help prioritize investigation on this one? 

Product 	Version Percentage Number Of Crashes
Firefox 	44.0b6 	9.06% 	163
Firefox 	44.0b4 	7.11% 	128
Firefox 	44.0b7 	2.83% 	51
Firefox 	44.0b1 	1.06% 	19
Firefox 	44.0b2 	0.56% 	10
Flags: needinfo?(jst)
Flags: needinfo?(bugs)
(Assignee)

Comment 84

a year ago
The "wontfix'd for too many release" doesn't quite apply - this bug is a bit of a meta bug for a few different problems, which is why the "leave open" is in place - but they are related and there are still some crashes with this signature.
Can we ask something at Intel about this crash just going by the crash address and driver version?
(Assignee)

Comment 86

a year ago
The conversation with Intel started, but nothing tangible happened yet.
Not much that can be done for Fx44, now a wontfix.
status-firefox44: affected → wontfix
Milan, any further followups with Intel around this issue?
Flags: needinfo?(jst) → needinfo?(milan)
(Assignee)

Comment 89

a year ago
No information from Intel.  I'm not holding my breath, these are old(er) drivers and systems.  We are blocklisting these since about 2/9/16 (bug 1207993), although, based on comment 82 and other data, actually not blocking the driver version that's in the summary of this bug.
Flags: needinfo?(milan)
OK, so we're blocking many of the drivers Anthony mentions in comment 82 but not the one in the bug summary (presumably it isn't in high enough volume to warrant blocking at this point). 
Blocked:
8.15.10.1851
8.15.10.1855
8.15.10.1872
8.15.10.1883
8.15.10.1892
8.15.10.1994

Should we try blocking the driver in the summary? Or is that unlikely to be fruitful? If not, then we should stop tracking this and move on. It isn't a startup crash and while it's in the top 50 for 44.0.2 it's around #48 (https://crash-stats.mozilla.com/report/list?signature=igd10umd32.dll%400x280a1) 

Anthony what do you think? How do you come up with your list of most commonly seen drivers for a signature? Is there an obvious candidate to add to our blocklist?
Flags: needinfo?(anthony.s.hughes)
According to supersearch this shows up exclusively on 8.15.10.2086:
https://crash-stats.mozilla.com/search/?product=Firefox&signature=%3Digd10umd32.dll%400x280a1&signature=%3Digd10umd64.dll%400x3045b&_facets=signature&_facets=adapter_driver_version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-adapter_driver_version

Firefox 44.0.2 has seen 1327 crashes over the last week which would place it around #46 on Release.
Flags: needinfo?(anthony.s.hughes)
(Assignee)

Comment 92

a year ago
May as well add that one as well:

Jorge, could you add this one to the downloadable list?

<gfxBlacklistEntry>
  <os>All</os>
  <vendor>0x8086</vendor>
  <devices>
    <device>0x2a42</device>
    <device>0x2e22</device>
    <device>0x2e12</device>
    <device>0x2e32</device>
    <device>0x0046</device>
  </devices>
  <featureStatus>BLOCKED_DRIVER_VERSION</featureStatus>
  <driverVersion>8.15.10.2086</driverVersion>
  <driverVersionComparator>EQUAL</driverVersionComparator>
</gfxBlacklistEntry>
Flags: needinfo?(jorge)
Switching assignee to Milan since I don't think there's anything actionable for me to do here.
Assignee: matt.woodrow → milan
Done.
Flags: needinfo?(jorge)
(Assignee)

Comment 95

a year ago
If we see the reduction in crashes, we should just close this bug.
Flags: needinfo?(bugs)
Whiteboard: [leave-open]
When will we know whether we've done what we can do here?
Flags: needinfo?(milan)
I think we can call this done. There is a big reduction in crashes after Feb. 10 and then another drop after Feb. 15. There are still a few crashes on beta 10 with this signature and 1 on 45.0b99. But clearly we got most of them.  

Milan and Sylvestre, your call if you want to close this and declare it fixed for 45.
Status: NEW → RESOLVED
Last Resolved: a year ago
Flags: needinfo?(sledru)
Resolution: --- → FIXED
(Assignee)

Comment 98

a year ago
Closing this works for me.
Flags: needinfo?(milan)
Untracking for 45.
tracking-firefox45: + → -
17 crashes with 45, not too much indeed
Flags: needinfo?(sledru)
You need to log in before you can comment on or make changes to this bug.