Closed Bug 590373 Opened 9 years ago Closed 9 years ago

Crash mainly at startup [@ _VEC_memzero] with MS hotfix KB2454826 installed (automatic on 1/11/11), mainly dual graphics, mainly Intel driver version 8.15.10.2141 and below

Categories

(Core :: Graphics, defect, critical)

x86
Windows 7
defect
Not set
critical

Tracking

()

RESOLVED FIXED
Tracking Status
blocking2.0 --- betaN+

People

(Reporter: RyanVM, Assigned: bjacob)

References

(Depends on 1 open bug, Blocks 2 open bugs)

Details

(Keywords: crash, topcrash, Whiteboard: [#1 topcrash on Fx4.0b8 and Fx4.0b9][hardblocker])

Crash Data

Attachments

(2 files, 1 obsolete file)

After installing the new MS hotfix for graphics performance released today:
http://www.microsoft.com/downloads/details.aspx?FamilyId=e04dd3bd-c34b-42ef-978b-10b73e7d5bba&displaylang=en

I get crashes on startup immediately after the main window loads.
http://crash-stats.mozilla.com/report/index/bp-88680c8a-dee6-4db0-b9dd-bbade2100824
blocking2.0: --- → ?
I can't reproduce this. You rebooted after installing it, as it asked, right?
Yes
Works fine for me.  Using an ATI HD5870 with latest drivers.

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b5pre) Gecko/20100824 Minefield/4.0b5pre - Build ID: 20100824170052
Ugh, still no symbols :s.
OK, apparently Intel updated their drivers on their website but not through Windows Update. I upgraded from the v2141 to v2182 drivers and it's not crashing anymore. Maybe a driver issue after all?
Interesting. I suspect that we won't need to worry about this once Microsoft pushes this hotfix out in a service pack - at least, I hope not.

Something I'm interested in is if, with the old driver, you install IE9 (installing the three hotfixes it requires) Firefox still crashes on startup. If one of the other two hotfixes also fixes this problem, we have less to worry about.
ryan/joe: there's a .dump file available for the original crash report. That means that we should be able to get a better stack for this if you want it:

UUID	88680c8a-dee6-4db0-b9dd-bbade2100824
Signature [Expand]	Source
_VEC_memzero	
d2d1.dll@0x42e44	
d2d1.dll@0x4286c	
d2d1.dll@0x43627	
d2d1.dll@0x322f2	
d2d1.dll@0x5f049	
d2d1.dll@0x334ea	
d2d1.dll@0x5ed60	
d2d1.dll@0x28027	
d2d1.dll@0x26e73	
d2d1.dll@0x27521	
d2d1.dll@0x1e5e0	
_cairo_d2d_flush	gfx/cairo/cairo/src/cairo-d2d-surface.cpp:2555
_moz_cairo_surface_flush	gfx/cairo/cairo/src/cairo-surface.c:974
_moz_cairo_destroy	gfx/cairo/cairo/src/cairo.c:344
mozilla::layers::BasicThebesLayer::Paint	gfx/layers/basic/BasicLayers.cpp:426
mozilla::layers::BasicLayerManager::PaintLayer	gfx/layers/basic/BasicLayers.cpp:1069
mozilla::layers::BasicLayerManager::PaintLayer	gfx/layers/basic/BasicLayers.cpp:1077
mozilla::layers::BasicLayerManager::EndTransaction	gfx/layers/basic/BasicLayers.cpp:977
nsDisplayList::PaintForFrame	layout/base/nsDisplayList.cpp:395
nsLayoutUtils::PaintFrame	layout/base/nsLayoutUtils.cpp:1409
PresShell::Paint	layout/base/nsPresShell.cpp:5915
nsViewManager::RenderViews	view/src/nsViewManager.cpp:459
nsViewManager::Refresh	view/src/nsViewManager.cpp:425
nsViewManager::DispatchEvent	view/src/nsViewManager.cpp:940
AttachedHandleEvent	view/src/nsView.cpp:193
nsWindow::DispatchEvent	widget/src/windows/nsWindow.cpp:3485
nsWindow::DispatchWindowEvent	widget/src/windows/nsWindow.cpp:3516
nsWindow::OnPaint	widget/src/windows/nsWindowGfx.cpp:567
nsWindow::ProcessMessage	widget/src/windows/nsWindow.cpp:4696
nsWindow::WindowProcInternal	widget/src/windows/nsWindow.cpp:4287
nsWindow::WindowProc	widget/src/windows/nsWindow.cpp:4239
InternalCallWinProc	
NtUserGetDC	
DispatchClientMessage	
__fnDWORD	
ntdll.dll@0x100e5	
DispatchMessageW	
nsBaseAppShell::OnProcessNextEvent	widget/src/xpwidgets/nsBaseAppShell.cpp:312
xul.dll@0x1cdd8f	
nsXPConnect::Push	js/src/xpconnect/src/nsXPConnect.cpp:2670
nsThread::ProcessNextEvent	xpcom/threads/nsThread.cpp:517
mozilla::ipc::MessagePump::Run	ipc/glue/MessagePump.cpp:134
xul.dll@0xb8cd53	
MessageLoop::RunInternal	ipc/chromium/src/base/message_loop.cc:219
MessageLoop::RunHandler	ipc/chromium/src/base/message_loop.cc:202
EmitNameOp	js/src/jsemit.cpp:2622
_SEH_epilog4	
MessageLoop::Run	ipc/chromium/src/base/message_loop.cc:176
No problems here after installing this performance hotfix. Win7 x64 with nVidia GTX260 card. No crashes on startup with either win32 or win64 latest nightly builds. So could be really Intel GMA related.
I agree that this was likely a driver issue. D2D enabled (nightlies) was crashing for me consistently with the same video card as the reporter. Installing the latest driver from Intel resolved the crashes.
Depending on what Microsoft does with SP1 and/or IE9, we may need to block on this in the future, but I think that just putting this into the driver blacklist would be enough.
blocking2.0: ? → -
It is #78 top crasher in 4.0b8pre for the last week.
It seems to happen only with Intel HD graphics and graphic driver version 8.15.10.2141.
Chris, can we have a breakdown by ID card and by igd10umd32.dll version to confirm that?
Blocks: 605779
Summary: Crash [@ _VEC_memzero] on startup with D2D enabled and MS performance hotfix installed → Crash [@ _VEC_memzero] at startup with D2D enabled, MS performance hotfix KB2028560 installed, Intel HD graphic card and graphic driver 8.15.10.2141
I'm finding a few reports with igd10umd32.dll 8.15.10.2125 like these.

igd10umd32.dll 8.15.10.2125     
Windows NT 6.1.7600     
AdapterVendorID: 8086, AdapterDeviceID: 0046    
20101030-crashdata.csv:_VEC
_memzero     
http://crash-stats.mozilla.com/report/index/462d6ffe-3102-4fe7-b88b-a75ba2101030

igd10umd32.dll 8.15.10.2125     Windows NT 6.1.7600     AdapterVendorID: 8086, AdapterDeviceID: 0046    20101030-crashdata.csv:_VEC
_memzero     http://crash-stats.mozilla.com/report/index/746d8917-95ab-4c9e-b1d5-4e8a02101030

igd10umd32.dll 8.15.10.2125     Windows NT 6.1.7600     AdapterVendorID: 8086, AdapterDeviceID: 0046    20101030-crashdata.csv:_VEC
_memzero     http://crash-stats.mozilla.com/report/index/91755c8b-3cc1-4c97-9a49-610ce2101030

igd10umd32.dll 8.15.10.2125     Windows NT 6.1.7600     AdapterVendorID: 8086, AdapterDeviceID: 0046    20101030-crashdata.csv:_VEC
_memzero     http://crash-stats.mozilla.com/report/index/8cf7d76e-516b-4620-b7db-68f422101030

igd10umd32.dll 8.15.10.2125     Windows NT 6.1.7600     AdapterVendorID: 8086, AdapterDeviceID: 0046    20101030-crashdata.csv:_VEC
_memzero     http://crash-stats.mozilla.com/report/index/951b1cca-4d3c-493f-97a2-2b3fd2101030

igd10umd32.dll 8.15.10.2125     Windows NT 6.1.7600     AdapterVendorID: 8086, AdapterDeviceID: 0046    20101030-crashdata.csv:_VEC
_memzero     http://crash-stats.mozilla.com/report/index/b1cf8c36-9c3c-4dfe-9f94-d11192101030

more in the attachment coming up.
here is a stat summary.  we might need to refine this because my serach appears to have picked up a few hang reports with similar signatures like 
 hang | _VEC_memzero


159 igd10umd32.dll 8.15.10.2141 Windows NT 6.1.7600 AdapterVendorID: 8086, AdapterDeviceID: 0046
  90 igd10umd32.dll 8.15.10.2125 Windows NT 6.1.7600 AdapterVendorID: 8086, AdapterDeviceID: 0046
  15 igd10umd32.dll 8.15.10.2141 Windows NT 6.1.7600 AdapterVendorID: 8086, AdapterDeviceID: 2a42
  11 igd10umd32.dll 8.15.10.2141 Windows NT 6.1.7601 Service Pack 1, v.153 AdapterVendorID: 8086, AdapterDeviceID: 0046
   8 igd10umd32.dll 8.15.10.2125 Windows NT 6.1.7600 AdapterVendorID: 8086, AdapterDeviceID: 0042
   7 igd10umd32.dll 8.15.10.2141 Windows NT 6.1.7601 Service Pack 1, v.721 AdapterVendorID: 8086, AdapterDeviceID: 0046
   7 igd10umd32.dll 8.15.10.2141 Windows NT 6.1.7600 AdapterVendorID: 8086, AdapterDeviceID: 2e32

and several reports where no version is reported for this signature.
  13 Windows NT 6.1.7600 AdapterVendorID: 1002, AdapterDeviceID: 95c5 20101101-crashdata.csv:_VEC_memzero
   7 Windows NT 6.1.7600 AdapterVendorID: 1002, AdapterDeviceID: 95c4 20101030-crashdata.csv:_VEC_memzero
   6 Windows NT 6.1.7601 Service Pack 1, v.178 AdapterVendorID: 1002, AdapterDeviceID: 9616 20101102-crashdata.csv:_VEC_memzero
   5 Windows NT 6.0.6002 Service Pack 2 \N 20101101-crashdata.csv:_VEC_memzero
   5 Windows NT 5.1.2600 Service Pack 3 \N 20101101-crashdata.csv:_VEC_memzero
   4 igd10umd32.dll 8.15.10.2131 Windows NT 6.1.7600 AdapterVendorID: 8086, AdapterDeviceID: 0046
   4 Windows NT 5.1.2600 Service Pack 3 \N 20101102-crashdata.csv:_VEC_memzero
   4 Windows NT 5.1.2600 Service Pack 3 \N 20101031-crashdata.csv:_VEC_memzero
   4 Windows NT 5.1.2600 Service Pack 2 \N 20101031-crashdata.csv:_VEC_memzero
   3 igd10umd32.dll 8.15.10.2125 Windows NT 6.1.7601 Service Pack 1, v.721 AdapterVendorID: 8086, AdapterDeviceID: 0042
   3 Windows NT 5.1.2600 Service Pack 2 \N 20101102-crashdata.csv:_VEC_memzero
   2 Windows NT 6.1.7600 \N 20101102-crashdata.csv:hang | _VEC_memzero
   2 Windows NT 6.1.7600 \N 20101102-crashdata.csv:_VEC_memzero
   2 Windows NT 6.1.7600 \N 20101030-crashdata.csv:hang | _VEC_memzero
   2 Windows NT 5.1.2600 Service Pack 3 \N 20101102-crashdata.csv:hang | _VEC_memzero
   2 Windows NT 5.1.2600 Service Pack 3 \N 20101101-crashdata.csv:hang | _VEC_memzero
   2 Windows NT 5.1.2600 Service Pack 3 \N 20101031-crashdata.csv:hang | _VEC_memzero
< long tail snipped>
Here are the latest crash reports:
http://crash-stats.mozilla.com/report/list?range_value=4&range_unit=weeks&signature=_VEC_memzero&version=Firefox%3A4.0b8pre

From the fixing of bug 600152 in 4.0b7pre/20100929, the minimum Intel driver version for the most recent cards and D2D/D3D features is 8.15.10.2202.

As there are crashes at startup with Intel card in 4.0b8pre, it seems that the driver blacklisting for D2D/D3 features is not taken into account during at least the first 14 seconds after the start of Firefox.
Summary: Crash [@ _VEC_memzero] at startup with D2D enabled, MS performance hotfix KB2028560 installed, Intel HD graphic card and graphic driver 8.15.10.2141 → Crash [@ _VEC_memzero] at startup with D2D enabled, MS performance hotfix KB2028560 installed, mainly with Intel driver version 8.15.10.2141 and below
It is #67 top crasher in 4.0b7 for the last week.

According to some comments, it happens with dual graphics, so Intel driver blacklisting does not work in that case, as Intel graphics is the secondary card, i.e. AdapterVendorID is either ATI or NVIDIA.

As there are a lot of bugs that implicated dual/switchable graphics, it would be helpful to handle several cards, for driver blacklisting and Troubleshooting page graphics section.
Summary: Crash [@ _VEC_memzero] at startup with D2D enabled, MS performance hotfix KB2028560 installed, mainly with Intel driver version 8.15.10.2141 and below → Crash [@ _VEC_memzero] at startup with D2D enabled, MS performance hotfix KB2028560 installed, with dual graphics, mainly with Intel driver version 8.15.10.2141 and below
Blocks: 605780
It is #37 top crasher in 4.0b8 for the last week.
blocking2.0: - → ?
Summary: Crash [@ _VEC_memzero] at startup with D2D enabled, MS performance hotfix KB2028560 installed, with dual graphics, mainly with Intel driver version 8.15.10.2141 and below → Crash mainly at startup [@ _VEC_memzero] with MS performance hotfix KB2028560 installed, dual graphics, mainly Intel driver version 8.15.10.2141 and below
I don't think #37 is high enough to block on, but I do think this should be looked at in the near future.
blocking2.0: ? → .x
It is #1 top crasher in 4.0b8 and 4.0b9. It is seldom for a startup crash.
A lot of new users can't even start Firefox.
I request a BetaN blocking.
Keywords: crash, topcrash
Please note that this crash can happen without dual graphics. My laptop has only Intel integrated graphics.
Summary: Crash mainly at startup [@ _VEC_memzero] with MS performance hotfix KB2028560 installed, dual graphics, mainly Intel driver version 8.15.10.2141 and below → Crash mainly at startup [@ _VEC_memzero] with MS performance hotfix KB2028560 installed, mainly dual graphics, mainly Intel driver version 8.15.10.2141 and below
MS performance hotfix KB2028560 should be installed manually.
MS performance hotfix KB2454826 (which includes KB2028560) has been installed automatically by Windows Update on 1/11/11. It explains the spike.
Summary: Crash mainly at startup [@ _VEC_memzero] with MS performance hotfix KB2028560 installed, mainly dual graphics, mainly Intel driver version 8.15.10.2141 and below → Crash mainly at startup [@ _VEC_memzero] with MS hotfix KB2454826 installed (automatic on 1/1/11), mainly dual graphics, mainly Intel driver version 8.15.10.2141 and below
Summary: Crash mainly at startup [@ _VEC_memzero] with MS hotfix KB2454826 installed (automatic on 1/1/11), mainly dual graphics, mainly Intel driver version 8.15.10.2141 and below → Crash mainly at startup [@ _VEC_memzero] with MS hotfix KB2454826 installed (automatic on 1/11/11), mainly dual graphics, mainly Intel driver version 8.15.10.2141 and below
BJacob, let's blacklist that intel driver version, and cry knowing an MS update can cause this.
As stated in comment 15, these versions are already blacklisted for D2D/D3D features!
Is the blacklist doing some wrong comparison then? Because these are clearly machines that have D2D loaded. There should not be any ignoring of the blacklist in the first <x> seconds before startup.
(In reply to comment #24)
> Is the blacklist doing some wrong comparison then? Because these are clearly
> machines that have D2D loaded. There should not be any ignoring of the
> blacklist in the first <x> seconds before startup.

Never mind, I should learn to read the bug properly first before posting. So we need the ability to consider all graphics cards considered on the system, and assume RyanVM is an 'exception'.
Considering the raised prevalence this should probably get up'ed in blocker status.
blocking2.0: .x → ?
Comment 15 about the blacklisting by-passing is applicable when Intel's GPU is the first or single card.
Comment 16 about the dual graphics managing is applicable when Intel's GPU is the secondary card.
Ouch, so many Intel users still haven't updated their drivers from looking at the crash logs. Intel G45(X4500HD/MHD IGP) chipset users should install the latest drivers for their IGP, 8.15.10.2226. 

32bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=19468&ProdId=2991&lang=eng

64bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=19469&ProdId=2991&lang=eng

For Intel HD Graphics(Clarkdale/Arrandale) and the newer HD Graphics 2000/3000(Sandy Bridge) users, you should install the latest drivers for those IGPs, 8.15.10.2266.

32bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3319&DwnldID=19645

64bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3319&DwnldID=19647
Would be nice if Intel could get Microsoft to push them out over Windows Update.
But at least this report states Intel HD graphics as the adapter/vendor:

http://crash-stats.mozilla.com/report/index/c439394d-2c11-44bd-a470-5b4a32110117

AdapterVendorID: 8086, AdapterDeviceID: 0046

igd10umd32.dll 	8.15.10.2141

How come we're not blacklisting that??
Whiteboard: [#1 topcrash on Fx4.0b8 and Fx4.0b9]
We should be blacklisting that. Let's reproduce by spoofing our vendor/device ids and version number...
I can't reproduce: when I spoof,

$ MOZ_GFX_SPOOF_VENDOR_ID=8086 MOZ_GFX_SPOOF_DEVICE_ID=0046 MOZ_GFX_SPOOF_DRIVER_VERSION=8.15.10.2141 dist/bin/firefox.exe -P test -no-remote


I do get blacklisted:

Adapter DescriptionNVIDIA Quadro FX 880M
Vendor ID 8086
Device ID 0046
Adapter RAM 1024
Adapter Drivers nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um
Driver Version 8.15.10.2141
Driver Date 6-27-2010
Direct2D Enabled Blocked on your graphics driver. Try updating your graphics driver to version 8.15.10.2202 or newer.
DirectWrite Enabled false (6.1.7600.16385)
WebGL Renderer (WebGL unavailable)
GPU Accelerated Windows0/1
blocking2.0: ? → betaN+
Whiteboard: [#1 topcrash on Fx4.0b8 and Fx4.0b9] → [#1 topcrash on Fx4.0b8 and Fx4.0b9][hardblocker]
Two remarks:

 * the prevalence of this bug seems to be going down quickly: it was #1 in beta9, now crash-stats lists it as #40 in b10pre.

 * we have already had in the past some cases with intel drivers where the DriverVersion reported in the registry wasn't equal to the DLL version (igd10umd32.dll), see for instance bug 595364 comment 9. That was associated with crashes. As far as I could understand, this was a result of a messed up driver installation/upgrade.
So do you know a quick way to check the igd10umd32.dll version, without affecting startup time significantly? If yes, we should just do that, and completely blacklist the driver in case of a mismatch with the DriverVersion given by the registry.
>  * the prevalence of this bug seems to be going down quickly: it was #1 in
> beta9, now crash-stats lists it as #40 in b10pre.
Nightly users are a small stable geek community with usually up-to-date graphic drivers while the Beta tester public is wider and increases once the Fx4 release getting closer. In addition, it is mainly a startup crash, so it means that a lot of new users can't even start Firefox.

>  * we have already had in the past some cases with intel drivers where the
> DriverVersion reported in the registry wasn't equal to the DLL version
> (igd10umd32.dll), see for instance bug 595364 comment 9. That was associated
> with crashes. As far as I could understand, this was a result of a messed up
> driver installation/upgrade.
The only way to confim this theory is to fix bug 624703 then check the differences between driver and dll versions.
In addition, some crashes have an ATI or NVIDIA card as primary card and an Intel card as secondary card. The current blocklisting does not take into account the secondary card while it should.
Depends on: 624703
> Nightly users are a small stable geek community with usually up-to-date graphic
> drivers while the Beta tester public is wider and increases once the Fx4
> release getting closer. In addition, it is mainly a startup crash, so it means
> that a lot of new users can't even start Firefox.

Thats a true. Its both composition and size of the testing pool that plays a role here.   There are currently 1.2 Million active daily users on beta 9, and about 15,000 on b10pre.
(In reply to comment #35)
> >  * we have already had in the past some cases with intel drivers where the
> > DriverVersion reported in the registry wasn't equal to the DLL version
> > (igd10umd32.dll), see for instance bug 595364 comment 9. That was associated
> > with crashes. As far as I could understand, this was a result of a messed up
> > driver installation/upgrade.
> The only way to confim this theory is to fix bug 624703 then check the
> differences between driver and dll versions.

Patch attached to bug 624703.

But rather than waiting for it to be checked in and for data to come, in order to get a chance to have that fixed in beta 10, I'm going to go ahead and write the patch implementing my idea in case it's true.
Here you go.

 * when the registry lies about the DriverVersion, we block all gfx features

 * not using HAVE_64BIT_OS because it's an incredibly misleading name as discussed on bug 598615. But this sizeof(void*)==8 test is exactly what it does.

 * had to move the vendorIntel constant up; moved other constants too to keep them together.
Attachment #505558 - Flags: review?(jmuizelaar)
Comment on attachment 505558 [details] [diff] [review]
block graphics featurs when the windows registry lies about DriverVersion


Maybe only block DX10

> 
>+  mHasDriverVersionMismatch = PR_FALSE;
>+  if (mAdapterVendorID == vendorIntel) {
>+    // we've had big crashers (bugs 590373 and 595364) apparently correlated
>+    // with bad Intel driver installations where the DriverVersion reported by the registry was
>+    // not the version of the DLL.
>+    PRUnichar *dllFileName = sizeof(void*) == 8 ? L"igd10umd64.dll" : L"igd10umd32.dll";

I'd separate this out on to multiple lines

>+    nsString dllVersion;
>+    gfxWindowsPlatform::GetPlatform()->GetDLLVersion(dllFileName, dllVersion);

What happens if this fails

>+    PRUint64 dllNumericVersion = 0;
>+    ParseDriverVersion(dllVersion, &dllNumericVersion);
>+    PRUint64 driverNumericVersion = 0;

put the declarations together and add a blank line inbetween.

>+    ParseDriverVersion(mDriverVersion, &driverNumericVersion);
>+    if (dllNumericVersion != driverNumericVersion)
>+      mHasDriverVersionMismatch = PR_TRUE;
>+  }
>+
Attachment #505558 - Flags: review?(jmuizelaar) → review-
Attachment #505571 - Flags: review?(jmuizelaar) → review?(joe)
Attachment #505571 - Flags: review?(joe) → review+
Landed:

http://hg.mozilla.org/mozilla-central/rev/a7fa7b09e662

leaving open until it's verified. Will also need to check tomorrow with people with intel cards that we didn't break anything (fear of accidentally blacklisting good systems). If there's a problem, we will back out before beta 10.
In reply to comment 41
This patch depends on the theory about mismatch between driver and dll versions, which hasn't been proved yet. So let's wait the landing of bug 624703.
Even if it is true, this patch won't fix the case where the primary GPU is ATI or NVIDIA and the secondary GPU is Intel.
See for instance: bp-16a426c6-6144-4203-8a69-d16a52110120.
For more info, see comment 16 and comment 35.
Indeed. While it's easy to detect the case where there are 2 GPUs, I wonder what the criterion for blocking should be in that case? Hopefully now that bug 624703 is landed we'll get more data.
In today's nightly build we have a spike of crashes with this signature.

However, I am unsure about the revision used for this build. The 'App Notes' don't have the DriverVersion as I added in bug 624703. Even in case of a bug in the code I added, There should still be a comma now after the DeviceID, which there isn't.

What's happening?
In reply to comment 43
> While it's easy to detect the case where there are 2 GPUs, I wonder
> what the criterion for blocking should be in that case?
The switchable GPUs case is a more generic problem than only this startup crash. Because of possible GPU switching at any moment (after Windows sleeping mode for instance), HW acceleration should be available only if both GPU drivers are not blocklisted (HW acceleration enabling/disabling requires a cold start).
But the secondary GPU driver version does usually match the primary GPU driver version. This behaviour can be used. Indeed, after the landing of bug 623338, the driver cut-off date will be almost the same for all GPUs (mid-2010) and will fix this issue.

In reply to comment 44
> In today's nightly build we have a spike of crashes with this signature.
A spike for a startup crash in a nighlty build means nothing.
Depends on: 623338
(In reply to comment #44)
> In today's nightly build we have a spike of crashes with this signature.
> 
> However, I am unsure about the revision used for this build. The 'App Notes'
> don't have the DriverVersion as I added in bug 624703. Even in case of a bug in
> the code I added, There should still be a comma now after the DeviceID, which
> there isn't.
> 
> What's happening?

Holy ... !!

All those crashes I was seeing have this build ID:

   20110120030332

I mistakenly interpreted this as today's nightly, but it is yesterday's!

And we don't have any crash at the moment with today's nightly build ID:

   20110121030329

So it seems that this bug is likely fixed by my patch, which makes me happy :) Aside of course from the case that Scoobidiver mentions with multiple GPUs, but even then, if this particular crash was correlated with mismatched driver versions, it's going to be avoided now. Anyway, we'll see if crashes still happen now :-)
>+    PRUnichar *dllFileName = sizeof(void*) == 8 ? L"igd10umd64.dll" : L"igd10umd32.dll";

I may be completely off here, but will this work if I run a 32bit browser in a 64bit OS?
Still zero crashes in today (January 21)'s nightly build.

Still not conclusive: there also haven't been crashes with the January 19 build.
> Aside of course from the case that Scoobidiver mentions with multiple GPUs, but
> even then, if this particular crash was correlated with mismatched driver
> versions, it's going to be avoided now.
It is not a particular test case according to 4.0b9 crash-stats correlations and the attached file:
    100% (641/642) vs.   2% (914/38770) igd10umd32.dll
     95% (612/642) vs.   6% (2149/38770) atiuxpag.dll
ATI GPU as primary card: 5% of all crashes

Your patch does not work when an ATI GPU is the primary card (if (mAdapterVendorID == vendorIntel)).
For an Intel GPU as a primary card, your patch works so well than it prevents HW acceleration for dual GPUs even with the latest driver version.
(In reply to comment #50)
> > Aside of course from the case that Scoobidiver mentions with multiple GPUs, but
> > even then, if this particular crash was correlated with mismatched driver
> > versions, it's going to be avoided now.
> It is not a particular test case according to 4.0b9 crash-stats correlations
> and the attached file:
>     100% (641/642) vs.   2% (914/38770) igd10umd32.dll
>      95% (612/642) vs.   6% (2149/38770) atiuxpag.dll

Sorry, can you please explain what these numbers are?

Also, why does this bug's title say "mainly dual graphics"? Do we know that a majority of crashes here are on dual-GPU machines? That wasn't my understanding. The vast majority of crash reports I've seen here were showing only Intel graphics involved.
(In reply to comment #51)
> For an Intel GPU as a primary card, your patch works so well than it prevents
> HW acceleration for dual GPUs even with the latest driver version.

Why? With an Intel GPU as primary card, my patch looks at the DLL version of igd10umd32.dll and checks it is consistent with the Intel DriverVersion from the registry. Why would that block if the driver is correctly installed?

It is my understanding here that GetDLLVersion works equally well regardless of whether the DLL is the driver currently in use.
(In reply to comment #53)
> (In reply to comment #51)
> > For an Intel GPU as a primary card, your patch works so well than it prevents
> > HW acceleration for dual GPUs even with the latest driver version.
> 
> Why? With an Intel GPU as primary card, my patch looks at the DLL version of
> igd10umd32.dll and checks it is consistent with the Intel DriverVersion from
> the registry. Why would that block if the driver is correctly installed?
> 
> It is my understanding here that GetDLLVersion works equally well regardless of
> whether the DLL is the driver currently in use.

I'm guessing it might compare the igd10umd32.dll version with the version of the ATI/NVidia driver! That could be problematic for beta 10 for dual GPU users :(
(In reply to comment #54)
> (In reply to comment #53)
> > (In reply to comment #51)
> > > For an Intel GPU as a primary card, your patch works so well than it prevents
> > > HW acceleration for dual GPUs even with the latest driver version.
> > 
> > Why? With an Intel GPU as primary card, my patch looks at the DLL version of
> > igd10umd32.dll and checks it is consistent with the Intel DriverVersion from
> > the registry. Why would that block if the driver is correctly installed?
> > 
> > It is my understanding here that GetDLLVersion works equally well regardless of
> > whether the DLL is the driver currently in use.
> 
> I'm guessing it might compare the igd10umd32.dll version with the version of
> the ATI/NVidia driver! That could be problematic for beta 10 for dual GPU users
> :(

Why would it? It compares the igd10umd32.dll version with the DriverVersion read from the registry. With an Intel GPU as primary card, that would be the intel DriverVersion, no?
What happens when a user switches between two graphics cards while using Firefox? Is the device lost/removed and subsequently recovered/reinitialized? When this happens, is the blacklist consulted?
OK, looks like this patch continues to work, since there are no crashes in buildids since 2011-01-20. Let's call this fixed, and file followup bugs for future work (eg, verifying multi-GPU configurations).
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Assignee: nobody → bjacob
Depends on: 628129
>     100% (641/642) vs.   2% (914/38770) igd10umd32.dll
>      95% (612/642) vs.   6% (2149/38770) atiuxpag.dll

Sorry, can you please explain what these numbers are?


numbers on the right indicate how prevalent the dlls are around in crashes with this signature.  Numbers on the right indicate how prevalent the .dlls are around in crashes for all signatures.

explicitly

igd10umd32.dll is around 100% of the item in this crash signature, whereas its around only 2% in all crashes reports for a given day.

and atiuxpag.dll is around 95% of the item in this crash signature, whereas its only around 6% of the time in all crashes
In other words, 95% of these crashes are (were) on Intel/AMD dual GPU systems? (5% either Intel-only or Intel/NVIDIA dual GPU, presumably)
> In other words, 95% of these crashes are (were) on Intel/AMD dual GPU systems?
Yes. The bug summary has been changed for 2 months to reflect that analysis.
Yeah. I just hadn't really seen AMD/ATI mentioned before. Is Intel/ATI a lot more prevalent than Intel/NVIDIA? I suppose it doesn't really matter though.
I don't see any nv*.dll in correlations for this signature that could reveal the presence of a NVIDIA GPU. But may be it don't need one.

Dbaron's script allows to have correlations by version:
    100% (641/642) vs.   2% (914/38770) igd10umd32.dll
         12% (75/642) vs.   0% (75/38770) 8.15.10.2125
         88% (566/642) vs.   1% (567/38770) 8.15.10.2141
     95% (612/642) vs.   6% (2149/38770) atiuxpag.dll
         78% (502/642) vs.   2% (600/38770) 8.14.1.6117
         12% (77/642) vs.   0% (126/38770) 8.14.1.6126
          1% (7/642) vs.   0% (53/38770) 8.14.1.6136
          3% (17/642) vs.   1% (261/38770) 8.14.1.6160
          1% (8/642) vs.   1% (544/38770) 8.14.1.6170
I don't know how the atiuxpag.dll version is correlated to the AMD driver version that looks like 8.741.0.0.

Let's continue this discussion in bug 628129.
No longer depends on: 623338
The latest Intel drivers for HD Graphics(Clarkdale & Arrandale) and HD Graphics 2000/3000(Sandy Bridge is 8.15.10.2279.

32bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3319&DwnldID=19759

64bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3319&DwnldID=19761
Intel has finally released new drivers, 8.15.10.2281 for Intel G45 chipset(X4500HD/MHD IGP), the last driver released was 8.15.10.2226 previously.

32bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=2991&DwnldID=19788

64bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=2991&DwnldID=19784

For HD Graphics(Clarkdale/Arrandale) and HD Graphics 2000/3000(Sandy Bridge), Intel also has released new drivers, 8.15.10.2291.

32bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3319&DwnldID=19807

64bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3319&DwnldID=19809
FYI, Microsoft has "blacklisted" 8.15.10.2104 through 8.15.10.2141.
http://support.microsoft.com/kb/2498452
Users cannot install Win7 SP1, and "Windows Update will not offer you update KB2454826."
kitchin, see bug 635464.
Intel has released new drivers 8.15.10.2321 for HD Graphics(Clarkdale/Arrandale) and HD Graphics 2000/3000(Sandy Bridge).

32bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3319&DwnldID=19903

64bit
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3319&DwnldID=19906
Crash Signature: [@ _VEC_memzero]
Blocks: 666569
What about DX9 cards like atom intel 3150. The driver file is only igdumdx32
 without igd10umd32. about:support page says that the D2D is blocked because of driver. Intel dev=a011 8.14.10.2364 driver igdumdx32 .
You need to log in before you can comment on or make changes to this bug.