988549 - startup crash in _VEC_memzero | _VEC_memzero

Status: VERIFIED FIXED

(Core :: Graphics, defect)

Description

[u279076]

This bug was filed from the Socorro interface and is 
report bp-c32134e2-9298-4d96-b701-7d6f62140325.
=============================================================
0 	msvcrt.dll 	_VEC_memzero 	
1 	msvcrt.dll 	_VEC_memzero 	

More reports:
https://crash-stats.mozilla.com/report/list?product=Firefox&signature=_VEC_memzero+%7C+_VEC_memzero
=============================================================

Filing this bug as "security restricted" as a precaution since this appears to be an EXCEPTION_ACCESS_VIOLATION_WRITE. The volume is enough to make this the #9 topcrash on Firefox 31 but it seems like most of these are dupes.

Top 5 URLs:
25 	https://www.google.co.in/
23 	about:sessionrestore
12 	http://www.awesomehp.com/...
11 	about:blank
3 	about:home

Comment 1

[u279076]

Robert, do you think this is something we need to worry about given the dupes?

Comment 2

[Robert Kaiser]

It has been around for a while. If it's not security-sensitive and not reproducible, I think it might not be worrisome.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

Dan can you take a look here and comment on whether this is security-sensitive?

Comment 4

[Daniel Veditz [:dveditz]]

I don't think we need to hide this bug, and I'm not worried about security attacks on startup. But 30% or so of these crashes are not at startup.

Comment 5

[Benjamin Kerensa [:bkerensa]]

Tracking this since its still a top crasher as of today on Win

Comment 6

[u279076]

I think this is gone now for whatever reason. I'm seeing only two crashes with Firefox 31, both crashes with builds that are about a month old (possibly users who can't update because they are hitting this crash). We can likely just mark this bug resolved WORKSFORME. Can someone please double check?

Source:
https://crash-stats.mozilla.com/report/list?signature=_VEC_memzero+|+_VEC_memzero&product=Firefox&query_type=contains&range_unit=weeks&process_type=any&version=Firefox%3A31.0a1&hang_type=any&date=2014-04-29+16%3A00%3A00&range_value=1#tab-reports

Comment 7

[u279076]

...and it's back again:
https://crash-analysis.mozilla.com/rkaiser/2014-05-05/2014-05-05.firefox.31.explosiveness.html

Any advice on what we can do to investigate this further?

Comment 8

[(Away)]

Here's a better stack reconstructed by hand. All of the crashes have Intel graphics cards. Gfx team should have a look.

msvcrt!_VEC_memzero+0x36
d2d1!TextStageManager::MapTextureTransferSurface+0x101
d2d1!TextStageManager::AddStagesForSubrect+0x25
d2d1!HwGlyphRunRealizer::IssueRenderingCommands+0x5f7
d2d1!CHwSurfaceRenderTarget::DrawGlyphRunInternal+0x181
d2d1!CCommand_DrawGlyphRun::Execute+0x27
d2d1!CHwSurfaceRenderTarget::ProcessBatch+0x4c
d2d1!CBatchSerializer::FlushInternal+0x2e
d2d1!DrawingContext::FlushBatch+0x1a
d2d1!DrawingContext::Flush+0x26
d2d1!D2DRenderTargetBase<ID2D1RenderTarget>::Flush+0x52
xul!mozilla::gfx::DrawTargetD2D::Flush+0x21 
xul!gfxContext::~gfxContext+0xfb 
[Frame pointers stop making sense beyond that]

Comment 9

[Benjamin Smedberg]

ni?naveed for an owner

Comment 10

[Chris Peterson [:cpeterson]]

needinfo'ing Milan because this is a Gfx bug, not a JS bug.

Comment 11

[Milan Sreckovic [:milan] (needinfo for best results)]

Bas, any ideas? I'm making this up, but if that is a gfxContext getting destroyed, wouldn't that only happen when nsRenderingContext is going away, and would that mean we've already decided to shutdown, and this is a startup crash?

Comment 12

[Bas Schouten (:bas.schouten)]

(In reply to Milan Sreckovic [:milan] from comment #11)
> Bas, any ideas? I'm making this up, but if that is a gfxContext getting
> destroyed, wouldn't that only happen when nsRenderingContext is going away,
> and would that mean we've already decided to shutdown, and this is a startup
> crash?

No, we recreate gfxContext's all the time (Once per frame per layer at least). They're just a 'state machine' wrapping a DrawTarget or a cairo surface.

Comment 13

[Sylvestre Ledru [:Sylvestre]]

Milan, could you have someone working on it? It is still occurring. Thanks

Comment 14

[Milan Sreckovic [:milan] (needinfo for best results)]

I don't know how far we can get with just the info in this bug, but we can probably take a look in 33 timeframe.

Comment 15

[Sylvestre Ledru [:Sylvestre]]

Bas, any chance this bug could be fixed for 31? 31 is going to move to beta next Monday.
Thanks

Comment 16

[Bas Schouten (:bas.schouten)]

Not really, since we have no steps to reproduce. As Milan said, there's not a lot of information to go with here. We could try blanket blacklisting things based on the drivers/devices in crash reports but that doesn't really seem like a healthy way to go about it :).

Comment 17

[Robert Kaiser]

Well, we need to do something here. This is firmly the #1 crash in 31 Beta now, with 5% of the total crash volume. We can't ignore it.

Comment 18

[Robert Kaiser]

https://crash-stats.mozilla.com/report/list?signature=_VEC_memzero+%7C+TextStageManager%3A%3AMapTextureTransferSurface%28D2D_RECT_U+const%26%2C+unsigned+char%2A%2A%2C+unsigned+int%2A%29 is probably connected as well and is #4 with 4% on 31.0b right now, so together those are at >9% of all Beta 31 crashes at this point.

Comment 19

[Benjamin Smedberg]

https://crash-analysis.mozilla.com/bsmedberg/bug988549-VEC-memzero-nightly.csv shows there may be a regression between:

20140318030202,68727,
20140319030201,76486,26

Although a single user crashing can affect the nightly stats disproportionately. Liz can you make a nightly regression link for that?

Comment 20

[Liz Henry (:lizzard) (relman/hg->git project)]

This signature is only showing up on x86.  They are nearly all still happening on startup for Windows 7.
 
Here is the graphics adapter info from the crash signature summary:

Vendor 	Adapter 	Report Count 	Percentage
0x8086 	0x0042	2236 	79.601 %
Intel Corporation 	Mobile Intel 4 Series Chipset Family Intel Mobile Graphic	235 	8.366 %
Intel Corporation 	PCI\VEN_8086&DEV_2E32&SUBSYS_31031565&REV_03 Intel G41 express graphics	228 	8.117 %
0x8086 	0x2e22	92 	3.275 %
Intel Corporation 	- Intel Q45/Q43 Express Chipset	13 	0.463 %
0x8086 	0x0046	4 	0.142 %
0x8086 	0x0102	1 	0.036 %

Comment 21

[Liz Henry (:lizzard) (relman/hg->git project)]

Benjamin, near as I can tell, http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=082761b7bc54&tochange=3bc3b9e2cd99 .

What do the second and third numbers in the rows in your csv mean?

Comment 22

[u279076]

(In reply to Liz Henry :lizzard from comment #20)
> This signature is only showing up on x86.  They are nearly all still
> happening on startup for Windows 7.
>  
> Here is the graphics adapter info from the crash signature summary:
> 
> Vendor 	Adapter 	Report Count 	Percentage
> 0x8086 	0x0042	2236 	79.601 %
> Intel Corporation 	Mobile Intel 4 Series Chipset Family Intel Mobile Graphic
> 235 	8.366 %
> Intel Corporation 	PCI\VEN_8086&DEV_2E32&SUBSYS_31031565&REV_03 Intel G41
> express graphics	228 	8.117 %
> 0x8086 	0x2e22	92 	3.275 %
> Intel Corporation 	- Intel Q45/Q43 Express Chipset	13 	0.463 %
> 0x8086 	0x0046	4 	0.142 %
> 0x8086 	0x0102	1 	0.036 %

These are all Intel cards. The top correlation (0x8086 0x0042) maps to Intel Q57/H55 Clarkdale according to http://www.pcidatabase.com/vendor_details.php?id=1302

Comment 23

[(Away)]

These crashes are all on Windows 7. If a single OS and gfx series produce the overall #1 top crash, then on those particular machines it must be especially crashy. Can we just find such a machine and play around until it crashes?

Comment 24

[u279076]

Juan, can you check the lab to see if we have a machine available which matches this criteria?

Comment 25

[Benjamin Smedberg]

The columns are buildid,cumulative-ID, cumulative-crashes

Comment 26

[Laura Thomson :laura]

Top graphics adapter is:
0x8086 	0x0042	2946 	83.480 % (of crashes)

No useful information there but the associated driver version in the App Notes is 8.15.10.2418 which is long enough to be useful. Googling it brings up a bunch of fairly elderly Acer Travelmates, Gateway laptops, etc. Does that help?

Comment 27

[Milan Sreckovic [:milan] (needinfo for best results)]

Thanks Laura, it doesn't hurt to have that info :)

Bas, while we're looking for the machine, is there anything you can suggest we do to get to the STR?  Is this configuration similar to what Matt had with all the problems we've seen last week?  Also, wasn't there a "it crashes the first time after X happens, but then it's OK after that" (I can't recall the details) - could this be it?

Comment 28

[juan becerra [:juanb]]

Marcia and I checked the machines in the QA lab including a few Lenovo Thinkpads, but we haven't been able to find one that matches the specs in hardware mentioned here.

I opened a ticket in the ServiceNow site to see if we have any of the machines in this site: http://support.lenovo.com/en_US/detail.page?LegacyDocID=MIGR-70128 (which bmoss referred me to)

Comment 29

[Bas Schouten (:bas.schouten)]

Not really :s The codepath in the stack is -really- common (i.e. to the extent I can see it, it would be called every frame for any window with text). The pages it occurs on also show little interesting except that it happens on common startup pages, which is no surprise. There really isn't any information here which is specific to any sort of action or behavior.

The one thing that's interesting here is that it's all Intel, but we didn't see this on the machine Matt had.

Comment 30

[Milan Sreckovic [:milan] (needinfo for best results)]

Thanks, I appreciate the details.  OK, lets see if we can find one of these and reproduce ourselves (see comment 28)

Comment 31

[Robert Kaiser]

What I find interesting as well is that many of the crashes with the signature here have addresses ending in fbd0, with the https://crash-stats.mozilla.com/report/list?signature=_VEC_memzero+%7C+TextStageManager%3A%3AMapTextureTransferSurface%28D2D_RECT_U+const%26%2C+unsigned+char%2A%2A%2C+unsigned+int%2A%29 signature, the addresses vary more, but a number have similar end bytes like fb29, fb89  or fdbe.

Comment 32

[Sylvestre Ledru [:Sylvestre]]

Milan, Any progress on this? It is 4-5% of the overall crashes. thanks

Comment 33

[Milan Sreckovic [:milan] (needinfo for best results)]

See comment 28.  Need the hardware, the stack is too general to give us any useful information.

Comment 34

[(Away)]

Yikes, these reports are with old drivers that have other issues too.

8.15.10.2141 	81.07 %
8.15.10.2125 	18.91 %

The newest of those is from June 2010. According to [1] those versions have known D2D crashes, and MS won't offer Windows 7 SP1 to machines with those drivers. That's confirmed on crash-stats: 96% are on plain Windows 7 (not SP1).

[1] http://support.microsoft.com/kb/2498452

Comment 35

[Robert Kaiser]

The reports all say D2D+ from what I can see, should we be blocking those old drivers from D2D? Do we think that would help those crashes?

Comment 36

[Bas Schouten (:bas.schouten)]

(In reply to Robert Kaiser (:[e-mail]) from comment #35)
> The reports all say D2D+ from what I can see, should we be blocking those
> old drivers from D2D? Do we think that would help those crashes?

I suspect it will, I thought we were blocking these from D2D anyway, but it looks like we're not.

Comment 37

[Robert Kaiser]

(In reply to Bas Schouten (:bas.schouten) from comment #36)
> (In reply to Robert Kaiser (:[e-mail]) from comment #35)
> > The reports all say D2D+ from what I can see, should we be blocking those
> > old drivers from D2D? Do we think that would help those crashes?
> 
> I suspect it will, I thought we were blocking these from D2D anyway, but it
> looks like we're not.

Maybe some change in 31 unblocked them for some reason? That would explain why this regressed between 30 and 31.

Comment 38

[Bas Schouten (:bas.schouten)]

Jeff, you played around with blacklisting I believe? Any chance this happened.

Comment 39

[Milan Sreckovic [:milan] (needinfo for best results)]

Bug 904266 landed in 26, I don't recall us doing much since.

Comment 40

[Robert Kaiser]

Can we please check that D2D blocklisting works correctly in Firefox 31?

Comment 41

[u279076]

(In reply to Robert Kaiser (:[e-mail]) from comment #40)
> Can we please check that D2D blocklisting works correctly in Firefox 31?

Florin, can you please set up some regression testing for 31.0b7?

Comment 42

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

We have done some testing with Firefox 31 Beta 7 (BuildID: 20140703154127) on Direct2D blocklisting, by spoofing graphics system information (see bug 604771), and verifying the "about:support" page. We compared the results to those found in "browser/blocklist.xml" and those from "https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers#Intel_cards" (for Win 7, Vista and 8).

Issues encountered were:

1. Different driver versions than expected (https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers#Intel_cards) were seen for some of the Intel cards. See in https://docs.google.com/spreadsheets/d/12yEBqNZqaR6_2XbqFrVwRVxt_URjNN3iMmu-GX60Z0Q/edit#gid=0.	
				
2. Setting for Direct2D does not always change if the same user profile is used (I could not quite figure out when it works and when not). I think this means that a user may still have Direct2D as enabled, in the case where the driver is NOt initially blacklisted, but later becomes blacklisted. See in https://docs.google.com/spreadsheets/d/12yEBqNZqaR6_2XbqFrVwRVxt_URjNN3iMmu-GX60Z0Q/edit#gid=0.
					
3. "browser/blocklist.xml" - at blockID="g278", the value 0x9803 appears twice.	

Let me know if additional investigation is needed.

Comment 43

[Robert Kaiser]

Bas, does the testing in comment #42 help you? If blocking doesn't work correctly, we're going to see a ton of crashing once we hit release (for now, we mostly saw this in beta).

Comment 44

[Bas Schouten (:bas.schouten)]

(In reply to Robert Kaiser (:[e-mail]) from comment #43)
> Bas, does the testing in comment #42 help you? If blocking doesn't work
> correctly, we're going to see a ton of crashing once we hit release (for
> now, we mostly saw this in beta).

I don't see any information in there that helps. It's not clear to me from the comment whether blacklisting is working correctly or not. I'm most interested in what happens on a clean profile/clean machine. Preferably not trying to spoof graphics information but actually installing and testing the relevant drivers.

Comment 45

[Robert Kaiser]

Florin, any chance you can help with the request in comment #44?

Comment 46

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

(In reply to Robert Kaiser (:[e-mail]) from comment #45)
> Florin, any chance you can help with the request in comment #44?

I've done some additional investigation and there are two big issues with blacklisting in Firefox 31, both regressions from Firefox 30. Testing was still done via spoofing, as there's no way we can do it with actual hardware, since time is extremely limited for us.

1. Different (smaller) values for minimum accepted driver versions are used in Firefox 31 compared to Firefox 30 (see lines 20, 23, 25, 29, 30, and 32 in https://docs.google.com/spreadsheets/d/12yEBqNZqaR6_2XbqFrVwRVxt_URjNN3iMmu-GX60Z0Q/edit#gid=0)

a) Intel 
- Win Vista - GMA 500 - Expected=Firefox30=7.14.10.1006 vs. Firefox31=3.0.20.32
- Win Vista - GMA 3150 - Expected=Firefox30=7.14.10.2124 vs. Firefox31=7.14.10.1910
- Win Vista - GMA GMA X4500/HD - Expected=Firefox30=8.15.10.2202 vs. Firefox31=7.15.10.1666
- Win 7 - GMA 3150 - Expected=Firefox30=8.14.10.2117 vs. Firefox31=8.14.10.1972
- Win 7 - GMA X3000 - Expected=Firefox30=8.15.10.1930 vs. Firefox31=7.15.10.1666
- Win 7 - GMA X3000 - Expected=Firefox30=8.15.10.2202 vs. Firefox31=7.15.10.1666

------> Impact: a user with Intel GMA 500 on Win Vista, has driver 4.0.0.0 which is blocked in Firefox 30, but NOT in Firefox 31.

b) NVidia, AMD
- the behavior here is rather weird in Firefox 31, as it seems that two different values are used for blacklisting: one from "browser/blocklist.xml" and one probably from the same place where the values for Intel GPUs come from. See a specific case below, compared for Firefox 31 and Firefox 30:

Environment: Windows 7 (0x60001) + AMD RadeonTM HD 8570

STR:
i. Create "spoofed-firefox.bat" file with following content:
SET MOZ_GFX_SPOOF_WINDOWS_VERSION=60001
SET MOZ_GFX_SPOOF_VENDOR_ID=0x1002
SET MOZ_GFX_SPOOF_DEVICE_ID=0x5049
SET MOZ_GFX_SPOOF_DRIVER_VERSION=
"C:\Mozilla\Firefox\firefox.exe" -p -no-remote
ii. Set the driver version value, save the file, open it to launch Firefox, create a new profile, start Firefox, and verify "about: support" -> Graphics -> Direct2D Enabled.
iii. Execute step "ii" for Firefox 31 with the following values for the Driver version: 8.6.9.9, 8.7.0.0, 8.982.0.0, 8.982.0.1.
iv. Execute step "ii" for Firefox 30 with the following values for the Driver version: 8.7.9.9, 8.8.0.0, 8.982.0.0, 8.982.0.1.

Results:
At step iii. - Direct2D = "Blocked for your graphics driver version. Try updating your graphics driver to version 9.6 or newer." up to version ~8.6.9.9 -> "true" for version 8.7.0.0 and up (except for 8.982.0.0) -> "Blocked for your graphics driver version." for version=8.982.0.0.
At step iv. - Direct2D = "Blocked for your graphics driver version. Try updating your graphics driver to version 10.6 or newer." up to version ~8.7.9.9 -> "true" for version 8.8.0.0 and up (except for 8.982.0.0) -> "Blocked for your graphics driver version." for version=8.982.0.0

Notice above how the Firefox 31 error message says that the minimum driver version is 9.6, while Firefox 30 says the minimum version is 10.6.

------> Impact: a user with the above setup, has driver 8.7.0.0 which is blocked in Firefox 30, but NOT in Firefox 31.

2. For some driver version values, Firefox 31 enables/disables Direct2D when the new profile is created, but then no longer updates it when the driver version changes (see lines 6 to 12 in https://docs.google.com/spreadsheets/d/12yEBqNZqaR6_2XbqFrVwRVxt_URjNN3iMmu-GX60Z0Q/edit#gid=0). 

------> Impact: User with Firefox 31 on Windows 7 + AMD RadeonTM HD 8570, has driver version 8.900.0.0 (for which Direct2D is "true" on both Firefox 30 and 31), user then updates to version 8.982.0.0 (which should be blocked for both Firefox 30 and 31) => Unless the user creates a new profile, Direct2D will NOT be blocked as expected.

Comment 47

[Robert Kaiser]

Bas, it looks from comment #46 like we have heavily broken D2D blocking on 31, and I believe this to be one of the prime issues with our instabilities on that train. Are you the one to look into that or someone else? Note that Beta 9 is going to build on Thursday and should be the last build to land any code changes in this train.

Comment 48

[Bas Schouten (:bas.schouten)]

(In reply to Robert Kaiser (:[e-mail]) from comment #47)
> Bas, it looks from comment #46 like we have heavily broken D2D blocking on
> 31, and I believe this to be one of the prime issues with our instabilities
> on that train. Are you the one to look into that or someone else? Note that
> Beta 9 is going to build on Thursday and should be the last build to land
> any code changes in this train.

I don't think I am, I will discuss this in the graphics daily. We'll make sure this gets addressed.

Comment 49

[Jeff Muizelaar [:jrmuizel]]

So just to clarify, is the problem here that were not blacklisting hardware that we used to or that we're trying to blacklist hardware but it's not working?

Comment 50

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #49)
> So just to clarify, is the problem here that were not blacklisting hardware
> that we used to or that we're trying to blacklist hardware but it's not
> working?

Jeff, in Firefox 31 we're not blacklisting hardware that we used to in Firefox 30 (minimum driver versions in 31 are not the expected ones from Firefox 30, but they are smaller versions). However note that there seem to be more problems than just this (see comment 46).

Comment 51

[Jeff Muizelaar [:jrmuizel]]

Attachment: Backout bug 984417

This is a patch to backout bug 984417. We should probably take this now so that we have more time to investigate the problem and apply a better fix afterwards.

Comment 52

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 8453524 [details] [diff] [review]
Backout bug 984417

Approval Request Comment
[Feature/regressing bug #]: 984417
[User impact if declined]: Lots more crashes
[Describe test coverage new/current, TBPL]: Was in the tree until 31
[Risks and why]: Should be relatively low risk because it just backs out a blocklist change
[String/UUID change made/needed]: None

Comment 53

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/49f4f9af2d64

Comment 54

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

I've verified today the behavior after the backout for bug 984417, with Firefox 31 Beta 9 (BuildID: 20140710141843) on Windows 7 x64 (Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0).

Blacklisting for Intel cards now works same as for Firefox 30, and as expected.

There are still 3 blacklisting issues (regressions from Firefox 30) that may still cause issues in the future, but should NOT affect too much the stability of Firefox 31 on the short term.

Remaining issues:

1. There are a few cases (non-Intel cards) where D2D is enabled for Firefox 30, but disabled for Firefox 31 (https://docs.google.com/spreadsheets/d/12yEBqNZqaR6_2XbqFrVwRVxt_URjNN3iMmu-GX60Z0Q/edit#gid=0 - lines 6 - 12 in yellow)
- drivers ADDITIONALLY blocked on Firefox 31 compared to Firefox 31:
* Win 7 - NVidia NVS 5100M - 0x10de - 0x0a6c - drivers 8.17.12.5720 to 8.17.12.5896
* Win Vista - AMD RadeonTM HD 8570? - 0x1002 - / - driver 8.982.0.0
* Win 7 - AMD RadeonTM HD 8570?	- 0x1002 - driver 8.982.0.0
* Win Vista - AMD? - 0x1022 - / - driver 8.982.0.0
* Win 7 - AMD? - 0x1022 - / - driver 8.982.0.0
* Win 8 - AMD RadeonTM HD 8570? - 0x1002 - / - drivers 8.8.0.0 to 9.10.8.0
* Win 8 - AMD?	- 0x1022 - / - drivers 8.8.0.0 to 9.10.8.0

Notes:
- the differences above come from the "browser\blocklist.xml" file. The file exists for Firefox 30 and other older versions, and has the same values, but it seems they were never in fact used in the previous versions, and Firefox 31 is the first version where they are used for blocking Direct2D. Note that these values are used in conjunction with the older ones from Firefox 30, without overriding them.

------> USER IMPACT: if the versions above are intended to be blocked, then there's no negative impact on the user. In any case, since we actually block more versions in 31 compared to 30, this should improve the stability.


2. Direct2D does NOT change state if a user (with Firefox 31) updates from one of the driver versions from the intervals shown above (issue #1) to any other version.

------> USER IMPACT: user with Firefox 31 on Windows 7 + AMD RadeonTM HD 8570, has driver version 8.900.0.0 (for which Direct2D is "true" on Firefox 31), user then updates to version 8.982.0.0 (which should be blocked for Firefox 31) => Unless the user changes/creates a new profile, Direct2D will NOT be blocked as expected, so it may cause issues.


3. Direct2D does NOT change state if a user with Firefox 30 updates to Firefox 31, and he has a driver version allowed on Firefox 30, but blocked in Firefox 31

------> USER IMPACT: user with Firefox 30 on Windows 7 + NVidia NVS 5100M, has driver version 8.17.12.5800 (for which Direct2D is "true" on Firefox 30), user then updates to Firefox 31 (for which the driver version should be blocked) => Unless the user changes/creates a new profile, Direct2D will NOT be blocked as expected, so it may cause issues.

Overall I think the backout has improved blacklisting, and brought it closer to Firefox 30 and to what's intended. Out of the remaining issues, #3 worries me the most as it may still cause problems after users update to Firefox 31 (but should not make things worse I think). Even so, if blacklisting was the main problem for the crashes, then these should go down.

Comment 55

[u279076]

I'm not going to block Firefox 31.0b9 on these issues since we want to get Beta feedback on the landed changes before we build RC next week. 

Bas and Jeff, what do you think about the issues described in comment 54? Should they be addressed in this bug or do they deserve follow-up reports? Can they be addressed in Firefox 31?

Comment 56

[Bas Schouten (:bas.schouten)]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #55)
> I'm not going to block Firefox 31.0b9 on these issues since we want to get
> Beta feedback on the landed changes before we build RC next week. 
> 
> Bas and Jeff, what do you think about the issues described in comment 54?
> Should they be addressed in this bug or do they deserve follow-up reports?
> Can they be addressed in Firefox 31?

Not really, but 3 also doesn't really concern me too much, we haven't blocked anything important in 31 that wasn't blocked on 30.

However this might be an issue for 32, so we should still keep an eye on it.

Comment 57

[u279076]

Thanks Bas, do you want bugs filed for those issues?

Comment 58

[Robert Kaiser]

On the actual crash, the volume is significantly down in 31.0b9 (and there was a residual crash with this signature before with the blocklisting working correctly) so I'm pretty confident we can mark this as fixed.

The issues that Florin turned up probably warrant a separate bug being filed for tracking, I guess.

Comment 59

[Benoit Jacob [:bjacob] (mostly away)]

I may have missed something because I haven't read throughly all of the above comments, but was any analysis of the per-GPU-vendor impact of this crash done? I just did some grepping of crashdata files, and it seems that while we always had some _VEC_memzero crashes on all GPU vendors, it spiked specifically on Intel with the patch from bug 984417 while remaining unaffected by this patch on other GPU vendors. Consequently, it seems that we only need to revert the Intel-specific portion of that patch.

bjacob@bjacob-dell:~/hack/crash-stats$ zcat 20140301-pub-crashdata.csv.gz | grep _VEC_memzero | grep AdapterVendorID | sed 's/.*AdapterVendorID\:\ \([0-9a-fA-FxX]*\).*/\1/g' | sort | uniq -c | sort -rn
    127 0x8086
     34 0x1002
     31 0x10de
     12 8086
     11 0x1106
      7 0x0000
      2 1106
      2 1002
      1 10de
      1 0x5333
      1 0x1039
      1 0x102b
      1 00ba
      1 0000
bjacob@bjacob-dell:~/hack/crash-stats$ zcat 20140401-pub-crashdata.csv.gz | grep _VEC_memzero | grep AdapterVendorID | sed 's/.*AdapterVendorID\:\ \([0-9a-fA-FxX]*\).*/\1/g' | sort | uniq -c | sort -rn
    159 0x8086
     32 0x1002
     29 0x10de
     22 8086
      5 0x1039
      3 10de
      2 1002
      2 0x1106
      2 0x0000
      1 0000
bjacob@bjacob-dell:~/hack/crash-stats$ zcat 20140501-pub-crashdata.csv.gz | grep _VEC_memzero | grep AdapterVendorID | sed 's/.*AdapterVendorID\:\ \([0-9a-fA-FxX]*\).*/\1/g' | sort | uniq -c | sort -rn
    139 0x8086
     27 0x10de
     20 0x1002
      9 8086
      9 0x0000
      5 0x1039
      3 0x1106
      1 10de
      1 1002
      1 0x5333
bjacob@bjacob-dell:~/hack/crash-stats$ zcat 20140601-pub-crashdata.csv.gz | grep _VEC_memzero | grep AdapterVendorID | sed 's/.*AdapterVendorID\:\ \([0-9a-fA-FxX]*\).*/\1/g' | sort | uniq -c | sort -rn
    163 0x8086
     42 0x10de
     25 0x1002
      8 8086
      7 0x1106
      4 1002
      3 0x1039
      3 0x0000
      2 0x5333
      2 0000
      1 10de
bjacob@bjacob-dell:~/hack/crash-stats$ zcat 20140701-pub-crashdata.csv.gz | grep _VEC_memzero | grep AdapterVendorID | sed 's/.*AdapterVendorID\:\ \([0-9a-fA-FxX]*\).*/\1/g' | sort | uniq -c | sort -rn
   1345 0x8086
     43 0x10de
     32 0x1002
      8 8086
      5 0x1039
      4 0x0000
      3 0x1106
      1 1002
      1 0000

Comment 60

[Benoit Jacob [:bjacob] (mostly away)]

Attachment: backout-intel

Comment 61

[Benoit Jacob [:bjacob] (mostly away)]

Comment on attachment 8455718 [details] [diff] [review]
backout-intel

Approval Request Comment - SEE ABOVE PATCH's APPROVAL - THIS IS JUST A SUBSET OF IT. Also, I know that normally you only approve already-landed patches, but here it saves us time to do this all at once, and again, a bigger version of this patch was already approved above.

[Feature/regressing bug #]: bug 984417
[User impact if declined]: top crasher on Intel GPUs only, on Windows
[Describe test coverage new/current, TBPL]: None since we dont have windows test slaves with Intel graphics
[Risks and why]: Not risky at all
[String/UUID change made/needed]: none

Comment 62

[(Away)]

(In reply to Benoit Jacob [:bjacob] from comment #59)

That's a good point... perhaps we can narrow it down even further? For example over 95% of these crashes are on Windows 7. Maybe it's particular to only some of those devFamily/driverVer?

Comment 63

[Benoit Jacob [:bjacob] (mostly away)]

Perhaps - but since at the moment I'm in "lets rush to get this fixed or else it will all get backed out" mode, I would rather land this conservative patch for now.

Comment 64

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Firefox 31 RC has been built today, without the change from Benoit. Do we plan on still having this in Firefox 31?

Comment 65

[Sylvestre Ledru [:Sylvestre]]

Benoit's change arrived too late for the RC. If we make another build, we could take it.

Comment 66

[Robert Kaiser]

I think we have pretty clearly established that this specific crash signature is old Intel drivers on Win7, we knew that for a long time. It was pretty much a surprise to us that gfx blocklisting changes were the cause. We should not run into those confusions and run experiment on our beta audience, so I do not think we should take a partial backout on 32 but a full one like on 31. See bug 984417 on a more detailed comment on how I see those things.

Comment 67

[Benoit Jacob [:bjacob] (mostly away)]

So would you be happy with:
 - full backout on 32  (we already did a full backout on 31, right?)
 - intel-only partial backout patch on 33

Comment 68

[Benoit Jacob [:bjacob] (mostly away)]

Landed on inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6876da7d8806

Comment 69

[Benoit Jacob [:bjacob] (mostly away)]

Comment on attachment 8455718 [details] [diff] [review]
backout-intel

Just talked with jrmuizel. We actually think that for mozilla-aurora (soon to be early beta) we want the Intel-only backout i.e. this patch. Indeed, we have strong confidence that this will fix at least the bulk of the crashiness here, enough to make this acceptable for early beta. Therefore, keeping the mozilla-aurora request here.

For beta on the other hand, the full backout had already landed earlier during the 31 cycle there, so there is no need for anything more.

Comment 70

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8455718 [details] [diff] [review]
backout-intel

Same as 31. Happy to take it.

Comment 71

[Benoit Jacob [:bjacob] (mostly away)]

Pushed the intel-only backout to Aurora:

https://hg.mozilla.org/releases/mozilla-aurora/rev/8d5e337d433b

At this point:

Gecko 32+ (inbound and aurora) have the Intel-only backout.
Gecko 31 (beta) already had the earlier full backout.

So at this point I'm considering this fixed, though of course I would be very interested in hearing about the crash-stats impact!

Comment 72

[Robert Kaiser]

As I mentioned to bjacob on IRC, my main goal is to ensure we do not end up again with the stability group not knowing something has been unblacklisted in some version and be surprised about eventual crashes. Can you please make sure the stability@m.o list gets a message whenever anything is unblacklisted and tell us there exactly which cards/chips/driver-versions have been unblacklisted in which Firefox version?
With that, we will know what to look for. We should probably be good with this in 32 (but please send that message) as with bug 918386 we should be able to search for driver versions in crash-stats when needed to dig into something like that (still need to expose it in crash-stats, I'll file bugs for that).

Comment 73

[Benoit Jacob [:bjacob] (mostly away)]

Mail sent to Stability.

Comment 74

[Robert Kaiser]

FYI, the remaining cases of bug 768395 (CDevice::DriverInternalErrorCB) is another signature that went down with this backout on 31 (also almost exclusively Intel).

Comment 75

[u279076]

Stats from the last week:
Firefox 33 has 19 crashes
Firefox 32 has 75 crashes
Firefox 31 has 72 crashes (beta9 or higher)

Crashes still obviously exist but this is far outside the topcrash range now. Can this bug report be closed or is there still work to be done?

Comment 76

[Robert Kaiser]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #75)
> Can this bug report be closed or is there still work to be done?

Given the discussion here, I would close it and file new followups for new/other issues.

Comment 77

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature '_VEC_memzero | _VEC_memzero':
 - nightly(version 50):0 crashes from 2016-06-06.
 - aurora (version 49):1 crash from 2016-06-07.
 - beta   (version 48):34 crashes from 2016-06-06.
 - release(version 47):189 crashes from 2016-05-31.
 - esr    (version 45):64 crashes from 2016-04-07.

Crash volume on the last weeks:
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly       0       0       0       0       0       0       0
 - aurora        0       1       0       0       0       0       0
 - beta          7       3       3       4       3       7       5
 - release      25      29      25      20      26      29      27
 - esr           4      10      11       7       7       9       4

Affected platform: Windows