Closed Bug 789251 Opened 13 years ago Closed 7 years ago

HWA Crash, 0-byte minidump with Catalyst 12.8 (8.982) (Empty Crash)

Categories

(Toolkit :: Crash Reporting, defect)

Product:
Toolkit
Component:
Crash Reporting
Version:
15 Branch
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED WONTFIX
Tracking Flags:
Tracking Status
firefox16 - ---
firefox17 - ---

People

(Reporter: tdowner, Assigned: bjacob)

References

Details

(Keywords: crash)

Crash Data

Attachments

(1 file)

qa lab machine information
2.53 KB, text/plain
Details
This is a substantial issue on SUMO the past few weeks. AMD released Catalyst 12.8 8/15. Since then there have been a significant increase in users reporting crashes with an Empty Crash thread. Disabling Hardware Acceleration for these users corrects the problem. bp-67fd3c88-98c1-4b0c-ae21-4e37d2120901 is a standard thread from one of these users. https://support.mozilla.org/en-US/questions/936087 Crashes seem to be quite random. There are also issues with Crossfire using alot of Memory with 12.8, https://support.mozilla.org/en-US/questions/934825, not sure if this is related (Perhaps an OOM crash?)
Well over 50 users complaining of this (crashing issue, more about the memory usage) on the forums this week alone.
(In reply to Tyler Downer [:Tyler] from comment #0) > https://support.mozilla.org/en-US/questions/936087 FWIW, the User's Crash in that Thread (bp-423a8aa4-aef4-418b-ac4a-56b072120902) is already filed as Bug 768395. I think that the "Empty Stack" Crashes are a) OOM's and b) useless as reported per Breakpad. Maybe to confirm a) some User can perform the "How to get a stacktrace with WinDbg" Procedure"?
If you read the bug, there are two crash threads, one that is Empty, and the second one. The empty crashes are the big issue. https://support.mozilla.org/en-US/questions/936286
It might be a dupe of bug 788559.
Keywords: crash
Hardware: x86_64 → x86
(In reply to Scoobidiver from comment #4) > It might be a dupe of bug 788559. Well, the ones there actually do have crash dumps, the reports in here comes with empty crash dumps. Still, the buggy driver causing it could be the same.
This appears to be a 0-byte minidump with 500MB of available VM and 3.4G of available physical memory. It's vaguely possible that we're actually out of VM space enough that we can't write a minidump, although that's still pretty surprising. I wonder if we have somebody in the QA lab who can reproduce the problem with an updated analyst. Tyler, do any of these users have steps that produce this problem reliably or more often? I'd like this bug to track specifically the 0-byte minidump issue; there may already be bugs on the graphics issue, but we can't really tell until we have some data.
Component: Graphics → Breakpad Integration
Product: Core → Toolkit
Summary: HWA Crash with Catalyst 12.8 (8.982) → HWA Crash, 0-byte minidump with Catalyst 12.8 (8.982)
It's plausible that there's heap corruption bad enough that causes dump generation to fail, since Microsoft's MinidumpWriteDump code does a bunch of allocation. If someone can reproduce this crash, we could pretty easily find out with a debug build.
Benjamin, No, that is the difficult thing that SUMO users are reporting, there aren't any clear STR. This crash happens randomly. I can reach out to specific users to see if they would be willing to run some stuff (windbg, etc) for us if QA can't reproduce this.
Let's not reach out to users directly yet.
QA Contact: mozillamarcia.knous
I was able to install that 12.8 version on one of the machines I have that is AMD, but in the about:support section it says the drivers are being blocked. I assume I can modify the blocklist file to override that setting?
(In reply to Marcia Knous [:marcia] from comment #10) > I assume I can modify the blocklist file to override that setting? Set gfx.direct2d.force-enabled to true.
Using Firefox 15.0.1, I force enabled the pref and I am not seeing any crashes on the one machine I have that I can upgrade to: Pavilion dm1. I confirmed hardware acceleration is on.
Blocks: 789260
Bug 789260 has a stack trace identical to the one of bug 788559.
Benjamin - we haven't had any luck with reproducing internally. I've asked Tyler to reach out to the user and find out their configuration so that we can order an equivalent. Anything else we should ask for from them?
Assignee: nobody → benjamin
(In reply to Alex Keybl [:akeybl] from comment #14) > I've asked Tyler to reach out to the user and find out their configuration so > that we can order an equivalent. There are a dxdiag log in bug 789260 (AMD Radeon HD 6900 Series) and App Notes in bug 788559 (AMD Radeon HD 5700 Series).
Two Cards which are seeing this crash are ATI Radeon HD 5900 Series AMD Radeon HD 6900M Series These are both totally different cards, so starting to wonder if there is a third-party thing interfering here (OS?). Doing user outreach now, please let me know if you have more information you'd like me to gather.
If we can order one of the 6900 Series(may be Radeon 6950 ), it help's with Bug 783517 as well
I put a request (REQ0010751) for the AMD Radeon HD 6900M Series (or 6950).
I've got the card from Amazon, and I've installed in a Win7 machine in the QA Lab. So far so good. If you have suggestions as to what to try, let me know. Saphhire Radeon HD 6950 Catalyst Control Center shows this information in the Software section: Driver Packaging Version 8.93-111205a-132104C-ATI Catalyst Version 12.1 Provider Advanced Micro Devices, Inc. 2D Driver Version 8.01.01.1223 2D Driver File Path /REGISTRY/MACHINE/SYSTEM/ControlSet001/Control/Class/{4D36E968-E325-11CE-BFC1-08002BE10318}/0002 Direct3D Version 7.14.10.0887 OpenGL Version 6.14.10.11399 Catalyst Control Center Version 2011.1205.2215.39827 I'll check for updates and try a few things.
You'll need to update to Catalyst 12.8 (you are on 21.1 right now). Most crashes seem random.
Yeah, I installed Catalyst 12.8 and I've been browsing with one and two monitors, Flash sties, multiple tabs, multiple windows, fiddled with Catalyst settings, but no luck yet. This is a Win7 (32bit) if it matters.
juan, is it possible to try with x64? I've seen at least one report that specified that, the others didn't say which they were using.
I've set up Win7 64bit on the same machine with same card and up to date software and I've been trying several things, but no luck. The most I've gotten is to display a black title bar while using Flash and and an Aero theme, but everything else seems to be working properly.
I've also commented in https://bugzilla.mozilla.org/show_bug.cgi?id=789260#c10 . I'm on W7 Pro x64 with a HD 5970 and two monitors. Using 12.8 / 8.982 and hardware acceleration -> crashes. No HWA -> no crash. Installed 12.6 / 8.980.0.0 -> no crashes with and without HWA. Yes, crashes are random but usually have been observed after a few minutes of running and using the browser. Crash happened during clicking on links or dragging scrollbar of browser window. Or when switching second monitor on/off via software (use this for playing games). I've read https://developer.mozilla.org/en-US/docs/How_to_get_a_stacktrace_with_WinDbg but the links pointing to http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx seem to have been written when it was not yet updated for Windows 8. I'm confused what exactly there I'd need to download to assist further (which I'm willing to do).
Note user of Catalyst 12.8 on support forum https://support.mozilla.org/en-US/questions/937737 has reproducible crashes with Empty Crash IDs posted, & resolved by disabling hardware acceleration. The user is happy to be contacted if that would help(Preferably by sumo pm: https://support.mozilla.org/en-US/messages/new?to=Ascention).
I commented in https://bugzilla.mozilla.org/show_bug.cgi?id=789251#c25 and I've dual GPU system. I read the report from the user John has posted ( https://crash-stats.mozilla.com/report/index/bc2f99fb-f705-44f1-936a-a90a12120922 ) and it says "Has dual GPUs" I read until page 6 on this thread http://hardforum.com/showthread.php?t=1711450&page=2 . I don't want to draw conclusions too early, but people mentioning explicit Firefox crashes (not flash) and providing more information seems all related to dual GPU systems (either single card or two cards). I also found this video http://www.youtube.com/watch?v=sUNE-9UnPI4 mentioning 12.8 on a dual GPU system and Firefox "issues" (no crash though).
i've gone through the crash report questions i've answered in the last few days and the one john was already mentioning was the only user out of ten which had a dual gpu setup, so i don't think this has to be related. the other reported crashes were for those graphic cards: ATI Radeon HD 5600, AdapterDeviceID: 0x68d8 https://support.mozilla.org/questions/937711 https://crash-stats.mozilla.com/report/index/74fb46d5-1201-4d28-9c3d-953c62120918 ATI Radeon HD 5700, AdapterDeviceID: 0x68be https://support.mozilla.org/questions/937584 https://crash-stats.mozilla.com/report/index/087fbef6-143c-4b34-811c-4b92e2120920 https://support.mozilla.org/questions/937586 https://crash-stats.mozilla.com/report/index/3c492bc7-9ed7-452b-94c7-c74692120920 ATI Radeon HD 5800, AdapterDeviceID: 0x689e https://support.mozilla.org/questions/937193 https://crash-stats.mozilla.com/report/index/c20357ae-fe29-4f9b-b831-4d2982120915 https://support.mozilla.org/questions/937723 https://crash-stats.mozilla.com/report/index/616b34f8-599c-4f82-925c-aca8f2120922 AMD Radeon HD 6800, AdapterDeviceID: 0x6738 https://support.mozilla.org/questions/936125 https://crash-stats.mozilla.com/report/index/58174619-6875-4044-9fee-4e92c2120902 https://support.mozilla.org/questions/937357 https://crash-stats.mozilla.com/report/index/c1ea8895-7bb4-4855-a099-24e2a2120918 AMD Radeon HD 6900, AdapterDeviceID: 0x671d https://support.mozilla.org/questions/937322 https://crash-stats.mozilla.com/report/index/7499595f-3566-4d9c-b76d-cfbfa2120917 https://support.mozilla.org/questions/937265 https://crash-stats.mozilla.com/report/index/9641d1bc-a703-42e0-bc2e-dbd362120916
This bug is to track the 0-byte minidump (see comment 6), not the Graphics issue which is bug 789260 that will be fixed by bug 792480.
It isn't really possible to look into the 0-byte minidump issue unless we can reproduce the bug on a debuggable machine. I'm going to hand this to Juan for now, but if we can't catch it we should probably just mark this INCOMPLETE.
Assignee: benjamin → jbecerra
Thought I would comment in case it helps. Referencing bug 788559 the crash dumps stopped being 0-byte after installing windbg and the related symbols.
Interesting, that would indicate that having a newer version of DbgHelp.dll makes us able to write a minidump.
I wonder if the newer versions of dbghelp don't use the default heap and instead use CreateHeap to do their internal allocations; that would avoid almost all of the memory corruption issues internal to creating a minidump.
We could try shipping a newer DbgHelp.dll with Firefox and seeing what that does to our crash-stats...
This was caused by a new release of 3rd party software, so we won't block FF16 on it. We will, however, continue to track for FF17.
tracking-firefox16: + → -
tracking-firefox17: --- → +
There are about 96,000 crashes over the last week and there were 120,000 ones over one week before September 24, so the AMD driver blocklist (bug 792480) makes this signature drop to 20%.
No longer blocks: 789260
Depends on: 789260
(In reply to Ted Mielczarek [:ted] from comment #34) > We could try shipping a newer DbgHelp.dll with Firefox and seeing what that > does to our crash-stats... Filed bug 797323 on this.
Summary: HWA Crash, 0-byte minidump with Catalyst 12.8 (8.982) → HWA Crash, 0-byte minidump with Catalyst 12.8 (8.982) (Empty Crash)
Is qawanted relevant to this bug anymore? If so, please comment as to the specific request.
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #38) > Is qawanted relevant to this bug anymore? If so, please comment as to the > specific request. No, I don't believe there's further QA action here. I'll ask Benoit if this is something we can bring up on Wednesday.
Assignee: jbecerra → bjacob
Keywords: qawanted
(at our upcoming meeting with AMD)
We didn't end up discussing this bug with AMD. If you still think that it is worth mentioning this to AMD, better do it by email. To be clear, though, GPU vendors have very little power to get people to update their drivers, so at most one could hope help understanding what the problem is, but for that we would need to be able to give them specific information about this bug, but if this bug gives empty minidumps we won't be able to. Is there anything you still want me to do on this bug?
(In reply to Benoit Jacob [:bjacob] from comment #41) > We didn't end up discussing this bug with AMD. If you still think that it is > worth mentioning this to AMD, better do it by email. To be clear, though, > GPU vendors have very little power to get people to update their drivers, so > at most one could hope help understanding what the problem is, but for that > we would need to be able to give them specific information about this bug, > but if this bug gives empty minidumps we won't be able to. > > Is there anything you still want me to do on this bug? Nope, I've emailed AMD now and they're taking a look. Nothing more to be done here.
tracking-firefox17: + → -
Have we heard from AMD (perhaps on if they have fixed this in later versions and trying to get existing users with 12.8 to update?)
Closing because no crash reported since 12 weeks.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
Closing because no crash reported since 12 weeks.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: