600870 - crashes [@ atioglxx.dll@0x...... ] often inside mozilla::WebGLContext::DrawArrays

Status: RESOLVED WORKSFORME

(Core :: Graphics: CanvasWebGL, defect)

Description

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

The last three days, we've seen atioglxx.dll@0x(some address) show up in the topcrash list, though the address has been different for each day's build (though always ending in 3):

http://crash-stats.mozilla.com/report/list?product=Firefox&branch=2.0&platform=windows&build_id=20100930041305&query_search=signature&query_type=exact&query=&date=09%2F30%2F2010%2009%3A35%3A20&range_value=30&range_unit=days&hang_type=any&process_type=all&plugin_field=&plugin_query_type=&plugin_query=&do_query=1&admin=&signature=atioglxx.dll%400x162fb3
http://crash-stats.mozilla.com/report/list?product=Firefox&branch=2.0&platform=windows&build_id=20100929041446&query_search=signature&query_type=exact&query=&date=09%2F30%2F2010%2009%3A35%3A22&range_value=30&range_unit=days&hang_type=any&process_type=all&plugin_field=&plugin_query_type=&plugin_query=&do_query=1&admin=&signature=atioglxx.dll%400x15a1d3
http://crash-stats.mozilla.com/report/list?product=Firefox&branch=2.0&platform=windows&build_id=20100928041914&query_search=signature&query_type=exact&query=&date=09%2F30%2F2010%2009%3A35%3A23&range_value=30&range_unit=days&hang_type=any&process_type=all&plugin_field=&plugin_query_type=&plugin_query=&do_query=1&admin=&signature=atioglxx.dll%400x152503

The stacks either:
 (1) have only addresses in atioglxx.dll, or
 (2) go down to mozilla::WebGLContext::DrawArrays and then beyond, as in bp-059f1779-022c-4e2e-86d2-672362100930
(It's expected that we have trouble walking stacks for DLLs that breakpad doesn't have, so this isn't too surprising.)

Only one crash (bp-2b72bc6f-0d79-42a4-b16a-68a872100929) had a user comment, which was "webGl".

All of the crashes were on "Windows NT 6.1.7600".  (Is that Win7?)

Comment 1

[Benoit Jacob [:bjacob] (mostly away)]

By default, OpenGL driver stack traces are meaningless because OpenGL is an asynchronous API: return immediately, crash later.

In order to debug this, we need that someone who can reproduce this applies the openGL Debug Mode patch for bug 597881, makes a debug build, and runs with this environment variable:

    MOZ_GL_DEBUG=1

This turns OpenGL into a synchronous API, making stack traces point to the actual cause of the problem.

Bonus points for attaching a debugger when it crashes and calling DumpJSStack() so we know where in the javascript it's crashing...

Comment 2

[Joe Drew (not getting mail)]

Hopefully we can find someone who has a crash like this.

Comment 3

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

<Mitch> was on IRC this morning asking about bp-995a1989-bbcd-480d-83d2-99ba42100930, which may be similar.  I'm not sure if I'm cc:ing the right <Mitch>, though.

Comment 4

[Mitchell Field [:Mitch]]

Attachment: stack trace

Here's a WinDbg log of Firefox, run with MOZ_GL_DEBUG=1, built in debug mode from http://hg.mozilla.org/mozilla-central/rev/e85f6f5ba0bf plus the patch from bug 597881.

(Note that C:\ff is a symlink to D:\ff (the objdir), so those paths in the log are interchangeable.)

Steps to reproduce:
* Go to http://www.khronos.org/webgl/wiki/Demo_Repository.
* In a normal release-mode Minefield trunk nightly build, the "Many Planets Deep" demo crashed 100% of the time, but in my local debug build, it didn't always crash, or it simply didn't (attempt to) run long enough to crash.
* Click through to the "san-angeles" demo, and if this doesn't crash, navigate back to the Demo Repository page and try the "Many Planets Deep" demo.
* Navigate back and forth between the Demo Repository, "san-angeles" and "Many Planets Deep" pages as many times as required to crash.

Comment 5

[Benoit Jacob [:bjacob] (mostly away)]

Part that seems relevant to me:

(eb0.1160): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for C:\Windows\SysWOW64\atioglxx.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Windows\SysWOW64\atioglxx.dll - 
atioglxx!atiPS+0xd5543:
10e62503 80bc287409000000 cmp     byte ptr [eax+ebp+974h],0 ds:002b:12910908=??
0:000:x86> |* ~* kp

.  0  Id: eb0.1160 Suspend: 1 Teb: 7efdb000 Unfrozen
ChildEBP RetAddr  
WARNING: Stack unwind information not available. Following frames may be wrong.
003fcfdc 10e61e6d atioglxx!atiPS+0xd5543
003fcffc 10e61838 atioglxx!atiPS+0xd4ead
00000000 00000000 atioglxx!atiPS+0xd4878


Since atiPS+0xd5543 is the crash address, and this address is only mentioned in this stack trace, this seems to be the crashing stack trace.


So, given that this was with MOZ_GL_DEBUG=1, this means an internal ATI driver crash. Indeed, MOZ_GL_DEBUG=1 makes glFinish() be called after every GL command, so any crash that is our fault would crash there at the latest and the stack trace would point to there. Here this stack trace does not trace back to our code, only to ATI driver code, so it's their crash.

---> conclusion: ATI driver bug.

Comment 6

[Benoit Jacob [:bjacob] (mostly away)]

So, another person hit this driver bug today on #gfx. We should contact ATI about it. How do we do that?

Comment 7

[Rain [:rain1] (inactive)]

I was the one who hit the bug (with the Flight of the Navigator demo). This is all with the latest ATI Catalyst Mobility drivers (10.8) on Win7 x64.

Comment 8

[timeless]

(just got back from my month-long summer vacation)

i filed bug 584403 about ati w/ a cc list and it hasn't had any nibbles. iirc i also called someone from ati, but if i did that, it too hasn't seen anything.

Comment 10

[Scoobidiver (away)]

This is not a moving crash signature as it is written in comment 0, these are three different crash signatures. Here below are crash reports for each one without build id filter:

* atioglxx.dll@0x15a1d3:
Crashes first appeared in b7pre/20100929 build.
http://crash-stats.mozilla.com/report/list?product=Firefox&branch=2.0&platform=windows&query_search=signature&query_type=exact&query=&range_value=30&range_unit=days&hang_type=any&process_type=all&plugin_field=&plugin_query_type=&plugin_query=&do_query=1&admin=&signature=atioglxx.dll%400x15a1d3

* atioglxx.dll@0x152503:
Crashes first appeared in b7pre/20100928 build.
http://crash-stats.mozilla.com/report/list?product=Firefox&branch=2.0&platform=windows&query_search=signature&query_type=exact&query=&range_value=30&range_unit=days&hang_type=any&process_type=all&plugin_field=&plugin_query_type=&plugin_query=&do_query=1&admin=&signature=atioglxx.dll%400x152503

* atioglxx.dll@0x162fb3:
Crashes only appeared in b7pre/20100930 build.
http://crash-stats.mozilla.com/report/list?product=Firefox&branch=2.0&platform=windows&query_search=signature&query_type=exact&query=&range_value=30&range_unit=days&hang_type=any&process_type=all&plugin_field=&plugin_query_type=&plugin_query=&do_query=1&admin=&signature=atioglxx.dll%400x162fb3

According to me, it is not an ATI driver bug but a WebGL bug.
The regression range is :
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=6272ce1da6f5&tochange=c257bfb8cad0
It could be caused by the fixing of one of the following bugs :
bug 592769, bug 596034

Comment 11

[Benoit Jacob [:bjacob] (mostly away)]

The fixing of bug 596034 did unearth a new bug: bug 602183 which is definitely a ATI driver bug. However, bug 602183 was fixed on Oct. 6. If this crash is still persisting, then 596034 can't be the cause.

As for 592769, I don't believe that it can be a cause of such crashes. It only takes effect on shutdown, anyway.

That the crash appears in different signatures doesn't mean much by itself: since OpenGL is an asynchronous API, where functions return immediately and crash later, stack traces are often pointing to an irrelevant place. Addressing this issue is one of the points of bug 597881. With this patch applied, in a debug build, with MOZ_GL_DEBUG=1, stack traces point consistently to the relevant crash location.

Comment 12

[Scoobidiver (away)]

> However, bug 602183 was fixed on Oct. 6
According to crash stats, there have not been crashes since b7pre/20101007 build. So it seems to be fully fixed by bug 602183. It can be even considered as a dupe.
Can someone with an ATI card try to reproduce the crash to be sure of that ?

Comment 13

[Rain [:rain1] (inactive)]

It doesn't happen for me any more with the Flight of the Navigator demo.

Comment 14

[MrX1980]

Hello

It seams that I have no atioglxx.dll@0x crash anymore but I get "Minefiled not responding" If I open:
http://cubicvr.org/CubicVR.js/fluid_sim/GPUFluid.html
http://webglsamples.googlecode.com/hg/spacerocks/spacerocks.html (first Meteor impact)

Tested with:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b8pre) Gecko/20101011 Firefox/4.0b8pre
ATI Radeon HD 4670 AGP
ATI Catalyst 10.9 (8.771.0.0)

I will continuing the WebGL tests.