700103 - MapsGL triggers C++ exceptions in D3D and eventually, crashes Firefox, when D3D9 debugging is enabled

Status: RESOLVED WONTFIX

(Core :: Graphics: CanvasWebGL, defect)

Description

[Robert O'Callahan (:roc) (email my personal email if necessary)]

1) In DirectX Control Panel, D3D9 tab, select "Use Debug version of Direct3D 9" and enable all options in the "Debugging" group. Set Debug output level to "More".
2) Attach to browser trunk build with Visual Studio 2010 debugger.
3) In Debug menu, "Exceptions...", enable breaking on C++ and Win32 exceptions
4) Visit Google Maps and enable MapsGL
5) Pan around and zoom in a lot

I get a lot of
WARNING: Failed to create new texture for CreateOptimalSurface!

Pretty soon I get exceptions being thrown, triggering debugger breaks:
 	msvcrt.dll!__CxxThrowException@8()  + 0x45 bytes	
>	d3d9d.dll!CD3DBase::DoValidateVSToPSLinkage()  + 0x69d bytes	
 	d3d9d.dll!CD3DBase::DrawPrimitive()  + 0x2ec bytes	
 	libGLESv2.dll!gl::Context::drawArrays(unsigned int mode, int first, int count)  Line 2716	C++
 	libGLESv2.dll!glDrawArrays(unsigned int mode, int first, int count)  Line 65535	C++
 	xul.dll!mozilla::gl::GLContext::fDrawArrays(unsigned int mode, int first, int count)  Line 948	C++
 	xul.dll!mozilla::WebGLContext::DrawArrays(unsigned int mode, int first, int count)  Line 1649	C++
 	xul.dll!nsIDOMWebGLRenderingContext_DrawArrays(JSContext * cx, unsigned int argc, JS::Value * vp)  Line 24431	C++

Eventually Firefox crashes and attaching the debugger to the crashed process shows the same sort of stack. Seems like it fails to recover from the exception being thrown.

Comment 1

[Benoit Jacob [:bjacob] (mostly away)]

Can any crash or misbehavior be reproduced without D3D debugging? I.e. could this be a bug purely in D3D debugging?

Comment 2

[Benoit Jacob [:bjacob] (mostly away)]

Another question: in a debug build of Firefox, with MOZ_GL_DEBUG=1, do you get a different stack? Because of the DoValidateVSToPSLinkage frame in your above stack, I expect that the bug could happen as early as in the linkProgram call.

Comment 3

[Benoit Jacob [:bjacob] (mostly away)]

Also, can the problem be reproduced with 32bit Firefox? We already know that the 64bit version of D3D9 is quite buggy and that it shows on Google Maps, see bug 694495.

Comment 4

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to Benoit Jacob [:bjacob] from comment #1)
> Can any crash or misbehavior be reproduced without D3D debugging? I.e. could
> this be a bug purely in D3D debugging?

If I skip step 1 (don't enable debugging) then everything seems to work OK from the user point of view. (Which doesn't necessarily mean everything *is* working OK, of course.)

(In reply to Benoit Jacob [:bjacob] from comment #2)
> Another question: in a debug build of Firefox, with MOZ_GL_DEBUG=1, do you
> get a different stack? Because of the DoValidateVSToPSLinkage frame in your
> above stack, I expect that the bug could happen as early as in the
> linkProgram call.

I get the same stack using MOZ_GL_DEBUG=1.

(In reply to Benoit Jacob [:bjacob] from comment #3)
> Also, can the problem be reproduced with 32bit Firefox? We already know that
> the 64bit version of D3D9 is quite buggy and that it shows on Google Maps,
> see bug 694495.

This is a 32-bit Firefox build.

Comment 5

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #4)
> If I skip step 1 (don't enable debugging) then everything seems to work OK
> from the user point of view. (Which doesn't necessarily mean everything *is*
> working OK, of course.)

In the sense that I probably wouldn't notice if some map details were drawn incorrectly.

I noticed that in one Street View instance the display was corrupt, but that might have been bad data for that one location, or a MapsGL bug, or something else entirely unrelated.

Comment 6

[Kelsey Gilbert [:jgilbert]]

D3D9 is WONTFIX