WebGL ANGLE D3D11 Crash in [@ msvcr120.dll@0x5da69 ]

RESOLVED FIXED in Firefox 39

Status

()

defect
--
critical
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: vasilica.mihasca, Assigned: kamidphish)

Tracking

({crash})

37 Branch
mozilla41
x86
Windows 8.1
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox37+ wontfix, firefox38+ wontfix, firefox39+ fixed, firefox40 fixed, firefox41 fixed, firefox-esr3839+ affected)

Details

(Whiteboard: [gfx-noted], crash signature)

Attachments

(2 attachments)

This bug was filed from the Socorro interface and is 
report bp-77264948-764b-44dd-9363-573bf2150325 .


- Ran into this on Windows 8.1 32-bit, using Firefox 37 (build ID: 20150324194038).
- I don't have proper STR, but it happened during WebGl games testing (e.g. http://oos.moxiecode.com/js_webgl/monster_truck/ , http://racer.nomo.hu/ )
- Tried several other times to reproduce it, without any luck.
- The crash rates increased in Firefox 37 Beta 6 and doubled in Firefox 37 Beta 7 and are affected only 32bit versions of Windows
Summary: Crash in [@ msvcr120.dll@0x5da69 ] → WebGL ANGLE D3D11 Crash in [@ msvcr120.dll@0x5da69 ]
[Tracking Requested - why for this release]: A possible WebGL regression. The severity of this depends on widespread the crash is. It might be hidden among multiple signature because it shows up as 

0 	kernelbase.dll 	RaiseException 	
1 	msvcr120.dll 	_CxxThrowException 	f:\dd\vctools\crt\crtw32\eh\throw.cpp:152
2 	msvcr120.dll 	msvcr120.dll@0x5da69 	
3 	libglesv2.dll 	std::allocator<char>::allocate(unsigned int) 	c:/tools/vs2013/vc/include/xmemory0:578
4 	libglesv2.dll 	std::vector<unsigned char, std::allocator<unsigned char> >::_Reserve(unsigned int) 	c:/tools/vs2013/vc/include/vector:1617
5 	libglesv2.dll 	rx::d3d11::GenerateInitialTextureData(int, unsigned int, unsigned int, unsigned int, unsigned int, std::vector<D3D11_SUBRESOURCE_DATA, std::allocator<D3D11_SUBRESOURCE_DATA> >*, std::vector<std::vector<unsigned char, std::allocator<unsigned char> >, std::allocator<std::vector<unsigned char, std::allocator<unsigned char> > > >*)

We really want to be looking to see how much this is happening from GenerateInitialTextureData.
Tracking to follow up on severity of the crash.
The stack from the crash report in comment 0 looks like this in a debugger:
0099ba20 6c379339 KERNELBASE!RaiseException+0x62
0099ba60 6c3bda6a msvcr120!_CxxThrowException+0x5b
0099ba80 6bd44388 msvcr120!operator new+0x50
0099ba88 6bdc3fb0 libGLESv2!std::allocator<char>::allocate(unsigned int _Count = 0x369f10)+0x16 [c:\tools\vs2013\vc\include\xmemory0 @ 578]
0099baa0 6bdc229d libGLESv2!std::vector<unsigned char,std::allocator<unsigned char> >::_Reserve(unsigned int _Count = 0x369f10)+0x53 [c:\tools\vs2013\vc\include\vector @ 1617]
0099bae0 6bdb2e86 libGLESv2!rx::d3d11::GenerateInitialTextureData(int internalFormat = 0n3, unsigned int width = 0, unsigned int height = 0x3b2, unsigned int depth = 1, unsigned int mipLevels = 1, class std::vector<D3D11_SUBRESOURCE_DATA,std::allocator<D3D11_SUBRESOURCE_DATA> > * outSubresourceData = 0x779547d0, class std::vector<std::vector<unsigned char,std::allocator<unsigned char> >,std::allocator<std::vector<unsigned char,std::allocator<unsigned char> > > > * outData = 0x0099bb2c)+0x153 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\d3d11\renderer11_utils.cpp @ 1032]
0099bb64 6bdb25db libGLESv2!rx::Image11::createStagingTexture(void)+0x1ee [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\d3d11\image11.cpp @ 548]
0099bba4 6bdb257e libGLESv2!rx::Image11::copyToStorageImpl(class rx::TextureStorage11 * storage11 = 0x3ce97670, struct gl::ImageIndex * index = 0x0099bc3c, struct gl::Box * region = 0x0099bc24)+0x56 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\d3d11\image11.cpp @ 143]
0099bbbc 6bda8062 libGLESv2!rx::Image11::copyToStorage2DArray(class rx::TextureStorage * storage = 0x3ce97670, struct gl::ImageIndex * index = 0x0099bc3c, struct gl::Box * region = 0x0099bc24)+0x14 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\d3d11\image11.cpp @ 107]
0099bc04 6bda7f27 libGLESv2!rx::TextureD3D_2D::commitRegion(struct gl::ImageIndex * index = 0x0099bc3c, struct gl::Box * region = 0x0099bc24)+0x49 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 907]
0099bc6c 6bda7e9a libGLESv2!rx::TextureD3D_2D::updateStorageLevel(int level = 0n0)+0x7d [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 865]
0099bc88 6bda7daa libGLESv2!rx::TextureD3D_2D::updateStorage(void)+0x3e [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 847]
0099bc98 6bda72c3 libGLESv2!rx::TextureD3D_2D::initializeStorage(bool renderTarget = true)+0x46 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 810]
0099bcd0 6bda7c74 libGLESv2!rx::TextureD3D::ensureRenderTarget(void)+0x1d [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 385]
0099bcd8 6bd80811 libGLESv2!rx::TextureD3D_2D::getRenderTargetSerial(struct gl::ImageIndex * index = 0x77884684)+0x8 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 723]
0099bce8 6bdb73d4 libGLESv2!rx::GetAttachmentSerial(class gl::FramebufferAttachment * attachment = 0x00000003)+0x26 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\framebuffer.cpp @ 55]
0099bd74 6bd7ddb2 libGLESv2!rx::Renderer11::applyRenderTarget(class gl::Framebuffer * framebuffer = 0x3ce7dab8)+0xb6 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\d3d11\renderer11.cpp @ 850]
0099bdac 6bd7e8ef libGLESv2!gl::Context::applyRenderTarget(unsigned int drawMode = 4, bool ignoreViewport = true)+0x2c [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\context.cpp @ 1327]
0099be5c 6bd9227a libGLESv2!gl::Context::clear(unsigned int mask = 0x4500)+0x60 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\context.cpp @ 1622]
0099beb0 538a94ef libGLESv2!glClear(unsigned int mask = 0x4500)+0x53 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\libglesv2.cpp @ 652]
0099bf64 538c9439 xul!mozilla::WebGLContext::ForceClearFramebufferWithDefaultValues(unsigned int mask = 0x4500, bool * colorAttachmentsMask = 0x0099bf98)+0x1db [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\dom\canvas\webglcontext.cpp @ 1367]
0099bfa8 538a8be2 xul!mozilla::WebGLFramebuffer::CheckAndInitializeAttachments(void)+0x156 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\dom\canvas\webglframebuffer.cpp @ 891]
0099bfe0 538a8806 xul!mozilla::WebGLContext::DrawElements_check(int count = 0n6, unsigned int type = 0x1403, int64 byteOffset = 0n0, int primcount = 0n1, char * info = 0x54565990 "drawElements", unsigned int * out_upperBound = 0x0099c010)+0x2da [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\dom\canvas\webglcontextdraw.cpp @ 284]
0099c014 53856713 xul!mozilla::WebGLContext::DrawElements(unsigned int mode = 4, int count = 0n6, unsigned int type = 0x1403, int64 byteOffset = 0n0)+0x4d [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\dom\canvas\webglcontextdraw.cpp @ 317]
0099c048 52c29698 xul!mozilla::dom::WebGLRenderingContextBinding::drawElements(struct JSContext * cx = 0x4bded6d0, class JS::Handle<JSObject *> obj = class JS::Handle<JSObject *>, class mozilla::WebGLContext * self = 0x28485400, class JSJitMethodCallArgs * args = 0x00001403)+0xa8 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\obj-firefox\dom\bindings\webglrenderingcontextbinding.cpp @ 10207]
0099c08c 28485400 xul!mozilla::dom::GenericBindingMethod(struct JSContext * cx = 0x4bded6d0, unsigned int argc = 4, class JS::Value * vp = 0x0099c0c0)+0xc8 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\dom\bindings\bindingutils.cpp @ 2485]

It's an OOM, but it's coming by way of a thrown C++ exception. I guess since this is in ANGLE we don't get our standard infallible malloc behavior, and it's clearly built with C++ exceptions.
Whiteboard: [gfx-noted]
Dan, can you also take a look?
Flags: needinfo?(dglastonbury)
Assigning to Dan. Feel free to reassign if you are not going to work on this.
Assignee: nobody → dglastonbury
This is out-of-memory, right? Maybe out-of-address space on 32-bit windows. Oh, I see that Ted wrote the same at the bottom of his long call stack comment.

Do we want to catch the exception from ANGLE, or do we want to disable exceptions in ANGLE.
Flags: needinfo?(dglastonbury)
If the allocation size is content-controllable then we ought to be able to handle OOM here. I don't know what ANGLE does in these situations, is it intended to be built with exceptions enabled and catch allocation failure? Does it have any existing policy for OOM handling?
d3d11::GenerateInitialTextureData seems to be called with width 0, height 946.  Its caller, Image11::createStagingTexture, asserts that width and height are positive.  That at least seems wrong, but I don't know that it's the cause.
On the other hand, this zero gets taken out (and set to 1) before we crash, so it probably isn't the cause.
Dan, any progress on this issue? We just have a few beta for 38 (btw, it is an ESR release). Thanks
Flags: needinfo?(dglastonbury)
I've just got free of other tasks to start looking at this. It's top of my priority list.
Flags: needinfo?(dglastonbury)

Updated

4 years ago
Keywords: crash
Digging into this, I think we are still being bitten by Bug 1134565, or something similar to it.  ANGLE is throwing an exception on OOM. The ANGLE coding standard[1], in section "Other C++ Features", says:

> {DONT} use C++ exceptions, they are disabled in the builds and not caught. 

So, here, ANGLE is throwing an exception, which it should not, and it's bubbling through our code and results in a crash.

[1] https://code.google.com/p/angleproject/wiki/CodingStandard
I've traced through the code in the MSVC STL implementation and the assertion in the ANGLE coding standard about exceptions and the reality don't match.

From my understanding there is no way to disable the exception thrown from std::vector when memory is exhausted because std::vector uses std:allocator, and allocate is spec'd to throw std::bad_alloc. (http://en.cppreference.com/w/cpp/memory/allocator/allocate)

Milan, where should we go from here?
Flags: needinfo?(milan)
I need input from a build peer. Should we been using the Mozilla STL wrapper, or the system one? ANGLE's moz.build files seem keen to disable the STL wrapper.
Flags: needinfo?(mh+mozilla)
I've modified the moz.build files for ANGLE and now allocation calls into moz_xmalloc. Simulating OOM condition in moz_xmalloc results in mozalloc_abort and program crash. (Still)
(In reply to Dan Glastonbury :djg :kamidphish from comment #15)
> I've modified the moz.build files for ANGLE

modified how?

(In reply to Dan Glastonbury :djg :kamidphish from comment #14)
> I need input from a build peer. Should we been using the Mozilla STL
> wrapper, or the system one? ANGLE's moz.build files seem keen to disable the
> STL wrapper.

After bug 1134565, it shouldn't be. If it is, then either your tree is too old or something regressed it.
Flags: needinfo?(mh+mozilla)
(In reply to Mike Hommey [:glandium] from comment #16)
> (In reply to Dan Glastonbury :djg :kamidphish from comment #15)
> > I've modified the moz.build files for ANGLE
> 
> modified how?

Removed DISABLE_STL_WRAPPING = True from the moz.build files in src/libGLESv2 and src/libEGL.

> After bug 1134565, it shouldn't be. If it is, then either your tree is too
> old or something regressed it.

That bug moved DISABLE_STL_WRAPPINGS into the libGLESv2 & libEGL. This is what I've removed locally.
(In reply to Dan Glastonbury :djg :kamidphish from comment #17)
> That bug moved DISABLE_STL_WRAPPINGS into the libGLESv2 & libEGL. This is
> what I've removed locally.

While it might look this way if you look at the generate_mozbuild.py change, that's actually not the case if you look at the resulting moz.build change that landed in m-c. That is, DISABLE_STL_WRAPPING was already there and was left there. It was only removed for angle itself (what ends up in libxul).
This doesn't fix the crash for user when OOM, but uses Moz infallible
allocator, so is reporting the OOM via that mechanism instead of by
uncaught exception.
Flags: needinfo?(milan)
This is too late for 38 but we could take a patch for 38 esr if it is safe.
Attachment #8599072 - Flags: review?(mh+mozilla)
Attachment #8599072 - Flags: review?(jmuizelaar)
Attachment #8599073 - Flags: review?(mh+mozilla)
Attachment #8599073 - Flags: review?(jmuizelaar)
Attachment #8599072 - Flags: review?(mh+mozilla) → review+
Attachment #8599073 - Flags: review?(mh+mozilla) → review+
Attachment #8599073 - Flags: review?(jmuizelaar) → review+
Attachment #8599072 - Flags: review?(jmuizelaar) → review+
https://hg.mozilla.org/mozilla-central/rev/928a48a1e116
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla41
I don't see any crashes with this signature in the last two weeks for 39. 
Dan, do you think that 39 is still affected?
Flags: needinfo?(dglastonbury)
I'd expect nobody to be using 39 at this point, since it's not aurora anymore, and there's not been a beta yet.
Liz, I think that 39 will still be affected but I defer to :glandium for the reason.
Flags: needinfo?(dglastonbury)
Can you request uplift?  It looks like this should be uplifted to aurora and probably beta.
Flags: needinfo?(dglastonbury)
Comment on attachment 8599072 [details] [diff] [review]
Enable STL wrapping for libGLESv2 and libEGL.

Approval Request Comment
[Feature/regressing bug #]: 1134565
[User impact if declined]: Very little. This change doesn't fix the OOM crash, but changes the reporting path. Instead of throwing a std::bad_alloc exception on OOM, mozalloc infallible allocator is used. 
[Describe test coverage new/current, TreeHerder]: Changes ANGLE allocator which is tested by all Windows runs of mochitest-gl.
[Risks and why]: Low. This patch has been landed on nightly for 4 weeks with out causing issues.
[String/UUID change made/needed]:
Flags: needinfo?(dglastonbury)
Attachment #8599072 - Flags: approval-mozilla-beta?
Attachment #8599072 - Flags: approval-mozilla-aurora?
Comment on attachment 8599072 [details] [diff] [review]
Enable STL wrapping for libGLESv2 and libEGL.

Looks like a relatively safe change and early in the cycle. Taking it.
Attachment #8599072 - Flags: approval-mozilla-beta?
Attachment #8599072 - Flags: approval-mozilla-beta+
Attachment #8599072 - Flags: approval-mozilla-aurora?
Attachment #8599072 - Flags: approval-mozilla-aurora+
You need to log in before you can comment on or make changes to this bug.