Closed Bug 673075 Opened 13 years ago Closed 13 years ago

Recursion in WebGL shader causes memory corruption


(Core :: Graphics: CanvasWebGL, defect)

5 Branch
Not set



Tracking Status
firefox6 - affected
firefox7 - affected
firefox8 + fixed
firefox9 --- fixed
status1.9.2 --- unaffected


(Reporter: marc, Assigned: bjacob)



(Whiteboard: [sg:dos][sg:vector-?][qa-])


(1 file)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30

Steps to reproduce:

gdb --args /Applications/

i) open

ii) Paste the following into the script textarea 
     It basically recalls main() within the shader code    

==start paste==
#ifdef GL_ES
precision highp float;

uniform vec2 resolution;
uniform float time;
uniform sampler2D tex0;
uniform sampler2D tex1;

float reccount; 

void main(void)
    reccount = reccount + 1.0 ; 
    if (reccount > 60.0) {
==end paste==

iv) Compile (alt+enter)

Actual results:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00007fff5f3fffc4
0x00007fff80b8d40f in glvmStoreDestTempWithMask ()

(gdb) i r 
rax            0x72041000	1912868864
rbx            0x3902	14594
rcx            0x7fff5f400250	140734791418448
rdx            0x7fff5f400480	140734791419008
rsi            0x7fff5fbfad60	140734799785312
rdi            0x7fff5fbfb5e0	140734799787488
rbp            0x7fff5f400240	0x7fff5f400240
rsp            0x7fff5f400038	0x7fff5f400038
r8             0x22000072041000	9570151121031168
r9             0x4000000000000022	4611686018427387938
r10            0xe408	58376
r11            0x7204	29188
r12            0x22	34
r13            0x7fff5fbfad50	140734799785296
r14            0x1c81	7297
r15            0x22	34
rip            0x7fff80b8d40f	0x7fff80b8d40f <glvmStoreDestTempWithMask+63>
eflags         0x10246	66118
cs             0x27	39
ss             0x0	0
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0
(gdb) disass $pc $pc+1
Dump of assembler code from 0x7fff80b8d40f to 0x7fff80b8d410:
0x00007fff80b8d40f <glvmStoreDestTempWithMask+63>:	mov    %r12d,-0x27c(%rbp)

(gdb) bt
#0  0x00007fff80b8d40f in glvmStoreDestTempWithMask ()
#1  0x00007fff80b9f89f in glvmInterpretFPTransformFourInner ()
#2  0x00007fff80b9d8fb in glvmInterpretFPTransformFourInner ()
#3  0x00007fff80b9d8fb in glvmInterpretFPTransformFourInner ()
#10671 0x00007fff80b9d8fb in glvmInterpretFPTransformFourInner ()
#10672 0x00007fff80b9d8fb in glvmInterpretFPTransformFourInner ()
#10673 0x00007fff80ba14d0 in glvmInterpretFPTransformFour ()
#10674 0x0000000114d51b3c in gldLLVMFPTransformFallback ()
#10675 0x0000000114d601a3 in gldLLVMVecPolyRender ()
#10676 0x0000000114d5c371 in gldSetPolyRenderFunc ()
#10677 0x0000000114d6d23f in gldTessellateTrianglesRGBA ()
#10678 0x0000000114d6d315 in gldRenderTriangles ()
#10679 0x0000000115828b51 in gleVPRenderTrianglesSmooth ()
#10680 0x0000000115795e13 in gleDrawArraysOrElements_ExecCore ()
#10681 0x000000011579c1e5 in glDrawArrays_ACC_Exec ()

Expected results:

looks like invalid write, probably those recursion constructs could be detected before they hit a place they could cause harm.
Component: Security → Canvas: WebGL
Product: Firefox → Core
QA Contact: firefox → canvas.webgl
Benoit or Jeff: please investigate.
Assignee: nobody → bjacob
At first glance, I would guess this is a problem with 'reccount' never being initialized, and likely starting at some large negative value, making the stack likely to explode. I'm not sure what the spec says re: uninitialized values.

Testing on 5 and 8 show that they both work for me, so maybe it depends on implementation/platform? (Tested on Win7 w/nVidia card)

"The GLSL memory model does not allow for recursive function calls. This allows GLSL to execute on hardware that simply doesn't allow for recursion."

So it seems like a bug in both ANGLE and the driver, that they allow recursion. We should get this fixed in ANGLE.
Here, I confirm that ANGLE allows recursion. I don't get the crash however because my NVIDIA driver correctly rejects the shader.
Summary: Recursion in WebGL script causes memory corruption → Recursion in WebGL shader causes memory corruption
Mac: what's your GPU?
Marc: on your machine that exhibits this problem please open about:support and paste the "Graphics" block into this bug.

I think we've confirmed at least a DoS recursion bug, and possibly something exploitable with Marc's specific configuration that we have not been able to reproduce.
Ever confirmed: true
Whiteboard: [sg:dos][sg:vector-?]
MacBook Air late 2010 model that crashes: 
Adapter Description 0x22600,0x20400
Vendor ID 0000
Device ID 0000
Adapter RAM
Adapter Drivers
Driver Version
Driver Date
Direct2D Enabled false
DirectWrite Enabled false
WebGL Renderer NVIDIA Corporation -- NVIDIA GeForce 320M OpenGL Engine -- 2.1 NVIDIA-1.6.36
GPU Accelerated Windows 1/1 OpenGL
Not crashing on iMac early 2010, ATI:  

Karten-Beschreibung 0x21a00,0x20400
Direct2D aktiviert false
DirectWrite aktiviert false
WebGL-Renderer ATI Technologies Inc. -- ATI Radeon HD 4850 OpenGL Engine -- 2.1 ATI-1.6.36
GPU-beschleunigte Fenster1/1 OpenGL
Recursion detection has been implemented in ANGLE r711.  Such shaders won't be allowed in WebGL when that revision is rolled in.
Imported ANGLE r711:
Closed: 13 years ago
Resolution: --- → FIXED
Illustrates crash scenario eating all available hard disk space , 51G in this case and DoSing the machine. The sligthly modified reproducer is calling main() two times per recursion level.
I don't understand how a shader can eat all disk space, but anyways, Firefox 8+ should now be rejecting recursive shaders.

Do we want to take the fix in Beta / Firefox 7? It adds a nontrivial amount of new code, but fixes a serious issue.
I can't see ANGLE bug 191 -- did they think this was worse than a recursion DoS? I also don't understand how a recursive shader uses any disk let alone 51G; swap maybe?
Target Milestone: --- → mozilla8
I presume it was talking lots of memory and eventually swap.  Recursion isn't allowed in an version of GLSL so it's now properly detected.
(In reply to Daniel Veditz from comment #15)
> I can't see ANGLE bug 191 -- did they think this was worse than a recursion
> DoS?

I just initially filed it as a security bug, don't read anything specific into that. As Daniel says it's been fixed for a while anyways.
qa- as this requires a debug build to verify the fix. If someone has a debug build, please verify the fix using comment 0.
Whiteboard: [sg:dos][sg:vector-?] → [sg:dos][sg:vector-?][qa-]
Group: core-security
You need to log in before you can comment on or make changes to this bug.