Android ANGLE compiler allocator-mismatch crashes (e.g. TCompiler::~TCompiler | TranslatorGLSL::~TranslatorGLSL | DeleteCompiler )

VERIFIED FIXED in Firefox 15

Status

()

Firefox for Android
General
--
critical
VERIFIED FIXED
6 years ago
2 years ago

People

(Reporter: nhirata, Assigned: glandium)

Tracking

(Depends on: 1 bug, {crash, reproducible})

14 Branch
Firefox 15
ARM
Android
crash, reproducible
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox15 verified, blocking-fennec1.0 +)

Details

(Whiteboard: [native-crash], webgl, crash signature, URL)

Attachments

(2 attachments, 1 obsolete attachment)

This bug was filed from the Socorro interface and is 
report bp-28a12a6a-1aa6-4d6f-a494-327a82120418 .
============================================================= 
Frame 	Module 	Signature 	Source
0 	libc.so 	libc.so@0x12014 	
1 	libmozglue.so 	libmozglue.so@0x6e11 	
2 	libmozglue.so 	libmozglue.so@0x876b 	
3 	libmozglue.so 	libmozglue.so@0x9627 	
4 	libmozalloc.so 	moz_free 	memory/mozalloc/mozalloc.cpp:81
5 	libxul.so 	std::__node_alloc::deallocate 	mozalloc.h:253
6 	libxul.so 	std::vector<TVariableInfo, std::allocator<TVariableInfo> >::~vector 	_alloc.h:323
7 	libxul.so 	TCompiler::~TCompiler 	gfx/angle/src/compiler/Compiler.cpp:101
8 	libxul.so 	TranslatorGLSL::~TranslatorGLSL 	gfx/angle/src/compiler/TranslatorGLSL.h:12
9 	libxul.so 	TranslatorGLSL::~TranslatorGLSL 	gfx/angle/src/compiler/TranslatorGLSL.h:12
10 	libxul.so 	DeleteCompiler 	gfx/angle/src/compiler/CodeGenGLSL.cpp:33
11 	libxul.so 	ShDestruct 	gfx/angle/src/compiler/ShaderLang.cpp:160
12 	libxul.so 	mozilla::WebGLContext::CompileShader 	content/canvas/src/WebGLContextGL.cpp:4697
13 	libxul.so 	nsIDOMWebGLRenderingContext_CompileShader 	obj-firefox/js/xpconnect/src/dom_quickstubs.cpp:23060
14 	libxul.so 	js::Interpret 	js/src/jscntxtinlines.h:314
15 	libxul.so 	js::RunScript 	js/src/jsinterp.cpp:475
16 	libxul.so 	js::Execute 	js/src/jsinterp.cpp:674
17 	libxul.so 	JS_EvaluateUCScriptForPrincipalsVersionOrigin 	js/src/jsapi.cpp:5291
18 	libxul.so 	nsJSContext::EvaluateString 	dom/base/nsJSEnvironment.cpp:1455
19 	libxul.so 	nsScriptLoader::EvaluateScript 	content/base/src/nsScriptLoader.cpp:918
20 	libxul.so 	nsScriptLoader::ProcessRequest 	content/base/src/nsScriptLoader.cpp:811
21 	libxul.so 	nsScriptLoader::ProcessScriptElement 	content/base/src/nsScriptLoader.cpp:658
22 	libxul.so 	nsScriptElement::MaybeProcessScript 	content/base/src/nsScriptElement.cpp:169
23 	libxul.so 	nsIScriptElement::AttemptToExecute 	nsIScriptElement.h:253
24 	libxul.so 	nsHtml5TreeOpExecutor::RunScript 	parser/html/nsHtml5TreeOpExecutor.cpp:779
25 	libxul.so 	nsHtml5TreeOpExecutor::RunFlushLoop 	parser/html/nsHtml5TreeOpExecutor.cpp:583
26 	libxul.so 	nsHtml5ExecutorReflusher::Run 	parser/html/nsHtml5TreeOpExecutor.cpp:97
27 	libxul.so 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:656
28 	libxul.so 	NS_ProcessNextEvent_P 	obj-firefox/xpcom/build/nsThreadUtils.cpp:245
29 	libxul.so 	mozilla::ipc::MessagePump::Run 	ipc/glue/MessagePump.cpp:110
30 	libxul.so 	MessageLoop::RunInternal 	ipc/chromium/src/base/message_loop.cc:208
31 	libxul.so 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:201
32 	libxul.so 	nsBaseAppShell::Run 	widget/xpwidgets/nsBaseAppShell.cpp:189
33 	libxul.so 	nsAppStartup::Run 	toolkit/components/startup/nsAppStartup.cpp:295
34 	libxul.so 	XREMain::XRE_mainRun 	toolkit/xre/nsAppRunner.cpp:3768
35 	libxul.so 	XREMain::XRE_main 	toolkit/xre/nsAppRunner.cpp:3845
36 	libxul.so 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3921
37 	libxul.so 	GeckoStart 	toolkit/xre/nsAndroidStartup.cpp:109
38 	libmozglue.so 	libmozglue.so@0x10535 	
39 	dalvik-heap (deleted) 	dalvik-heap @0xdfaa16 	
40 	dalvik-heap (deleted) 	dalvik-heap @0xf3352e 	
41 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x298086 	
42 	libdvm.so 	libdvm.so@0x1ec72 	
43 	dalvik-heap (deleted) 	dalvik-heap @0xdfaa16 	
44 	libdvm.so 	libdvm.so@0x5906b 	
45 	data@app@org.mozilla.fennec-1.apk@classes.dex 	data@app@org.mozilla.fennec-1.apk@classes.dex@0x106496 	
46 	libmozglue.so 	libmozglue.so@0x104e3 	
47 	dalvik-heap (deleted) 	dalvik-heap @0xdfaa16 	
48 	dalvik-heap (deleted) 	dalvik-heap @0x257dffe 	
49 	data@app@org.mozilla.fennec-1.apk@classes.dex 	data@app@org.mozilla.fennec-1.apk@classes.dex@0xf3692 	
50 	libc.so 	libc.so@0x14a13 	
51 	libdvm.so 	libdvm.so@0x98f4d 	
52 	libc.so 	libc.so@0x15877 	
53 	libmozglue.so 	libmozglue.so@0x104e3 	
54 	data@app@org.mozilla.fennec-1.apk@classes.dex 	data@app@org.mozilla.fennec-1.apk@classes.dex@0xf3692 	
55 	libc.so 	libc.so@0x15877 	
56 	libmozglue.so 	libmozglue.so@0x104e3 	
57 	data@app@org.mozilla.fennec-1.apk@classes.dex 	data@app@org.mozilla.fennec-1.apk@classes.dex@0xf3692 	
58 	libc.so 	libc.so@0x15ed9 	
59 	libdvm.so 	libdvm.so@0x5b009 	
60 	core.odex 	core.odex@0x1e46b6 	
61 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x347e 	
62 	dalvik-heap (deleted) 	dalvik-heap @0xf336ce 	
63 	dalvik-heap (deleted) 	dalvik-heap @0xf336ce 	
64 	libdvm.so 	libdvm.so@0xa2631 	
65 	dalvik-heap (deleted) 	dalvik-heap @0xf336ce 	
66 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x29809a 	
67 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x298086 	
68 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x298086 	
69 	data@app@org.mozilla.fennec-1.apk@classes.dex 	data@app@org.mozilla.fennec-1.apk@classes.dex@0x116ed1 	
70 	libdvm.so 	libdvm.so@0x8e9cb 	
71 	libdvm.so 	libdvm.so@0x749fb 	
72 	dalvik-heap (deleted) 	dalvik-heap @0xdfaa16 	
73 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x298086 	
74 	dalvik-heap (deleted) 	dalvik-heap @0x257dffe 	
75 	dalvik-heap (deleted) 	dalvik-heap @0x157dffe 	
76 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x298086 	
77 	libdvm.so 	libdvm.so@0x58e9b 	
78 	dalvik-heap (deleted) 	dalvik-heap @0xdfaa16 	
79 	libdvm.so 	libdvm.so@0x5ad99 	
80 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x298086 	
81 	data@app@org.mozilla.fennec-1.apk@classes.dex 	data@app@org.mozilla.fennec-1.apk@classes.dex@0x8caa6 	
82 	dalvik-heap (deleted) 	dalvik-heap @0xdfaa16 	
83 	libdvm.so 	libdvm.so@0x1edfe 	
84 	libdvm.so 	libdvm.so@0x30ace 	
85 	libdvm.so 	libdvm.so@0xb2f8e 	
86 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x2a0696 	
87 	libdvm.so 	libdvm.so@0x34326 	
88 	dalvik-heap (deleted) 	dalvik-heap @0x257dffe 	
89 	dalvik-heap (deleted) 	dalvik-heap @0x157dffe 	
90 	dalvik-heap (deleted) 	dalvik-heap @0x157dffe 	
91 	dalvik-heap (deleted) 	dalvik-heap @0x257dffe 	
92 	dalvik-LinearAlloc (deleted) 	dalvik-LinearAlloc @0x2a0696 	
93 	dalvik-heap (deleted) 	dalvik-heap @0xe11ec6 	
94 	data@app@org.mozilla.fennec-1.apk@classes.dex 	data@app@org.mozilla.fennec-1.apk@classes.dex@0x10615c 	
95 	libdvm.so 	libdvm.so@0x6ca95 	
96 	libdvm.so 	libdvm.so@0xb2f8e 	
97 	libdvm.so 	libdvm.so@0xb7c56 	
98 	libdvm.so 	libdvm.so@0x5fb3f 	
99 	libdvm.so 	libdvm.so@0x6cabb 	
100 	libdvm.so 	libdvm.so@0xb7c56 	
101 	libdvm.so 	libdvm.so@0x5fb3f 	
102 	libdvm.so 	libdvm.so@0xb2f8e 	
103 	libdvm.so 	libdvm.so@0x5fbef 	
104 	libdvm.so 	libdvm.so@0x5fb3f 	
105 	libc.so 	libc.so@0x12c1e 	
106 	libc.so 	libc.so@0x12772

STR:
1. go to http://helloracer.com/webgl/
Expected: a race car
Actual: crash and crash reporter listing the page prior to the actual page I crashed on.
Keywords: reproducible
Whiteboard: [native-crash]
Yea these reports were me hitting a bunch of WebGL demos during mobile triage. I can crash consistently on each and every single demo available at [1]

[1] http://www.khronos.org/webgl/wiki/Demo_Repository

Tested with today's Nightly (04/18), Galaxy Nexus (Android 4.0.4)
OS: Linux → Android
Hardware: All → ARM
Whiteboard: [native-crash] → [native-crash], webgl
also on www.ro.me and a bunch of other webgl demo sites.

It's not just you, I'm hitting them myself with the Nightly and Galaxy Nexus.  We're probably going to see a spike in the crash count due to us skewing the data.
Nominating due to ease of reproducibility.
blocking-fennec1.0: --- → ?

Updated

6 years ago
Assignee: nobody → bjacob
Duplicate of this bug: 746727
This looks like bug 746727, but missing some symbols.

Updated

6 years ago
Crash Signature: [@ libmozglue.so@0x6e11] → [@ libmozglue.so@0x6e11] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ __atomic_cmpxchg]
Summary: crash in [@ libmozglue.so@0x6e11] → crash in TCompiler::~TCompiler | TranslatorGLSL::~TranslatorGLSL | DeleteCompiler

Updated

6 years ago
Crash Signature: [@ libmozglue.so@0x6e11] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ __atomic_cmpxchg] → [@ libmozglue.so@0x6e11] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex_lock | malloc_mutex_lock | arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ __atomic_cmpxchg]

Updated

6 years ago
Crash Signature: [@ libmozglue.so@0x6e11] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex_lock | malloc_mutex_lock | arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ __atomic_cmpxchg] → [@ libmozglue.so@0x6e11] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex_lock | malloc_mutex_lock | arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ malloc_mutex_lock | arena_&hellip;
Either this is the same allocator mismatch as in bug 709947, in which case we need to prioritize this anyway to be able to reenable shader translation on Android; or this is something specific here that can only be fixed by reproducing in a debugger...
blocking-fennec1.0: ? → soft
This crash occurs on the Galaxy Nexus.  Other devices show the crash stack that's shown in bug 702651 for webgl.

Updated

6 years ago
Crash Signature: [@ libmozglue.so@0x6e11] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex_lock | malloc_mutex_lock | arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ malloc_mutex_lock | arena_&hellip; → [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x877a] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex_lock | malloc_mutex_lock | arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ &hellip;
So, it seems to be crashing while tearing down Compiler, apparently at 'attribs' or 'uniforms', which are both std::vector<TVariableInfo>.

Is it unusual that we see TranslatorGLSL::~TranslatorGLSL twice?
blocking-fennec1.0: soft → ?
blocking-fennec1.0: ? → soft

Updated

6 years ago
Depends on: 748654
I get always this crash with my Nexus S at the URL http://www.everyday3d.com/j3d/demo/004_Glass.html -> bp-e1d81cc6-7761-4d73-8c0e-f94fb2120425

Updated

6 years ago
Crash Signature: [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x877a] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex_lock | malloc_mutex_lock | arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ &hellip; → [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x88fc] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex_lock | malloc_mutex_lock | arena_dalloc | __wrap_free | moz_free | std::__nod&hellip;

Updated

6 years ago
Crash Signature: [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x88fc] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex_lock | malloc_mutex_lock | arena_dalloc | __wrap_free | moz_free | std::__nod&hellip; → [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x88fc] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex_lock | malloc_mutex_lock | arena_dalloc | __wrap_fr&hellip;

Updated

6 years ago
Crash Signature: [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x88fc] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex_lock | malloc_mutex_lock | arena_dalloc | __wrap_fr&hellip; → [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x6ee9] [@ libmozglue.so@0x8852] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x88fc] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex&hellip;

Updated

6 years ago
Crash Signature: [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x6ee9] [@ libmozglue.so@0x8852] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x88fc] [@ arena_dalloc | __wrap_free | moz_free | std::__node_alloc::deallocate ] [@ pthread_mutex&hellip; → [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x6ee9] [@ libmozglue.so@0x6f31] [@ libmozglue.so@0x8852] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x889a] [@ libmozglue.so@0x88fc] [@ libmozglue.so@0x89c4] [@ arena_dallo&hellip;

Updated

6 years ago
Duplicate of this bug: 750851
I seriously hope that bug 748654 fixes this. It's on inbound but will only reach central when the trees reopen.
The URL in bug 750851 is still crashing, even with bug 748654 fixed :(
Yes, Cody has been debugging this and saw that we have an allocator mismatch even with that fix. I'll let Cody expand.

Updated

6 years ago
status-firefox14: --- → affected
status-firefox15: --- → affected
Chris Jones and I worked on tracking all of this down yesterday.  We're not adding the STL wrappers for allocation if certain bugs are found in the configure.in.  Specifically $ac_cv_have_visibility_builtin_bug and $ac_cv_have_visibility_class_bug must not be present or we don't wrap them; that seems to have broken it.
Duplicate of this bug: 752829
(Assignee)

Comment 19

6 years ago
(In reply to Cody Brocious [:Daeken] from comment #16)
> Chris Jones and I worked on tracking all of this down yesterday.  We're not
> adding the STL wrappers for allocation if certain bugs are found in the
> configure.in.  Specifically $ac_cv_have_visibility_builtin_bug and
> $ac_cv_have_visibility_class_bug must not be present or we don't wrap them;
> that seems to have broken it.

These should only affect the visibility wrappers, not the STL wrappers. IOW, I think making STL_FLAGS and WRAP_STL_INCLUDES unconditional would work.
Duplicate of this bug: 702651
Summary: crash in TCompiler::~TCompiler | TranslatorGLSL::~TranslatorGLSL | DeleteCompiler → Android ANGLE compiler allocator-mismatch crashes (e.g. TCompiler::~TCompiler | TranslatorGLSL::~TranslatorGLSL | DeleteCompiler )

Updated

6 years ago
Crash Signature: [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x6ee9] [@ libmozglue.so@0x6f31] [@ libmozglue.so@0x8852] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x889a] [@ libmozglue.so@0x88fc] [@ libmozglue.so@0x89c4] [@ arena_dallo&hellip; → [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x6ee9] [@ libmozglue.so@0x6f31] [@ libmozglue.so@0x8852] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x884a] [@ libmozglue.so@0x887a] [@ libmozglue.so@0x8892] [@ libmozglue.&hellip;

Updated

6 years ago
Whiteboard: [native-crash], webgl → [native-crash], webgl [games:p2]

Updated

6 years ago
Whiteboard: [native-crash], webgl [games:p2] → [native-crash], webgl
Very similar/related bug: bug 740194.
Crash Signature: [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x6ee9] [@ libmozglue.so@0x6f31] [@ libmozglue.so@0x8852] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x884a] [@ libmozglue.so@0x887a] [@ libmozglue.so@0x8892] [@ libmozglue.&hellip; → [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x6ee9] [@ libmozglue.so@0x6f31] [@ libmozglue.so@0x8852] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x884a] [@ libmozglue.so@0x887a] [@ libmozglue.so@0x8892] [@ libmozglue.&hellip;
This is blocking verification of bug 739648.

Updated

6 years ago
Duplicate of this bug: 756917

Updated

6 years ago
Duplicate of this bug: 757013
Crash Signature: [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x6ee9] [@ libmozglue.so@0x6f31] [@ libmozglue.so@0x8852] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x884a] [@ libmozglue.so@0x887a] [@ libmozglue.so@0x8892] [@ libmozglue.&hellip; → [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x6ee9] [@ libmozglue.so@0x6f31] [@ libmozglue.so@0x8852] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x884a] [@ libmozglue.so@0x887a] [@ libmozglue.so@0x8892] [@ libmozglue.&hellip;
I see this crash on Nightly, Aurora and Beta when trying to display the demos here:
http://www.ibiblio.org/e-notes/webgl/webgl.htm

Comment 26

6 years ago
WebGL is unsupported while Flash is supported. That's odd for Mozilla that promotes HTML5 technologies.
Scoobidiver, AIUI we only turn off WebGL on Adreno phones because of unstable drivers. Are we doing more blocking that I don't know about?

Comment 28

6 years ago
(In reply to Joe Drew (:JOEDREW!) from comment #27)
> Scoobidiver, AIUI we only turn off WebGL on Adreno phones because of
> unstable drivers. Are we doing more blocking that I don't know about?
When WebGL is enabled,  it's so crashy that it's unusable, so it's unsupported. See comment 0, comment 1, comment 3, comment 10, comment 15 and comment 25.
Understood. We think we have a fix to this; Cody Brocious just needs to post it.
Created attachment 626514 [details] [diff] [review]
Configure patch

Attached is a patch for configure.in that fixes the issue for me in my testing, but it's very hacky and I don't think it actually fixes the root cause.

Here's what's know:
1) ANGLE is not having its allocator wrapped; it's ending up with new/delete pointing at the system allocator, which is ok in most cases, but not so much now.
2) Including mozalloc.h in one of the top-level ANGLE includes fixes this, as it causes the allocator to be wrapped properly.
3) The patch acts to nullify the checks for two bugs that seem to only affect GCC on x86_64, but the tests are being run across the board, and I believe it's setting off false positives
4) If one of those tests fails, we don't wrap anything.

However, I don't think this part of the configure.in is the actual root cause; it's code from 2007 that's served us well thus far, and I think that it's simply masking the actual problem.  More research has to be done.

In the meantime, we can fix the ANGLE issues entirely by including mozalloc.h, much like in the Skia bug #740194 and another place I can't recall now.  But the real, systemic problem really should be fixed as well, so this doesn't bite us in the future.

Updated

6 years ago
Assignee: bjacob → cbrocious
Status: NEW → ASSIGNED
We need someone knowledgeable to explain:
 * what determines which source files have their allocator overridden by jemalloc?
 * is this always done by #including "mozalloc.h" or is there another way?
 * if including "mozalloc.h" is the only way, is there a way to do this automatically for many files at once?

I wouldn't want to have to include mozalloc.h in some ANGLE header and have to rely on all ANGLE source files including that header... that would be very fragile.

Comment 32

6 years ago
<mwu> on android, it's done at link time (so creation)
<mwu> we tell the linker to wrap the memory allocator functions
<mwu> so calls to malloc get converted to __wrap_malloc
<mwu> and __wrap_malloc is jemalloc

However, if you're using some sort of stl container, the mozalloc stuff is probably necessary. The containers will tend to use the wrong allocator otherwise.
Thanks. Do you have any idea why we are currently not overriding the allocator in ANGLE, contrary to the rest of the codebase? The makefile is:

http://hg.mozilla.org/mozilla-central/file/320b16daa7c0/gfx/angle/Makefile.in

Is this doing something special that prevents the allocator overriding from working as it does in other directories? This code goes into libgkmedias, but we had the same issues before (last fall) when it was still going into libxul.

Can you also please comment on Cody's patch: is there something to patch in configure.in or should we do a fix in the ANGLE Makefile? If the latter, then what's our plan to avoid having these bugs over and over again in the future, in other directories? See bug 740194
The link-time stuff is indeed correct, but the problem is that without the wrappers, we end up linking to system new/delete.  The include side of things defines our own new/delete with a forced inline, which calls to jemalloc on the backend, so we never end up with these symbols existing at link time at all.
[16:43] <glandium> bjacob: i /think/ the stl wrappers are broken on android
[16:43] <mwu> bjacob: I don't see any issues with the angle makefile
[16:43] <cjones> bjacob, what's the problem?
[16:43] <bjacob> glandium: oh.
[16:43] <joe> it's drawn to a thebeslayer at some later time
[16:44] <-- Sander has left this server (Quit: And back he spurred like a madman, shrieking a curse to the sky.).
[16:44] <bjacob> cjones: see the bug; but i'm getting help from glandium and mwu now
[16:44] <mwu> cjones: they're hitting the same allocator mismatch we were seeing on b2g
[16:44] <jlebar> joe, But you're saying that if it's discarded before it's drawn to the thebeslayer, we'll trigger another decode?
[16:44] <glandium> bjacob: broken as in not used
[16:44] <cjones> bjacob, if any of the ANGLE code is linking to system ::operator new you're gonna have a bad time
[16:44] <cjones> or ::operator delete
[16:45] *** adrian is now known as adrian|away.
[16:45] <bjacob> glandium: so are you saying that using c++ (like operator new) allocations is completely supported?
[16:45] <mwu> but angle should be in libxul, no?
[16:45] <mwu> which autowraps everything
[16:45] --> sheppy has joined this channel (sheppy@moz-4BE034AB.ptr.us.xo.net).
[16:45] <mwu> not sure how TCompiler is escaping that
[16:45] <glandium> bjacob: i'm saying that android build system is ****
[16:45] <bjacob> cjones: angle is using some operator new but i have to dust my c++ to remember what's ::operator new again
[16:46] <cjones> i don't believe we wrap ::operator new/delete
[16:46] <cjones> _Znj or something like that
[16:46] <cjones> mangled
[16:46] <cjones> nm | c++filt will tell you
[16:46] <mwu> oh right, we don't have the --wrap thing for it
[16:46] <mwu> ****
[16:46] <bjacob> cjones: if i do:   int *i  = new int;    is that using ::operator new? or you mean something else?
[16:46] <cjones> if mozalloc is included you use mozalloc
[16:46] <cjones> otherwise system 
[16:46] <cjones> this works on other platforms because of the way we hook malloc
[16:47] <cjones> android is a special little bird
[16:47] <bjacob> cjones: are you saying that we should include mozalloc.h in all ANGLE files?
[16:47] <cjones> that would work
[16:47] <mwu> you could also add -include "path/to/mozalloc.h"
[16:47] <glandium> cjones: on other platforms we use stl wrappers, and we basically include mozalloc.h whenever a stl header is included
[16:47] <bjacob> mwu: sure, i was thinking of that
[16:47] <cjones> we solved this for skia by using skia's allocator wrappers
[16:48] <cjones> glandium, stl wrappers solve another problem
[16:48] <glandium> which we appear do *not* being doing on android
[16:48] <cjones> that's a separate but also bad problem
[16:48] <cjones> well, less bad
[16:48] <bjacob> cjones: could we do that in global CFLAGS so we dont run into this bug over and over again for each library we use?
[16:49] <cjones> sadly no
[16:49] <bjacob> cjones: why not
[16:49] <bjacob> ?
[16:49] <-- vladan1 has left this server (Ping timeout).
[16:49] <cjones> as i recall we have code that can't link to mozalloc
[16:49] <dbaron> Ms2ger`, I think that auto-refreshing thing is worth fixing
[16:49] <cjones> that may have changed in the meantime
[16:49] <mwu> cjones: so what's stopping us from wrapping operator new and delete?
[16:49] --> vladan has joined this channel (vladan@F2D29657.F60B0462.67AC9B1.IP).
[16:49] <cjones> also if we have code that needs to interoperate with system malloc we need to be able to call that
[16:49] --> cviecco_ has joined this channel (cviecco@moz-BBE3ABD.mv.mozilla.com).
[16:49] <cjones> mwu, ^^ that
[16:50] <cjones> i'm not 100% sure --wrap works with mangled C++ but that's easy to test
[16:50] <glandium> cjones: i do think using the stl wrappers would solve bjacob's problem
[16:50] <cjones> new Foo() is part of the C++ language
[16:50] <cjones> not part of STL
[16:50] <glandium> cjones: the stl wrappers take care of new Foo()
[16:51] <mwu> cjones: why the operator new, specifically? because we can't call the _real_ versions?
[16:51] <mwu> __real_
[16:51] <cjones> glandium, mozalloc takes care of that
[16:51] <cjones> stl wrappers solve the problem of making stl exception safe
[16:51] <bjacob> glandium: cjones: mwu: allow me to paste this conversation into the bug?
[16:51] <glandium> cjones: the stl wrappers take care of including mozalloc everywhere
[16:51] <cjones> mwu, not sure how that work work
[16:51] <cjones> *would work
[16:51] <cjones> it might
[16:52] <cjones> glandium, see above
[16:52] <cjones> they take care of ensuring STL headers use mozalloc and don't throw exceptions
[16:52] <mwu> not wrapping operator new and delete seems more of an oversight than intentional decision
[16:53] <cjones> possibly
[16:53] <cjones> we may need to hand new'd memory to libflash.so
[16:53] <mwu> random googling suggests it's doable - http://sources.redhat.com/ml/binutils/2004-08/msg00330.html
[16:53] <cjones> that'd be a case where we need to choose
[16:53] <glandium> bjacob: try export STL_FLAGS='-I$(DIST)/stl_wrappers' and export WRAP_STL_INCLUDES=1 in your mozconfig
[16:54] <-- avih has left this server (Ping timeout).
[16:54] <cjones> glandium, BTW the stl wrappers are being disabled on android because that gcc has a bug our configure.in finds
[16:54] <bjacob> glandium: that is exactly what Daeken's patch is doing
[16:54] [Invite] You invited Daeken to channel #developers.
[16:54] <-- davidb has left this server (Quit: davidb).
[16:54] <glandium> cjones: i know, and that code path is pretty dumb. there's no relationship between the gcc bug and stl wrappers
[16:54] <bjacob> glandium: and he says that works but he also says he doesn't think that's the right solution
[16:54] <bjacob> glandium: can you read the bug and his patch?
[16:54] <cjones> it's a visibility problem
[16:55] <cjones> i'm not 100% sure they're unrelated
[16:55] <cjones> (if you know they aren't, cool, just saying i don't know)
[16:55] <glandium> cjones: his patch is wrong, but the underlying idea is the right one, imho
[16:55] <cjones> android --wrap didn't exist when stl wrappers were written
[16:55] <cjones> it's not, unfortunately
[16:55] <cjones> it might work for libs we use
[16:56] <cjones> it'll still miss new Foo()
[16:56] <cjones> in a file that doesn't use STL
[16:56] <bjacob> can i paste this conversation to the bug?
[16:56] <-- JonathanS has left this server (Quit: Computer has gone to sleep.).
[16:56] --> mreavy has joined this channel (chatzilla@moz-C03D0C61.vlan426.asr1.sfo1.gblx.net).
[16:56] <-- artur has left this server (Input/output error).
[16:56] *** andreasn is now known as andreasn_away.
[16:56] <cjones> orthogonally, there are two right ideas
[16:56] <mwu> why not? people post random things to qdb all the time
[16:57] <cjones> one is to globally --wrap new/delete if we can get away with it
[16:57] <cjones> two is to fix the STL wrappers if we know we can enabled them with the android gcc
Filed bug 758010 for the long-term fix.

Updated

6 years ago
Depends on: 758010
(Assignee)

Comment 37

6 years ago
Created attachment 626711 [details] [diff] [review]
Always use the STL wrappers when #pragma visibility is supported

This is enough for the ANGLE case, because it includes STL headers. Resulting objects in objdir/gfx/angle don't contain any reference to the operator new/delete symbols.
This however doesn't solve the entire problem (libxul.so still has references to the operator new/delete symbols, from somewhere else), which will be addressed in bug 758010.
The change in mozglue/build is required because a) xpcom-config.h (which mozalloc.h include) is not available when building there, and b) mozglue can't depend on mozalloc.
<none>
Attachment #626711 - Flags: review?(ted.mielczarek)

Updated

6 years ago
Assignee: cbrocious → mh+mozilla
(Assignee)

Comment 38

6 years ago
Created attachment 626739 [details] [diff] [review]
Always use the STL wrappers when #pragma visibility is supported

With an additional change to make Mac gcc happy. (we happened not to be using the stl wrappers on mac either)
Attachment #626739 - Flags: review?(ted.mielczarek)
(Assignee)

Updated

6 years ago
Attachment #626711 - Attachment is obsolete: true
Attachment #626711 - Flags: review?(ted.mielczarek)
Attachment #626739 - Flags: review?(ted.mielczarek) → review+
Hey, this seems to have hit central already:
https://hg.mozilla.org/mozilla-central/rev/81c2e2ea2dbf

Close? Wait to see if it bounces? Today's nightly has more allocator mismatches than ever, so it's not impossible that this would bounce.
https://hg.mozilla.org/mozilla-central/rev/81c2e2ea2dbf
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Assignee)

Comment 42

6 years ago
Backed out because of bug 758648.
https://hg.mozilla.org/mozilla-central/rev/034bbdc7b9c9

It looks like we won't have cheese until bug 758010.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Updated

6 years ago
Depends on: 758648
(Assignee)

Updated

6 years ago
Depends on: 758773
(Assignee)

Updated

6 years ago
Depends on: 758858
Crash Signature: [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x6ee9] [@ libmozglue.so@0x6f31] [@ libmozglue.so@0x8852] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x884a] [@ libmozglue.so@0x887a] [@ libmozglue.so@0x8892] [@ libmozglue.&hellip; → [@ libmozglue.so@0x6e11] [@ libmozglue.so@0x6ee9] [@ libmozglue.so@0x6f31] [@ libmozglue.so@0x8852] [@ libmozglue.so@0x877a] [@ libmozglue.so@0x87d2] [@ libmozglue.so@0x884a] [@ libmozglue.so@0x887a] [@ libmozglue.so@0x8892] [@ libmozglue.&hellip;
Blocks: 736481
blocking-fennec1.0: soft → +
qawanted to close this if the crash is fixed
Keywords: qawanted
> http://www.ibiblio.org/e-notes/webgl/deflate/spider.html

I can repro w/ Nexus S, ICS:
F/libc    ( 1227): @@@ ABORTING: INVALID HEAP ADDRESS IN dlfree
F/libc    ( 1227): Fatal signal 11 (SIGSEGV) at 0xdeadbaad (code=1)
I/ActivityManager(  148): Process org.mozilla.fennec_sewardj (pid 1227) has died.
This may be a total red herring, #include <disclaimer.h>, etc, anyway ...

I did a bunch of Valgrind runs of Fennec on Nexus S on ICS.  This is
with enhancements to support Jemalloc directly and also to detect
allocator mismatches.

Observations:

(1) it crashes natively all the time on all of the WebGL cases
    mentioned here (well, _occasionally_ it doesn't)

(2) it doesn't crash at all when running on V

(3) the set of errors reported by V each time is not identical,
    although there is some overlap

(4) V does not find any obvious smoking gun.  

(5) There's a lot of complaining about some compilery thing (is this
    the ANGLE compiler that keeps getting mentioned?) but still
    nothing that obviously would kill the process and clearly isn't a
    false positive.

The combination of (1) (2) (3) and (4) inclines me to wonder whether
this might be a threading problem.  In particular V screws extensively
with thread scheduling and it can happen that buggy threaded code that
crashes natively stays alive on V (or vice versa).


FWIW, here are selected lowlights of the V runs:


This and a bunch of clones of it, turned up in one run.  I rushed to
turn on my unwind-by-stack-scanning hack, to get more stacktrace, but
then it never showed up in any other run.  I don't know if it is a
false positive, but on contemplation of some V internal stuff, I don't
think it is.

Invalid read of size 4
   at 0x2CDF1C98: ??? (in /dev/ashmem)
 Address 0x1699bfe8 is 114,688 bytes inside a block of size 131,072 free'd
   at 0x480691C: __wrap_free (vg_replace_malloc.c:458)
   by 0x2CD6F51D: ??? (in /dev/ashmem)


This showed up one time.  No idea.  Anyway, is not a crasher.

 Mismatched free() / delete / delete []
    at 0x48061F0: __wrap__ZdlPv (vg_replace_malloc.c:494)
    by 0x2CBE193B: ??? (in /dev/ashmem)
    by 0x2CC063A5: ??? (in /dev/ashmem)
    by 0x2CBF07E1: ??? (in /dev/ashmem)
    by 0x2C3816AF: ??? (in /dev/ashmem)
    by 0x2CC94BA1: ??? (in /dev/ashmem)
    by 0x2C9E8F5F: ??? (in /dev/ashmem)
  Address 0x15abfd38 is 0 bytes inside a block of size 883 alloc'd
    at 0x48071D0: __wrap_malloc (vg_replace_malloc.c:275)
    by 0x17BD972B: ??? (in /dev/ashmem)
    by 0x2C070B3D: ??? (in /dev/ashmem)
    by 0x2CC0544D: ??? (in /dev/ashmem)
    by 0x2C89B9EF: ??? (in /dev/ashmem)
    by 0x2CC05551: ??? (in /dev/ashmem)
    by 0x2CE944BB: ??? (in /dev/ashmem)


There's a vast amount of complaining about some compilery thing, in
this case reading off the end of a block.  I don't think I saw any
invalid writes, though.

Invalid read of size 4
   at 0x173AE3B0: GetLiveChansInFNRMArgument (dce.c:2674)
   by 0x173AE237: GetLiveChansInArgument (dce.c:3476)
   by 0x173B203F: ComputeLivenessForInst (dce.c:5347)
   by 0x173B25A7: LivenessDF (dce.c:5715)
   by 0x1745224B: DoDataflow (usc.c:3312)
   by 0x173B10BB: DoLiveness (dce.c:5776)
   by 0x173B126F: DeadCodeElimination (dce.c:5907)
   by 0x174553CF: CompileIntermediate (usc.c:8290)
   by 0x1745694B: CompileUniflex (usc.c:9216)
   by 0x17456B77: PVRUniFlexCompileToUspBin (usc.c:9649)
   by 0x27E634FB: GenerateUniPatchInput (ic2uf.c:10345)
   by 0x27E5B5E7: GLSLCompileToUniflex (glsl2uf.c:195)
 Address 0x16ee995c is 0 bytes after a block of size 28 alloc'd
   at 0x4807648: malloc (vg_replace_malloc.c:273)
   by 0x174508AB: UscAllocfn (usc.c:390)
   by 0x17409CB3: InitInstructionTypeNRM (inst.c:10650)
   by 0x1740A903: SetOpcodeAndDestCount (inst.c:16200)
   by 0x173F74D3: ConvertInstToIntermediateF32 (icvt_f32.c:4030)
   by 0x173E050B: ConvertInstToIntermediate (icvt_core.c:7280)
   by 0x173E173F: DoConvert (icvt_core.c:10363)
   by 0x173E18DF: BuildCFG (icvt_core.c:8504)
   by 0x173E924B: ConvertToIntermediate (icvt_core.c:17604)
   by 0x1745693B: CompileUniflex (usc.c:9199)
   by 0x17456B77: PVRUniFlexCompileToUspBin (usc.c:9649)
   by 0x27E634FB: GenerateUniPatchInput (ic2uf.c:10345)
(In reply to Julian Seward from comment #46)
> (5) There's a lot of complaining about some compilery thing (is this
>     the ANGLE compiler that keeps getting mentioned?)

No, the stacks for the last invalid read in your comment show that it's the GLSL compiler of the system OpenGL driver on this phone, not the ANGLE GLSL compiler. Indeed, see the "PVR" prefixes, they stand for "PowerVR", the brand of GPU in your phone.

Since you mention possible threading issues, let's mention that typically, OpenGL implementations (like the one provided by this OpenGL driver) are asynchronous: OpenGL functions return as early as possible and the real work is executed in a worker thread. So, it is expected that the OpenGL implementation creates threads.

All in all, the typical picture is:

WebGL function: on Gecko main thread, code in mozilla-central/content/canvas/src

  |
  |
  V

If the WebGL function is compileShader, it will call a source-to-source shader compiler in ANGLE (mozilla-central/gfx/angle) to validate it and perform some transformations.

  |
  |
  V

The WebGL function then calls the corresponding OpenGL ES function in libGLESv2.so

  |
  |
  V

The OpenGL ES function tries to return as early as possible and do real work asynchronously in a worker thread. A few functions however have to do all/most of their work synchronously. It will typically eventually call into a device-specific driver library, in your case it should have 'PVR' or 'PowerVR' in its filename.
(followup to comment #46)

> This showed up one time.  No idea.  Anyway, is not a crasher.
> 
>  Mismatched free() / delete / delete []
>     at 0x48061F0: __wrap__ZdlPv (vg_replace_malloc.c:494)
>     by 0x2CBE193B: ??? (in /dev/ashmem)
>   Address 0x15abfd38 is 0 bytes inside a block of size 883 alloc'd
>     at 0x48071D0: __wrap_malloc (vg_replace_malloc.c:275)
>     by 0x17BD972B: ??? (in /dev/ashmem)

With symbols it is (alas still harmless):

Mismatched free() / delete / delete []
   at 0x48061F0: __wrap__ZdlPv (vg_replace_malloc.c:494)
   by 0x2C7D093B: TCompiler::~TCompiler() (_new.h:135)
   by 0x2C7F53A5: TranslatorESSL::~TranslatorESSL() (TranslatorESSL.h:12)
   by 0x2C7F2BE1: DeleteCompiler(TCompiler*) (CodeGenGLSL.cpp:33)
   by 0x2C7DF7E1: ShDestruct (ShaderLang.cpp:162)
   by 0x2BF706AF: mozilla::WebGLContext::CompileShader(mozilla::WebGLShader*) (WebGLContextGL.cpp:5014)
   by 0x2BF70779: mozilla::WebGLContext::CompileShader(nsIWebGLShader*) (WebGLContextGL.cpp:4840)
   by 0x2C2F32A9: nsIDOMWebGLRenderingContext_CompileShader(JSContext*, unsigned int, JS::Value*) (dom_quickstubs.cpp:23152)
   by 0x2C8B3339: js::InvokeKernel(JSContext*, js::CallArgs, js::MaybeConstruct) (jscntxtinlines.h:395)
   by 0x2C8AF8A5: js::Interpret(JSContext*, js::StackFrame*, js::InterpMode) (jsinterp.cpp:2512)
   by 0x2C8B2951: js::RunScript(JSContext*, JSScript*, js::StackFrame*) (jsinterp.cpp:266)
   by 0x2C8B33BB: js::InvokeKernel(JSContext*, js::CallArgs, js::MaybeConstruct) (jsinterp.cpp:326)
 Address 0x151c6f88 is 0 bytes inside a block of size 883 alloc'd
   at 0x48071D0: __wrap_malloc (vg_replace_malloc.c:275)
   by 0x17C7A72B: moz_xmalloc (mozalloc.cpp:54)
   by 0x2BC5FB3D: std::string::_M_append(char const*, char const*) (mozalloc.h:200)
   by 0x2C7F444D: TOutputGLSLBase::writeConstantUnion(TType const&, ConstantUnion const*) (_string.h:530)
   by 0x2C7F4635: TOutputGLSLBase::visitConstantUnion(TIntermConstantUnion*) (OutputGLSLBase.cpp:207)
   by 0x2C7DAF25: TIntermConstantUnion::traverse(TIntermTraverser*) (IntermTraverse.cpp:33)
   by 0x2C7DB0A3: TIntermAggregate::traverse(TIntermTraverser*) (IntermTraverse.cpp:161)
   by 0x2C7DB0A3: TIntermAggregate::traverse(TIntermTraverser*) (IntermTraverse.cpp:161)
   by 0x2C7DB0A3: TIntermAggregate::traverse(TIntermTraverser*) (IntermTraverse.cpp:161)
   by 0x2C7DAFB3: TIntermBinary::traverse(TIntermTraverser*) (IntermTraverse.cpp:89)
   by 0x2C7DAFB3: TIntermBinary::traverse(TIntermTraverser*) (IntermTraverse.cpp:89)
   by 0x2C7F4BB3: TOutputGLSLBase::visitAggregate(Visit, TIntermAggregate*) (OutputGLSLBase.cpp:454)
The stack in comment 48 is exactly the bug we've been dealing with here in this bug. Are you saying you got that with a build that included glandium's patch in bug 758010 (cset ef0180b53eae) ?

Not sure what you mean by 'harmless' in comment 48? This has been a major crasher, see earlier comments above.
(Assignee)

Comment 50

6 years ago
(In reply to Benoit Jacob [:bjacob] from comment #49)
> The stack in comment 48 is exactly the bug we've been dealing with here in
> this bug. Are you saying you got that with a build that included glandium's
> patch in bug 758010 (cset ef0180b53eae) ?
> 
> Not sure what you mean by 'harmless' in comment 48? This has been a major
> crasher, see earlier comments above.

It's harmless in the sense that they are not an actual allocator mismatch (both are the wrapped allocators, so both are jemalloc). However, it looks like the allocating path and the freeing path are very unrelated, so that would seem to indicate a corruption of some sort. But this is not an allocator mismatch anymore.

The allocator mismatch was probably there before bug 758010, but it was likely not the root cause.
Removing QA wanted from this bug, since this bug is not fixed.  Please add qawanted if there is some action QA is to take.
Keywords: qawanted
(Assignee)

Comment 52

6 years ago
To clarify: the reopening of bug 758010 probably doesn't mean much for the present bug, or at least for what valgrind found, since valgrind replaces both allocators.
(Assignee)

Comment 53

6 years ago
(In reply to Mike Hommey [:glandium] from comment #52)
> To clarify: the reopening of bug 758010 probably doesn't mean much for the
> present bug, or at least for what valgrind found, since valgrind replaces
> both allocators.

FWIW, a build without the new patch from bug 758010 crashes for me on the spider, and a build with the patch doesn't. I still think the allocator mismatch is hiding another problem, though.
I confirm, the patch at 758010 comment 9 stops the crashing on the
spider.  It also successfully runs the first 4 demos at http://www.khronos.org/webgl
/wiki/Demo_Repository, including running the San Angeles demo for several
minutes without failing.

Also, the mismatch problem of comment #46 is still present.

Comment 55

6 years ago
This bug is blocking the fennec native release, but comments 49/50 indicate it might be harmless now and with the crashes fixed, I'm wondering if this blocks now.
(Assignee)

Comment 56

6 years ago
I think we can mark this bug as fixed. The various allocator mismatches that have been identified will be fixed in bug 758858.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED

Updated

6 years ago
status-firefox14: affected → fixed
status-firefox15: affected → fixed
No crashes could be reproduced on any of the different URLs provided in this bug. Marking as verified

Tested on:
Firefox Mobile 15.0b3/Nightly 17.0a1 2012-08-02/Aurora 16.0a2 2012-08-02
Samsung Galaxy Tab (Android 3.1)/ HTC Desire Z (Android 2.3.3)
Status: RESOLVED → VERIFIED
status-firefox14: fixed → ---
status-firefox15: fixed → verified
You need to log in before you can comment on or make changes to this bug.