Closed Bug 941788 Opened 11 years ago Closed 10 years ago

Intermittent crashes in content/canvas/test/reftest/webgl-* tests on Android x86 emu

Categories

(Core :: Graphics: CanvasWebGL, defect)

Product:
Core
Component:
Graphics: CanvasWebGL
Platform:
x86
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED DUPLICATE of bug 1001682

People

(Reporter: gbrown, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: webgl-internal)

Attachments

(1 file)

skip all canvas reftests on Android 4.2 x86 emu
893 bytes, patch
jgilbert
: review-
Details | Diff | Splinter Review 
I am trying to enable a new Android test environment, based on the Android x86 emulator running Android 4.2. In this new test environment, there are intermittent crashes and intermittent failures in many of the content/canvas/test/reftest/webgl-* tests, especially the webgl-clear-test and webgl-color-test variants. 

Many of these tests are already unreliable on B2G and Tegras:

random-if(B2G) random-if(Android&&AndroidVersion<15)

My default solution for getting the reftests green would change these to:

skip-if(Android&&AndroidVersion>=17) random-if(B2G) random-if(Android&&AndroidVersion<15)

Alternately, can someone investigate the crashes soon-ish?


One example:

https://tbpl.mozilla.org/php/getParsedLog.php?id=30805403&tree=Cedar&full=1

19:24:34     INFO - REFTEST TEST-END | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-clear-test.html?__&_____&________
19:24:34     INFO - REFTEST TEST-START | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-clear-test.html?aa&_____&________
19:24:34     INFO - RESTORE PREFERENCE pref(webgl.force-enabled,false)
19:24:34     INFO - SET PREFERENCE pref(webgl.force-enabled,true)
19:24:34     INFO - REFTEST TEST-LOAD | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-clear-test.html?aa&_____&________ | 1084 / 3781 (28%)
19:24:34     INFO - RESTORE PREFERENCE pref(webgl.force-enabled,false)
19:24:34     INFO - SET PREFERENCE pref(webgl.force-enabled,true)
19:24:34     INFO - REFTEST TEST-LOAD | http://10.0.2.2:8856/tests/content/canvas/test/reftest/wrapper.html?green.png | 1084 / 3781 (28%)
19:24:34     INFO - INFO | automation.py | Application ran for: 0:06:02.801106
19:24:34     INFO - INFO | zombiecheck | Reading PID log: /tmp/tmpM5rIsUpidlog
19:24:34     INFO - mozcrash INFO | Downloading symbols from: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/cedar-android-x86/1384909322/fennec-28.0a1.en-US.android-i386.crashreporter-symbols.zip
19:24:34     INFO - /data/anr/traces.txt not found
19:24:34     INFO - PROCESS-CRASH | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-clear-test.html?aa&_____&________ | application crashed [@ jemalloc_crash]
19:24:34     INFO - Crash dump filename: /tmp/tmpe5DitR/68806b2e-44bf-a816-4f66bbcc-76d74827.dmp
19:24:34     INFO - Operating system: Android
19:24:34     INFO -                   0.0.0 Linux 2.6.29 #2 PREEMPT Thu Nov 15 13:38:07 CST 2012 i686 generic_x86/sdk_x86/generic_x86:4.2/JOP40C/eng.android-build.20121231.103448:eng/test-keys
19:24:34     INFO - CPU: x86
19:24:34     INFO -      GenuineIntel family 6 model 3 stepping 3
19:24:34     INFO -      1 CPU
19:24:34     INFO - 
19:24:34     INFO - Crash reason:  SIGSEGV
19:24:34     INFO - Crash address: 0x0
19:24:34     INFO - 
19:24:34     INFO - Thread 12 (crashed)
19:24:34     INFO -  0  libmozglue.so!jemalloc_crash [jemalloc.c:a24bf8dce031 : 1572 + 0x0]
19:24:34     INFO -     eip = 0xaa1c439c   esp = 0xa9765f80   ebp = 0xa9765f88   ebx = 0xaa205d4c
19:24:34     INFO -     esi = 0xaa09b040   edi = 0xaa09ba58   eax = 0x982b6000   ecx = 0x98200898
19:24:34     INFO -     edx = 0x00000000   efl = 0x00210282
19:24:34     INFO -     Found by: given as instruction pointer in context
19:24:34     INFO -  1  libmozglue.so!arena_malloc [jemalloc.c:a24bf8dce031 : 4194 + 0x4]
19:24:34     INFO -     eip = 0xaa1c67ad   esp = 0xa9765f90   ebp = 0xa9765fc8   ebx = 0xaa205d4c
19:24:34     INFO -     esi = 0xaa09b040   edi = 0xaa09ba58
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO -  2  libmozglue.so!__wrap_malloc [jemalloc.c:a24bf8dce031 : 4211 + 0x10]
19:24:34     INFO -     eip = 0xaa1c6fe9   esp = 0xa9765fd0   ebp = 0xa9765fe8   ebx = 0xaa205d4c
19:24:34     INFO -     esi = 0x00000300   edi = 0x0000001c
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO -  3  libxul.so!PL_DHashAllocTable [pldhash.cpp:a24bf8dce031 : 72 + 0x7]
19:24:34     INFO -     eip = 0xa156156e   esp = 0xa9765ff0   ebp = 0xa9766008   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x95e13c00   edi = 0x0000001c
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO -  4  libxul.so!PL_DHashTableInit [pldhash.cpp:a24bf8dce031 : 222 + 0xb]
19:24:34     INFO -     eip = 0xa1561685   esp = 0xa9766010   ebp = 0xa9766038   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x95e13c00   edi = 0x0000001c
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO -  5  libxul.so!RuleCascadeData::RuleCascadeData(nsIAtom*, bool) [nsCSSRuleProcessor.cpp:a24bf8dce031 : 937 + 0xe]
19:24:34     INFO -     eip = 0xa20f7422   esp = 0xa9766040   ebp = 0xa9766088   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x95e13c00   edi = 0x95e13b6c
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO -  6  libxul.so!nsCSSRuleProcessor::RefreshRuleCascade(nsPresContext*) [nsCSSRuleProcessor.cpp:a24bf8dce031 : 3330 + 0x1c]
19:24:34     INFO -     eip = 0xa20f772f   esp = 0xa9766090   ebp = 0xa9766158   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x9f059260   edi = 0x96e32c00
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO -  7  libxul.so!nsCSSRuleProcessor::GetRuleCascade(nsPresContext*) [nsCSSRuleProcessor.cpp:a24bf8dce031 : 3299 + 0x8]
19:24:34     INFO -     eip = 0xa20f7b79   esp = 0xa9766160   ebp = 0xa9766178   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x970ad5e0   edi = 0x96e32c00
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO -  8  libxul.so!nsCSSRuleProcessor::RulesMatching(AnonBoxRuleProcessorData*) [nsCSSRuleProcessor.cpp:a24bf8dce031 : 2448 + 0x9]
19:24:34     INFO -     eip = 0xa20f81b0   esp = 0xa9766180   ebp = 0xa97661b8   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa97662b0   edi = 0x9573c700
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO -  9  libxul.so!EnumRulesMatching<AnonBoxRuleProcessorData> [nsStyleSet.cpp:a24bf8dce031 : 658 + 0x8]
19:24:34     INFO -     eip = 0xa213600a   esp = 0xa97661c0   ebp = 0xa97661d8   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa97662a4   edi = 0x9573c700
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 10  libxul.so!nsStyleSet::FileRules(bool (*)(nsIStyleRuleProcessor*, void*), RuleProcessorData*, mozilla::dom::Element*, nsRuleWalker*) [nsStyleSet.cpp:a24bf8dce031 : 970 + 0xa]
19:24:34     INFO -     eip = 0xa2144ff5   esp = 0xa97661e0   ebp = 0xa9766258   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa97662a4   edi = 0x9573c700
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 11  libxul.so!nsStyleSet::ResolveAnonymousBoxStyle(nsIAtom*, nsStyleContext*) [nsStyleSet.cpp:a24bf8dce031 : 1495 + 0x6]
19:24:34     INFO -     eip = 0xa214af24   esp = 0xa9766260   ebp = 0xa97662d8   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x9573c700   edi = 0xa97662a4
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 12  libxul.so!nsCSSFrameConstructor::ConstructRootFrame() [nsCSSFrameConstructor.cpp:a24bf8dce031 : 2517 + 0x9]
19:24:34     INFO -     eip = 0xa216eaa3   esp = 0xa97662e0   ebp = 0xa9766328   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x9573c700   edi = 0x9550be90
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 13  libxul.so!PresShell::Initialize(int, int) [nsPresShell.cpp:a24bf8dce031 : 1664 + 0x8]
19:24:34     INFO -     eip = 0xa21b16f1   esp = 0xa9766330   ebp = 0xa9766398   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x98566d20   edi = 0x9550be90
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 14  libxul.so!nsContentSink::StartLayout(bool) [nsContentSink.cpp:a24bf8dce031 : 1179 + 0xd]
19:24:34     INFO -     eip = 0xa1e0b941   esp = 0xa97663a0   ebp = 0xa97663e8   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x954757b0   edi = 0x0000bb80
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 15  libxul.so!nsHtml5TreeOpExecutor::StartLayout() [nsHtml5TreeOpExecutor.cpp:a24bf8dce031 : 696 + 0x9]
19:24:34     INFO -     eip = 0xa1cd10cf   esp = 0xa97663f0   ebp = 0xa9766408   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x954757b0   edi = 0x96e88c58
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 16  libxul.so!nsHtml5TreeOperation::Perform(nsHtml5TreeOpExecutor*, nsIContent**) [nsHtml5TreeOperation.cpp:a24bf8dce031 : 678 + 0xa]
19:24:34     INFO -     eip = 0xa1cdef88   esp = 0xa9766410   ebp = 0xa9766618   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x96e88c58   edi = 0x96e88c58
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 17  libxul.so!nsHtml5TreeOpExecutor::RunFlushLoop() [nsHtml5TreeOpExecutor.cpp:a24bf8dce031 : 523 + 0xb]
19:24:34     INFO -     eip = 0xa1cdfe5b   esp = 0xa9766620   ebp = 0xa9766678   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x954757b0   edi = 0x96e88c58
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 18  libxul.so!nsHtml5ExecutorFlusher::Run() [nsHtml5StreamParser.cpp:a24bf8dce031 : 131 + 0x8]
19:24:34     INFO -     eip = 0xa1ce0050   esp = 0xa9766680   ebp = 0xa9766698   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa9f3c4c0   edi = 0x00000000
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 19  libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:a24bf8dce031 : 610 + 0x5]
19:24:34     INFO -     eip = 0xa15a151c   esp = 0xa97666a0   ebp = 0xa9766708   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa9f3c4c0   edi = 0x00000000
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 20  libxul.so!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:a24bf8dce031 : 251 + 0x11]
19:24:34     INFO -     eip = 0xa1560694   esp = 0xa9766710   ebp = 0xa9766738   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x00000000   edi = 0xa9f6a0c0
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 21  libxul.so!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [MessagePump.cpp:a24bf8dce031 : 85 + 0xb]
19:24:34     INFO -     eip = 0xa1735643   esp = 0xa9766740   ebp = 0xa9766788   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa9f01fa0   edi = 0xa9f6a0c0
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 22  libxul.so!MessageLoop::RunInternal() [message_loop.cc:a24bf8dce031 : 220 + 0x6]
19:24:34     INFO -     eip = 0xa1720eb8   esp = 0xa9766790   ebp = 0xa97667a8   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa9f6a0c0   edi = 0xa9f3c4c0
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 23  libxul.so!MessageLoop::Run() [message_loop.cc:a24bf8dce031 : 213 + 0x7]
19:24:34     INFO -     eip = 0xa1720fe9   esp = 0xa97667b0   ebp = 0xa97667d8   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa9f6a0c0   edi = 0xa9f3c4c0
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 24  libxul.so!nsBaseAppShell::Run() [nsBaseAppShell.cpp:a24bf8dce031 : 161 + 0xd]
19:24:34     INFO -     eip = 0xa1c39267   esp = 0xa97667e0   ebp = 0xa9766808   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa9f4f700   edi = 0xa9f3c4c0
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 25  libxul.so!nsAppStartup::Run() [nsAppStartup.cpp:a24bf8dce031 : 267 + 0x5]
19:24:34     INFO -     eip = 0xa24fe7ff   esp = 0xa9766810   ebp = 0xa9766828   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0x9edb0670   edi = 0x9edb0670
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 26  libxul.so!XREMain::XRE_mainRun() [nsAppRunner.cpp:a24bf8dce031 : 3976 + 0x8]
19:24:34     INFO -     eip = 0xa24be418   esp = 0xa9766830   ebp = 0xa9766908   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa9766a8d   edi = 0x9edb0670
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 27  libxul.so!XREMain::XRE_main(int, char**, nsXREAppData const*) [nsAppRunner.cpp:a24bf8dce031 : 4044 + 0x8]
19:24:34     INFO -     eip = 0xa24bff76   esp = 0xa9766910   ebp = 0xa9766958   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa976699c   edi = 0x00000000
19:24:34     INFO -     Found by: call frame info
19:24:34     INFO - 28  libxul.so!XRE_main [nsAppRunner.cpp:a24bf8dce031 : 4246 + 0x5]
19:24:34     INFO -     eip = 0xa24c01a0   esp = 0xa9766960   ebp = 0xa9766a98   ebx = 0xa3808f9c
19:24:34     INFO -     esi = 0xa976699c   edi = 0xa9f10520
19:24:34     INFO -     Found by: call frame info
https://tbpl.mozilla.org/php/getParsedLog.php?id=30809877&tree=Cedar&full=1#error0

21:13:22     INFO - REFTEST INFO | Loading a blank page
21:13:22     INFO - REFTEST TEST-END | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-clear-test.html?aa&_____&preserve
21:13:22     INFO - REFTEST TEST-START | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-clear-test.html?__&alpha&preserve
21:13:22     INFO - RESTORE PREFERENCE pref(webgl.force-enabled,false)
21:13:22     INFO - SET PREFERENCE pref(webgl.force-enabled,true)
21:13:22     INFO - REFTEST TEST-LOAD | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-clear-test.html?__&alpha&preserve | 1089 / 3781 (28%)
21:13:22     INFO - 
21:13:22     INFO - INFO | automation.py | Application ran for: 0:06:22.481850
21:13:22     INFO - INFO | zombiecheck | Reading PID log: /tmp/tmpJEv4dnpidlog
21:13:22     INFO - mozcrash INFO | Downloading symbols from: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/cedar-android-x86/1384909322/fennec-28.0a1.en-US.android-i386.crashreporter-symbols.zip
21:13:22     INFO - /data/anr/traces.txt not found
21:13:22     INFO - PROCESS-CRASH | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-clear-test.html?__&alpha&preserve | application crashed [@ jemalloc_crash]
21:13:22     INFO - Crash dump filename: /tmp/tmp7A0Iu7/164d3a57-4f34-39e7-5c6a97c9-2c5d40e0.dmp
21:13:22     INFO - Operating system: Android
21:13:22     INFO -                   0.0.0 Linux 2.6.29 #2 PREEMPT Thu Nov 15 13:38:07 CST 2012 i686 generic_x86/sdk_x86/generic_x86:4.2/JOP40C/eng.android-build.20121231.103448:eng/test-keys
21:13:22     INFO - CPU: x86
21:13:22     INFO -      GenuineIntel family 6 model 3 stepping 3
21:13:22     INFO -      1 CPU
21:13:22     INFO - 
21:13:22     INFO - Crash reason:  SIGSEGV
21:13:22     INFO - Crash address: 0x0
21:13:22     INFO - 
21:13:22     INFO - Thread 12 (crashed)
21:13:22     INFO -  0  libmozglue.so!jemalloc_crash [jemalloc.c:a24bf8dce031 : 1572 + 0x0]
21:13:22     INFO -     eip = 0xaa33139c   esp = 0xa98d2d70   ebp = 0xa98d2d78   ebx = 0xaa372d4c
21:13:22     INFO -     esi = 0xaa208040   edi = 0xaa208668   eax = 0x98fc1000   ecx = 0x98f0091c
21:13:22     INFO -     edx = 0x00000000   efl = 0x00210282
21:13:22     INFO -     Found by: given as instruction pointer in context
21:13:22     INFO -  1  libmozglue.so!arena_malloc [jemalloc.c:a24bf8dce031 : 4194 + 0x4]
21:13:22     INFO -     eip = 0xaa3337ad   esp = 0xa98d2d80   ebp = 0xa98d2db8   ebx = 0xaa372d4c
21:13:22     INFO -     esi = 0xaa208040   edi = 0xaa208668
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO -  2  libmozglue.so!__wrap_malloc [jemalloc.c:a24bf8dce031 : 4211 + 0x10]
21:13:22     INFO -     eip = 0xaa333fe9   esp = 0xa98d2dc0   ebp = 0xa98d2dd8   ebx = 0xaa372d4c
21:13:22     INFO -     esi = 0x00000130   edi = 0x9615b6c0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO -  3  libxul.so!JSAutoStructuredCloneBuffer::copy(unsigned long long const*, unsigned int, unsigned int) [Utility.h:a24bf8dce031 : 144 + 0xa]
21:13:22     INFO -     eip = 0xa2aed14d   esp = 0xa98d2de0   ebp = 0xa98d2e18   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x9a597400   edi = 0x9615b6c0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO -  4  libxul.so!nsInProcessTabChildGlobal::DoSendAsyncMessage(JSContext*, nsAString_internal const&, mozilla::dom::StructuredCloneData const&, JS::Handle<JSObject*>, nsIPrincipal*) [nsInProcessTabChildGlobal.cpp:a24bf8dce031 : 69 + 0x14]
21:13:22     INFO -     eip = 0xa1fcf15b   esp = 0xa98d2e20   ebp = 0xa98d2e68   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x9a597400   edi = 0x9615b6c0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO -  5  libxul.so!nsFrameMessageManager::DispatchAsyncMessageInternal(JSContext*, nsAString_internal const&, mozilla::dom::StructuredCloneData const&, JS::Handle<JSObject*>, nsIPrincipal*) [nsFrameMessageManager.cpp:a24bf8dce031 : 614 + 0x16]
21:13:22     INFO -     eip = 0xa1fbcb86   esp = 0xa98d2e70   ebp = 0xa98d2ea8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x9a576dd0   edi = 0xa98d30a0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO -  6  libxul.so!nsFrameMessageManager::DispatchAsyncMessage(nsAString_internal const&, JS::Value const&, JS::Value const&, nsIPrincipal*, JSContext*, unsigned char) [nsFrameMessageManager.cpp:a24bf8dce031 : 645 + 0x10]
21:13:22     INFO -     eip = 0xa1fbf634   esp = 0xa98d2eb0   ebp = 0xa98d2f18   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xaa110f20   edi = 0xa98d30a0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO -  7  libxul.so!nsInProcessTabChildGlobal::SendAsyncMessage(nsAString_internal const&, JS::Value const&, JS::Value const&, nsIPrincipal*, JSContext*, unsigned char) [nsInProcessTabChildGlobal.h:a24bf8dce031 : 42 + 0x19]
21:13:22     INFO -     eip = 0xa1fce535   esp = 0xa98d2f20   ebp = 0xa98d2f48   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x00000004   edi = 0xa98d30b0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO -  8  libxul.so + 0x3919c7
21:13:22     INFO -     eip = 0xa17149c8   esp = 0xa98d2f50   ebp = 0xa98d2f98   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x00000004   edi = 0xa98d30b0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO -  9  libxul.so!XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) [XPCWrappedNative.cpp:a24bf8dce031 : 2593 + 0x15]
21:13:22     INFO -     eip = 0xa1df4de5   esp = 0xa98d2fa0   ebp = 0xa98d3128
21:13:22     INFO -     Found by: previous frame's frame pointer
21:13:22     INFO - 10  libxul.so!XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*) [XPCWrappedNativeJSOps.cpp:a24bf8dce031 : 1304 + 0x9]
21:13:22     INFO -     eip = 0xa1df7e59   esp = 0xa98d3130   ebp = 0xa98d31f8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xa98d3168   edi = 0x9b366d80
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 11  libxul.so!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [jscntxtinlines.h:a24bf8dce031 : 220 + 0x18]
21:13:22     INFO -     eip = 0xa2abb841   esp = 0xa98d3200   ebp = 0xa98d3438   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xa1df7ce4   edi = 0xa98d3244
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 12  libxul.so!Interpret [Interpreter.cpp:a24bf8dce031 : 2502 + 0x29]
21:13:22     INFO -     eip = 0xa2ab5e2b   esp = 0xa98d3440   ebp = 0xa98d39c8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xa3956b20   edi = 0x9f20e600
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 13  libxul.so!js::RunScript [Interpreter.cpp:a24bf8dce031 : 420 + 0x9]
21:13:22     INFO -     eip = 0xa2abb4ec   esp = 0xa98d39d0   ebp = 0xa98d3a38
21:13:22     INFO -     Found by: previous frame's frame pointer
21:13:22     INFO - 14  libxul.so!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [Interpreter.cpp:a24bf8dce031 : 389 + 0x10]
21:13:22     INFO -     eip = 0xa2abb764   esp = 0xa98d3a40   ebp = 0xa98d3c78   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xa98d3a84   edi = 0xaa110f20
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 15  libxul.so!js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) [Interpreter.cpp:a24bf8dce031 : 513 + 0x22]
21:13:22     INFO -     eip = 0xa2abd9ad   esp = 0xa98d3c80   ebp = 0xa98d3d38   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xaa110f20   edi = 0xaa110f20
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 16  libxul.so!JS_CallFunctionValue(JSContext*, JSObject*, JS::Value, unsigned int, JS::Value*, JS::Value*) [jsapi.cpp:a24bf8dce031 : 5032 + 0x29]
21:13:22     INFO -     eip = 0xa298c478   esp = 0xa98d3d40   ebp = 0xa98d3da8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xaa110f20   edi = 0xa98d4028
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 17  libxul.so!nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS*, unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) [XPCWrappedJSClass.cpp:a24bf8dce031 : 1425 + 0x2b]
21:13:22     INFO -     eip = 0xa1def7bf   esp = 0xa98d3db0   ebp = 0xa98d3f98   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x00000001   edi = 0xa98d4028
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 18  libxul.so!nsXPCWrappedJS::CallMethod(unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) [XPCWrappedJS.cpp:a24bf8dce031 : 505 + 0x13]
21:13:22     INFO -     eip = 0xa1dec1dd   esp = 0xa98d3fa0   ebp = 0xa98d3fd8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x9615b700   edi = 0x00000003
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 19  libxul.so!PrepareAndDispatch [xptcstubs_gcc_x86_unix.cpp:a24bf8dce031 : 60 + 0x1b]
21:13:22     INFO -     eip = 0xa17154d6   esp = 0xa98d3fe0   ebp = 0xa98d40b8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xa98d4020   edi = 0x9726be20
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 20  libxul.so!nsEventListenerManager::HandleEventSubType(nsListenerStruct*, nsIDOMEvent*, mozilla::dom::EventTarget*, nsCxPusher*) [nsEventListenerManager.cpp:a24bf8dce031 : 964 + 0x8]
21:13:22     INFO -     eip = 0xa2044714   esp = 0xa98d40c0   ebp = 0xa98d4118   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x9a597c40   edi = 0x9726be20
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 21  libxul.so!nsEventListenerManager::HandleEventInternal(nsPresContext*, mozilla::WidgetEvent*, nsIDOMEvent**, mozilla::dom::EventTarget*, nsEventStatus*, nsCxPusher*) [nsEventListenerManager.cpp:a24bf8dce031 : 1041 + 0x16]
21:13:22     INFO -     eip = 0xa2044902   esp = 0xa98d4120   ebp = 0xa98d4198   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x97b01510   edi = 0x9a597c40
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 22  libxul.so!nsEventTargetChainItem::HandleEvent(nsEventChainPostVisitor&, ELMCreationDetector&, nsCxPusher*) [nsEventListenerManager.h:a24bf8dce031 : 326 + 0xe]
21:13:22     INFO -     eip = 0xa20401f2   esp = 0xa98d41a0   ebp = 0xa98d41e8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xa98d4298   edi = 0x00000000
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 23  libxul.so!nsEventTargetChainItem::HandleEventTargetChain(nsTArray<nsEventTargetChainItem>&, nsEventChainPostVisitor&, nsDispatchingCallback*, ELMCreationDetector&, nsCxPusher*) [nsEventDispatcher.cpp:a24bf8dce031 : 313 + 0xc]
21:13:22     INFO -     eip = 0xa20405aa   esp = 0xa98d41f0   ebp = 0xa98d4238   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xa98d4290   edi = 0x9b90b01c
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 24  libxul.so!nsEventDispatcher::Dispatch(nsISupports*, nsPresContext*, mozilla::WidgetEvent*, nsIDOMEvent*, nsEventStatus*, nsDispatchingCallback*, nsCOMArray<mozilla::dom::EventTarget>*) [nsEventDispatcher.cpp:a24bf8dce031 : 609 + 0x5]
21:13:22     INFO -     eip = 0xa20417d0   esp = 0xa98d4240   ebp = 0xa98d4308   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x00000000   edi = 0x00000000
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 25  libxul.so!nsEventDispatcher::DispatchDOMEvent(nsISupports*, mozilla::WidgetEvent*, nsIDOMEvent*, nsPresContext*, nsEventStatus*) [nsEventDispatcher.cpp:a24bf8dce031 : 676 + 0x15]
21:13:22     INFO -     eip = 0xa2041ace   esp = 0xa98d4310   ebp = 0xa98d4368   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x97b01510   edi = 0x9726be20
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 26  libxul.so!nsPresContext::FireDOMPaintEvent(nsInvalidateRequestList*) [nsPresContext.cpp:a24bf8dce031 : 2175 + 0x14]
21:13:22     INFO -     eip = 0xa230e560   esp = 0xa98d4370   ebp = 0xa98d43b8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x96f1a000   edi = 0x9726be20
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 27  libxul.so!DelayedFireDOMPaintEvent::Run() [nsPresContext.cpp:a24bf8dce031 : 2382 + 0xb]
21:13:22     INFO -     eip = 0xa230e5ce   esp = 0xa98d43c0   ebp = 0xa98d43d8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x00000000   edi = 0x00000002
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 28  libxul.so!nsContentUtils::RemoveScriptBlocker() [nsContentUtils.cpp:a24bf8dce031 : 4787 + 0x8]
21:13:22     INFO -     eip = 0xa1f85afd   esp = 0xa98d43e0   ebp = 0xa98d4428   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x00000000   edi = 0x00000002
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 29  libxul.so!nsViewManager::ProcessPendingUpdatesForView(nsView*, bool) [nsContentUtils.h:a24bf8dce031 : 2201 + 0x4]
21:13:22     INFO -     eip = 0xa1f56ae3   esp = 0xa98d4430   ebp = 0xa98d4498   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x9b9b7100   edi = 0x9b9902e0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 30  libxul.so!nsViewManager::ProcessPendingUpdates() [nsViewManager.cpp:a24bf8dce031 : 1053 + 0xb]
21:13:22     INFO -     eip = 0xa1f56b9a   esp = 0xa98d44a0   ebp = 0xa98d44b8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x9b9b7100   edi = 0x9b9b7100
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 31  libxul.so!nsRefreshDriver::Tick(long long, mozilla::TimeStamp) [nsRefreshDriver.cpp:a24bf8dce031 : 1215 + 0x8]
21:13:22     INFO -     eip = 0xa22cc77d   esp = 0xa98d44c0   ebp = 0xa98d45b8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x9b480a00   edi = 0x9b9b7100
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 32  libxul.so!mozilla::RefreshDriverTimer::Tick() [nsRefreshDriver.cpp:a24bf8dce031 : 168 + 0x13]
21:13:22     INFO -     eip = 0xa22ccea2   esp = 0xa98d45c0   ebp = 0xa98d4628   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x948630b5   edi = 0x00000080
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 33  libxul.so!nsTimerImpl::Fire() [nsTimerImpl.cpp:a24bf8dce031 : 546 + 0x7]
21:13:22     INFO -     eip = 0xa1711d2e   esp = 0xa98d4630   ebp = 0xa98d4678   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x9c56b8b0   edi = 0xa22ccec7
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 34  libxul.so!nsTimerEvent::Run() [nsTimerImpl.cpp:a24bf8dce031 : 630 + 0x8]
21:13:22     INFO -     eip = 0xa1711e00   esp = 0xa98d4680   ebp = 0xa98d4698   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xaa13c4c0   edi = 0x00000000
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 35  libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:a24bf8dce031 : 610 + 0x5]
21:13:22     INFO -     eip = 0xa170f51c   esp = 0xa98d46a0   ebp = 0xa98d4708   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xaa13c4c0   edi = 0x00000000
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 36  libxul.so!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:a24bf8dce031 : 251 + 0x11]
21:13:22     INFO -     eip = 0xa16ce694   esp = 0xa98d4710   ebp = 0xa98d4738   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x00000000   edi = 0xaa16a0c0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 37  libxul.so!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [MessagePump.cpp:a24bf8dce031 : 85 + 0xb]
21:13:22     INFO -     eip = 0xa18a3643   esp = 0xa98d4740   ebp = 0xa98d4788   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xaa101fa0   edi = 0xaa16a0c0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 38  libxul.so!MessageLoop::RunInternal() [message_loop.cc:a24bf8dce031 : 220 + 0x6]
21:13:22     INFO -     eip = 0xa188eeb8   esp = 0xa98d4790   ebp = 0xa98d47a8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xaa16a0c0   edi = 0xaa13c4c0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 39  libxul.so!MessageLoop::Run() [message_loop.cc:a24bf8dce031 : 213 + 0x7]
21:13:22     INFO -     eip = 0xa188efe9   esp = 0xa98d47b0   ebp = 0xa98d47d8   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xaa16a0c0   edi = 0xaa13c4c0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 40  libxul.so!nsBaseAppShell::Run() [nsBaseAppShell.cpp:a24bf8dce031 : 161 + 0xd]
21:13:22     INFO -     eip = 0xa1da7267   esp = 0xa98d47e0   ebp = 0xa98d4808   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xaa14f700   edi = 0xaa13c4c0
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 41  libxul.so!nsAppStartup::Run() [nsAppStartup.cpp:a24bf8dce031 : 267 + 0x5]
21:13:22     INFO -     eip = 0xa266c7ff   esp = 0xa98d4810   ebp = 0xa98d4828   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0x9efb0670   edi = 0x9efb0670
21:13:22     INFO -     Found by: call frame info
21:13:22     INFO - 42  libxul.so!XREMain::XRE_mainRun() [nsAppRunner.cpp:a24bf8dce031 : 3976 + 0x8]
21:13:22     INFO -     eip = 0xa262c418   esp = 0xa98d4830   ebp = 0xa98d4908   ebx = 0xa3976f9c
21:13:22     INFO -     esi = 0xa98d4a8d   edi = 0x9efb0670
21:13:22     INFO -     Found by: call frame info
https://tbpl.mozilla.org/php/getParsedLog.php?id=30811220&tree=Cedar&full=1

21:50:52     INFO - REFTEST TEST-LOAD | http://10.0.2.2:8856/tests/content/canvas/test/reftest/wrapper.html?colors.png | 1205 / 3781 (31%)
21:50:52     INFO - REFTEST TEST-PASS | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-color-test.html?16bpp | image comparison (==)
21:50:52     INFO - REFTEST INFO | Loading a blank page
21:50:52     INFO - REFTEST TEST-END | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-color-test.html?16bpp
21:50:52     INFO - REFTEST TEST-START | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-color-test.html?16bpp&readback
21:50:52     INFO - RESTORE PREFERENCE pref(webgl.prefer-16bpp,false)
21:50:52     INFO - RESTORE PREFERENCE pref(webgl.force-enabled,false)
21:50:52     INFO - SET PREFERENCE pref(webgl.force-enabled,true)
21:50:52     INFO - SET PREFERENCE pref(webgl.prefer-16bpp,true)
21:50:52     INFO - SET PREFERENCE pref(webgl.force-layers-readback,true)
21:50:52     INFO - REFTEST TEST-LOAD | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-color-test.html?16bpp&readback | 1206 / 3781 (31%)
21:50:52     INFO - INFO | automation.py | Application ran for: 0:06:42.468555
21:50:52     INFO - INFO | zombiecheck | Reading PID log: /tmp/tmpuCjtFKpidlog
21:50:52     INFO - mozcrash INFO | Downloading symbols from: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/cedar-android-x86/1384909322/fennec-28.0a1.en-US.android-i386.crashreporter-symbols.zip
21:50:52     INFO - /data/anr/traces.txt not found
21:50:52     INFO - PROCESS-CRASH | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-color-test.html?16bpp&readback | application crashed [@ jemalloc_crash]
21:50:52     INFO - Crash dump filename: /tmp/tmp8MzKjS/4fab8bfd-36e2-32fd-61196a27-2e7546e6.dmp
21:50:52     INFO - Operating system: Android
21:50:52     INFO -                   0.0.0 Linux 2.6.29 #2 PREEMPT Thu Nov 15 13:38:07 CST 2012 i686 generic_x86/sdk_x86/generic_x86:4.2/JOP40C/eng.android-build.20121231.103448:eng/test-keys
21:50:52     INFO - CPU: x86
21:50:52     INFO -      GenuineIntel family 6 model 3 stepping 3
21:50:52     INFO -      1 CPU
21:50:52     INFO - 
21:50:52     INFO - Crash reason:  SIGSEGV
21:50:52     INFO - Crash address: 0x0
21:50:52     INFO - 
21:50:52     INFO - Thread 12 (crashed)
21:50:52     INFO -  0  libmozglue.so!jemalloc_crash [jemalloc.c:a24bf8dce031 : 1572 + 0x0]
21:50:52     INFO -     eip = 0xaa2b539c   esp = 0xa9856380   ebp = 0xa9856388   ebx = 0xaa2f6d4c
21:50:52     INFO -     esi = 0x966fb000   edi = 0x966fed40   eax = 0x00000000   ecx = 0x00000000
21:50:52     INFO -     edx = 0x96600bf8   efl = 0x00210282
21:50:52     INFO -     Found by: given as instruction pointer in context
21:50:52     INFO -  1  libmozglue.so!arena_dalloc [jemalloc.c:a24bf8dce031 : 4627 + 0x4]
21:50:52     INFO -     eip = 0xaa2b6c5f   esp = 0xa9856390   ebp = 0xa98563c8   ebx = 0xaa2f6d4c
21:50:52     INFO -     esi = 0x966fb000   edi = 0x966fed40
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO -  2  libxul.so!PL_DHashFreeTable [pldhash.cpp:a24bf8dce031 : 78 + 0x7]
21:50:52     INFO -     eip = 0xa165154b   esp = 0xa98563d0   ebp = 0xa98563e8   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x96166140   edi = 0x966fee80
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO -  3  libxul.so!PL_DHashTableFinish [pldhash.cpp:a24bf8dce031 : 311 + 0xa]
21:50:52     INFO -     eip = 0xa1651755   esp = 0xa98563f0   ebp = 0xa9856418   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x96166140   edi = 0x966fee80
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO -  4  libxul.so!mozilla::MaskLayerImageCache::~MaskLayerImageCache() + 0x21
21:50:52     INFO -     eip = 0xa165e3dc   esp = 0xa9856420   ebp = 0xa9856438   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x96166100   edi = 0x00000000
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO -  5  libxul.so!mozilla::css::Loader::Sheets::~Sheets() [nsBaseHashtable.h:a24bf8dce031 : 52 + 0x8]
21:50:52     INFO -     eip = 0xa21f3c92   esp = 0xa9856440   ebp = 0xa9856458   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x96166100   edi = 0x00000000
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO -  6  libxul.so!mozilla::css::Loader::~Loader() [nsAutoPtr.h:a24bf8dce031 : 77 + 0x12]
21:50:52     INFO -     eip = 0xa21fa19b   esp = 0xa9856460   ebp = 0xa9856478   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x96166100   edi = 0x00000000
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO -  7  libxul.so!mozilla::css::Loader::Release() [Loader.h:a24bf8dce031 : 136 + 0x5]
21:50:52     INFO -     eip = 0xa1efb3ca   esp = 0xa9856480   ebp = 0xa98564a8   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x95724e80   edi = 0x00000000
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO -  8  libxul.so!nsIDocument::~nsIDocument() [nsAutoPtr.h:a24bf8dce031 : 887 + 0x8]
21:50:52     INFO -     eip = 0xa1f247ca   esp = 0xa98564b0   ebp = 0xa98564d8   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x96265800   edi = 0x00000000
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO -  9  libxul.so!nsDocument::~nsDocument() [nsDocument.cpp:a24bf8dce031 : 1459 + 0x7]
21:50:52     INFO -     eip = 0xa1f27da6   esp = 0xa98564e0   ebp = 0xa9856528   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x96265800   edi = 0x962659f0
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 10  libxul.so!nsHTMLDocument::~nsHTMLDocument() [nsHTMLDocument.cpp:a24bf8dce031 : 199 + 0x78]
21:50:52     INFO -     eip = 0xa2038135   esp = 0xa9856530   ebp = 0xa9856548   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x96265800   edi = 0xa985660c
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 11  libxul.so!nsHTMLDocument::~nsHTMLDocument() [nsHTMLDocument.cpp:a24bf8dce031 : 201 + 0x5]
21:50:52     INFO -     eip = 0xa203815c   esp = 0xa9856550   ebp = 0xa9856568   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x96265800   edi = 0xa985660c
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 12  libxul.so!InMemoryDataSource::DeleteCycleCollectable() [nsInMemoryDataSource.cpp:a24bf8dce031 : 859 + 0x8]
21:50:52     INFO -     eip = 0xa191817b   esp = 0xa9856570   ebp = 0xa9856588   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x0000003f   edi = 0xa985660c
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 13  libxul.so!nsDocument::cycleCollection::DeleteCycleCollectable(void*) [nsDocument.h:a24bf8dce031 : 864 + 0xb]
21:50:52     INFO -     eip = 0xa1f0f4cf   esp = 0xa9856590   ebp = 0xa98565a8   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x0000003f   edi = 0xa985660c
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 14  libxul.so!SnowWhiteKiller::~SnowWhiteKiller() [nsCycleCollector.cpp:a24bf8dce031 : 2011 + 0xc]
21:50:52     INFO -     eip = 0xa1662ff4   esp = 0xa98565b0   ebp = 0xa98565d8   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x0000003f   edi = 0xa985660c
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 15  libxul.so!nsCycleCollector::FreeSnowWhite(bool) [nsCycleCollector.cpp:a24bf8dce031 : 2120 + 0x8]
21:50:52     INFO -     eip = 0xa1663090   esp = 0xa98565e0   ebp = 0xa9856618   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0xaa0a7054   edi = 0xa985660c
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 16  libxul.so!nsCycleCollector_doDeferredDeletion() [nsCycleCollector.cpp:a24bf8dce031 : 3158 + 0xa]
21:50:52     INFO -     eip = 0xa16630de   esp = 0xa9856620   ebp = 0xa9856638   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x9f10c0b0   edi = 0xa9856668
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 17  libxul.so!AsyncFreeSnowWhite::Run() [XPCJSRuntime.cpp:a24bf8dce031 : 205 + 0x4]
21:50:52     INFO -     eip = 0xa1d5bcd3   esp = 0xa9856640   ebp = 0xa9856698   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x9f10c0b0   edi = 0xa9856668
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 18  libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:a24bf8dce031 : 610 + 0x5]
21:50:52     INFO -     eip = 0xa169151c   esp = 0xa98566a0   ebp = 0xa9856708   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0xaa03c4c0   edi = 0x00000000
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 19  libxul.so!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:a24bf8dce031 : 251 + 0x11]
21:50:52     INFO -     eip = 0xa1650694   esp = 0xa9856710   ebp = 0xa9856738   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x00000000   edi = 0xaa06a0c0
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 20  libxul.so!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [MessagePump.cpp:a24bf8dce031 : 85 + 0xb]
21:50:52     INFO -     eip = 0xa1825643   esp = 0xa9856740   ebp = 0xa9856788   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0xaa001fa0   edi = 0xaa06a0c0
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 21  libxul.so!MessageLoop::RunInternal() [message_loop.cc:a24bf8dce031 : 220 + 0x6]
21:50:52     INFO -     eip = 0xa1810eb8   esp = 0xa9856790   ebp = 0xa98567a8   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0xaa06a0c0   edi = 0xaa03c4c0
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 22  libxul.so!MessageLoop::Run() [message_loop.cc:a24bf8dce031 : 213 + 0x7]
21:50:52     INFO -     eip = 0xa1810fe9   esp = 0xa98567b0   ebp = 0xa98567d8   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0xaa06a0c0   edi = 0xaa03c4c0
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 23  libxul.so!nsBaseAppShell::Run() [nsBaseAppShell.cpp:a24bf8dce031 : 161 + 0xd]
21:50:52     INFO -     eip = 0xa1d29267   esp = 0xa98567e0   ebp = 0xa9856808   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0xaa04f700   edi = 0xaa03c4c0
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 24  libxul.so!nsAppStartup::Run() [nsAppStartup.cpp:a24bf8dce031 : 267 + 0x5]
21:50:52     INFO -     eip = 0xa25ee7ff   esp = 0xa9856810   ebp = 0xa9856828   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0x9eeb0670   edi = 0x9eeb0670
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 25  libxul.so!XREMain::XRE_mainRun() [nsAppRunner.cpp:a24bf8dce031 : 3976 + 0x8]
21:50:52     INFO -     eip = 0xa25ae418   esp = 0xa9856830   ebp = 0xa9856908   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0xa9856a8d   edi = 0x9eeb0670
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 26  libxul.so!XREMain::XRE_main(int, char**, nsXREAppData const*) [nsAppRunner.cpp:a24bf8dce031 : 4044 + 0x8]
21:50:52     INFO -     eip = 0xa25aff76   esp = 0xa9856910   ebp = 0xa9856958   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0xa985699c   edi = 0x00000000
21:50:52     INFO -     Found by: call frame info
21:50:52     INFO - 27  libxul.so!XRE_main [nsAppRunner.cpp:a24bf8dce031 : 4246 + 0x5]
21:50:52     INFO -     eip = 0xa25b01a0   esp = 0xa9856960   ebp = 0xa9856a98   ebx = 0xa38f8f9c
21:50:52     INFO -     esi = 0xa985699c   edi = 0xaa010520
21:50:52     INFO -     Found by: call frame info
Blocks: 936226
No longer blocks: 891959
I tried skipping individual tests, but found that most/all of the canvas reftests need to be skipped to get a clean run. (Android&&AndroidVersion>=17) is the "standard" condition we are using for problem tests in this environment.
Attachment #8340540 - Flags: review?(bjacob)
Attachment #8340540 - Flags: review?(bjacob) → review?(jgilbert)
This really doesn't look like a WebGL bug. Something in the (extremely basic) WebGL reftests is causing these crashes, but none of these traces has anything relating to WebGL in it. 

In general, these tests are actually quite reliable on Android, but we don't have a way to mark them as failing for our ARMv6 slaves. I don't remember what the deal is for B2G, but I think it's also a problem with different slaves.

Really, we need to classify our test systems better, because I can guarantee that if you ran these tests on basically any Android 4.2 device, they will pass. 

Let's try to narrow this down a little bit more here, before we start down the path of disabling our way to victory.
Which of the following crashes, and which are fine:

                           == webgl-disable-test.html?nogl  wrapper.html?green.png
pref(webgl.disabled,true)  == webgl-disable-test.html       wrapper.html?green.png
== webgl-clear-test.html?__&_____&________  wrapper.html?green.png
== wrapper.html?green.png  wrapper.html?green.png
== webgl-clear-test.html?__&_____&________  webgl-clear-test.html?__&_____&________
Flags: needinfo?(gbrown)
Comment on attachment 8340540 [details] [diff] [review]
skip all canvas reftests on Android 4.2 x86 emu

Review of attachment 8340540 [details] [diff] [review]:
-----------------------------------------------------------------

Going to r- this for now. Rather than doing this, we should detect that we're running this particular config, and only disable it for this config. For instance, this will break as soon as we add new android device tests.
Attachment #8340540 - Flags: review?(jgilbert) → review-
(In reply to Jeff Gilbert [:jgilbert] from comment #5)
> Which of the following crashes, and which are fine:

Since the crashes are intermittent, and infrequent, a definitive answer may be elusive. In 10 runs, none of those failed. I'll run more tests today.
(In reply to Jeff Gilbert [:jgilbert] from comment #6)
> Going to r- this for now. Rather than doing this, we should detect that
> we're running this particular config, and only disable it for this config.

There have been a few discussions over the course of the x86 emulator project about what criteria should be used for failing or skipping tests. We considered adding support to detect running on x86, but overall felt that AndroidVersion would suffice, and would be - in the absence of determination of root cause of any given failure - just as likely to be the correct differentiating factor as any other (processor, graphics support, emulator/device, debug/opt, ...). AFAIK we have no way for a test harness to detect that it is running in a particular tbpl config. (Is that what you had in mind? skip-if(RunningOnTbpl&&TbplConfig=="Android 4.2 x86 Opt")?)

From the continuous integration/tbpl perspective, each of our current test environments runs a distinct version of Android and there are no plans to have more than one tbpl test configuration for the same Android version. Using AndroidVersion to skip/fail tests for a particular tbpl config means that some tests will be unnecessarily skipped and some will unexpectedly pass on some devices, but since we don't regularly test on all devices, I wouldn't expect a clean run anyway.

We settled on AndroidVersion>=17 at a time when it was unclear if the x86 emulator would be running 17 or 18. I'm happy to change that to AndroidVersion==17, if that helps.

> For instance, this will break as soon as we add new android device tests.

I'm not sure I understand "break". We may skip tests unnecessarily, but otherwise...?
(In reply to Geoff Brown [:gbrown] from comment #7)
> (In reply to Jeff Gilbert [:jgilbert] from comment #5)
> > Which of the following crashes, and which are fine:
> 
> Since the crashes are intermittent, and infrequent, a definitive answer may
> be elusive. In 10 runs, none of those failed. I'll run more tests today.

In 100 runs, none of these tests failed.

(However in 3 cases, reftest-2 failed sometime after running those tests: the webgl tests passed, dozens of additional tests ran and passed, and then the harness lost connectivity to the device. I don't know if this is related to the webgl tests -- I suspect not.)
Flags: needinfo?(gbrown)
A few more examples:

https://tbpl.mozilla.org/php/getParsedLog.php?id=31599646&full=1&branch=cedar#error0
https://tbpl.mozilla.org/php/getParsedLog.php?id=31599739&full=1&branch=cedar#error0
https://tbpl.mozilla.org/php/getParsedLog.php?id=31599588&full=1&branch=cedar#error0
https://tbpl.mozilla.org/php/getParsedLog.php?id=31597277&full=1&branch=cedar#error0
https://tbpl.mozilla.org/php/getParsedLog.php?id=31594742&full=1&branch=cedar#error0

I don't see much commonality between the crash stacks (other than jemalloc_crash), but all of the logcats have a similar Javascript Warning just before the crash:

18:24:46     INFO -  12-07 02:23:45.159 E/GeckoConsole( 1900): [JavaScript Warning: "Error: WebGL: Exceeded 2 live WebGL contexts for this principal, losing the least recently used one." {file: "http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-utils.js" line: 59}]
18:24:46     INFO -  12-07 02:23:45.489 E/eglCodecCommon( 1900): glUtilsParamSize: unknow param 0x00000bd0
18:24:46     INFO -  12-07 02:23:46.699 E/GeckoConsole( 1900): [JavaScript Warning: "Error: WebGL: Exceeded 2 live WebGL contexts for this principal, losing the least recently used one." {file: "http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-utils.js" line: 59}]
18:24:46     INFO -  12-07 02:23:47.109 E/eglCodecCommon( 1900): glUtilsParamSize: unknow param 0x00000bd0
18:24:46     INFO -  12-07 02:23:48.339 E/GeckoConsole( 1900): [JavaScript Warning: "Error: WebGL: Exceeded 2 live WebGL contexts for this principal, losing the least recently used one." {file: "http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-utils.js" line: 59}]
18:24:46     INFO -  12-07 02:23:48.719 E/eglCodecCommon( 1900): glUtilsParamSize: unknow param 0x00000bd0
18:24:46     INFO -  12-07 02:23:50.099 E/GeckoConsole( 1900): [JavaScript Warning: "Error: WebGL: Exceeded 2 live WebGL contexts for this principal, losing the least recently used one." {file: "http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-utils.js" line: 59}]
18:24:46     INFO -  12-07 02:23:50.179 F/libc    ( 1900): Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1), thread 1919 (Gecko)
18:24:46     INFO -  12-07 02:23:50.299 I/DEBUG   (  807): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***

Might that be significant?
No, it's fully expected. We actively 'lose' the oldest-used WebGL context when we create a new one, past a certain number of contexts. On mobile, that limit is 2 per principal. And in the test suite, a few hundreds of WebGL contexts are created, all on the same principal.

'Losing' a WebGL context means discarding all its graphics resources. This frees memory, so it shouldn't contribute to the problem.
jgilbert - Can you suggest a way forward here? 

If I can disable these tests, I can demonstrate passing reftest jobs on Cedar and get the Android x86 tests running on trunk trees. At that point, we can investigate the crashes further with try pushes. As long as the reftests are enabled and crashing, we cannot run Android x86 reftests outside of Cedar/Ash.

For the skip-if criteria, please see Comment 8.
Flags: needinfo?(jgilbert)
(In reply to Geoff Brown [:gbrown] from comment #12)
> jgilbert - Can you suggest a way forward here? 
> 
> If I can disable these tests, I can demonstrate passing reftest jobs on
> Cedar and get the Android x86 tests running on trunk trees. At that point,
> we can investigate the crashes further with try pushes. As long as the
> reftests are enabled and crashing, we cannot run Android x86 reftests
> outside of Cedar/Ash.
> 
> For the skip-if criteria, please see Comment 8.

We should absolutely skip-if based on TBPL configs, if we have no reason to suspect that it's an android-version-specific issue. Our skip-if/fail-if isn't just for TBPL. It's also a general assertion of what we think is true. We *don't* think that reftests are crasy on Android. They appear to be crashy on the emulator. (This seems like a pretty big bug for webgl) Thus, we should only disable them for the emulator. If I run this on my phone, I'm almost positive it'll work fine.

Skipping tests is awful, because realistically, no one tries to re-enable skipped tests, because there's always other work to do. Effectively, it'll just never get run again. We have these tests for a good reason, and we want to run them on every machine we can manage it. 

If we need to, we *can* skip it, but we should skip the fewest tests on the fewest platforms that we can manage.

To reiterate, these are comically simple tests. Some of them are literally 'clear the canvas to green, and make sure it looks green'. That they crash means that either WebGL or the reftest harness are comically broken on your platform. If it's the former, we should just disable WebGL for the emulator, but that makes the emulator that much worse at doing its job.

I also hadn't realized before, but the push you linked to only has an OPT build run, no DEBUG run. DEBUG runs are so ridiculously more important, because we actually get assert coverage. It's entirely possible that the runs here would gives us an easy-to-identify assert if run in DEBUG mode, since we assert a lot of things. Don't add OPT without adding DEBUG.

If we have to pick just one, only do DEBUG builds!

If the emulator is simple to compile and setup, I should be able to get to this at some point, though it's got a lot of competition for priority. It's quicker if you try a DEBUG build, and run with an env var of MOZ_GL_DEBUG=1, and see if the crash stacks improve.

(In reply to Geoff Brown [:gbrown] from comment #9)
> (In reply to Geoff Brown [:gbrown] from comment #7)
> > (In reply to Jeff Gilbert [:jgilbert] from comment #5)
> > > Which of the following crashes, and which are fine:
> > 
> > Since the crashes are intermittent, and infrequent, a definitive answer may
> > be elusive. In 10 runs, none of those failed. I'll run more tests today.
> 
> In 100 runs, none of these tests failed.
> 
> (However in 3 cases, reftest-2 failed sometime after running those tests:
> the webgl tests passed, dozens of additional tests ran and passed, and then
> the harness lost connectivity to the device. I don't know if this is related
> to the webgl tests -- I suspect not.)

Awesome, let's keep these enabled. Something more subtle is happening here. Let's try DEBUG builds with MOZ_GL_DEBUG.

I'm willing to r+ skipping some tests, but I'm not ready to scatter-shot-disable our way to victory.
Flags: needinfo?(jgilbert)
I ran the tests on a DEBUG build with MOZ_GL_DEBUG defined over the weekend. The frequency of intermittent crashes was significantly reduced compared to release builds, but I finally got 2 crashes, both of which were in eglCreatePbufferSurface -- like bug 948406. 

I re-tested on Cedar with the patch from bug 948406 applied. The frequency of crashing seems to be reduced:

https://tbpl.mozilla.org/?tree=Cedar&showall=1&rev=f392dbd98567

but still occurs:

https://tbpl.mozilla.org/php/getParsedLog.php?id=32042649&tree=Cedar

Also, sometimes webgl-color-test fails without crashing:

https://tbpl.mozilla.org/php/getParsedLog.php?id=32043049&tree=Cedar

13:17:43     INFO -  SET PREFERENCE pref(webgl.force-enabled,true)
13:17:43     INFO -  SET PREFERENCE pref(webgl.prefer-16bpp,true)
13:17:43     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8856/tests/content/canvas/test/reftest/wrapper.html?colors.png | 1269 / 3832 (33%)
13:17:43     INFO -  REFTEST INFO | START http://10.0.2.2:8856/tests/content/canvas/test/reftest/wrapper.html?colors.png
13:17:43     INFO -  REFTEST INFO | [CONTENT] OnDocumentLoad triggering AfterOnLoadScripts
13:17:43     INFO -  REFTEST INFO | Initializing canvas snapshot
13:17:43     INFO -  REFTEST INFO | DoDrawWindow 0,0,800,1000
13:17:43     INFO -  REFTEST INFO | [CONTENT] RecordResult fired
13:17:43     INFO -  REFTEST INFO | RecordResult fired
13:17:43     INFO -  REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8856/tests/content/canvas/test/reftest/webgl-color-test.html?16bpp | image comparison (==), max difference: 255, number of differing pixels: 1
(In reply to Jeff Gilbert [:jgilbert] from comment #13)
> I also hadn't realized before, but the push you linked to only has an OPT
> build run, no DEBUG run. DEBUG runs are so ridiculously more important,
> because we actually get assert coverage. It's entirely possible that the
> runs here would gives us an easy-to-identify assert if run in DEBUG mode,
> since we assert a lot of things. Don't add OPT without adding DEBUG.
> 
> If we have to pick just one, only do DEBUG builds!

We run Opt tests because we ship Opt. The scope of my project is Android x86 Opt. I understand the sentiment here, but it is unlikely that an attempt will be made to run any Android Debug configuration on tbpl until bug 866795 (or the more focused bug 940068) is resolved.
 
> If the emulator is simple to compile and setup, I should be able to get to
> this at some point, though it's got a lot of competition for priority. It's
> quicker if you try a DEBUG build, and run with an env var of MOZ_GL_DEBUG=1,
> and see if the crash stacks improve.

I would not recommend trying to reproduce these results on a local computer -- better to do that on the same hardware that is demonstrating the problem. Currently you would need to work on Cedar (but only Opt builds are available there), or request a loan of an ix machine.

I will try to reproduce the crash with a new Debug build, but given my experience so far, I am not optimistic.
Whiteboard: webgl-internal
Blocks: 967704
See Also: → 1000538
No longer blocks: 967704
(In reply to Geoff Brown [:gbrown] from comment #15)
> I will try to reproduce the crash with a new Debug build, but given my
> experience so far, I am not optimistic.

I tried a couple of times, but could not reproduce the crash. 

Tests for Android 2.3 became a higher priority, so I have not been working on the Android x86 tests lately. These crashes have persisted though, and can be seen on Ash and Cedar.

As luck would have it, we can now see that there is a very similar crash affecting the same tests on Android 2.3 -- bug 1000538!
See https://bugzilla.mozilla.org/show_bug.cgi?id=1000538#c13.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: