1113714 - Intermittent test_decodeMultichannel.html | application crashed [@ moz_speex_inner_product_single]

Status: RESOLVED INCOMPLETE

(Core :: Web Audio, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Related to bug 1101298?

08:21:18 INFO - 1014 INFO TEST-START | dom/media/webaudio/test/test_decodeMultichannel.html
08:21:18 INFO - 1015 INFO TEST-OK | dom/media/webaudio/test/test_decodeMultichannel.html | took 795ms
08:21:18 INFO - INFO | automation.py | Application ran for: 0:28:10.432283
08:21:18 INFO - INFO | zombiecheck | Reading PID log: /tmp/tmp04E8Llpidlog
08:21:18 INFO - Contents of /data/anr/traces.txt:
08:21:19 INFO - /data/tombstones does not exist; tombstone check skipped
08:21:19 INFO - mozcrash Downloading symbols from: https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-11-debug/1419001666/fennec-37.0a1.en-US.android-arm.crashreporter-symbols.zip
08:21:25 INFO - mozcrash Saved minidump as /builds/panda-0283/test/build/blobber_upload_dir/4f94e50a-cf6b-3b52-0cc55458-1e3c03c3.dmp
08:21:25 INFO - mozcrash Saved app info as /builds/panda-0283/test/build/blobber_upload_dir/4f94e50a-cf6b-3b52-0cc55458-1e3c03c3.extra
08:21:25 WARNING - PROCESS-CRASH | dom/media/webaudio/test/test_decodeMultichannel.html | application crashed [@ moz_speex_inner_product_single]
08:21:25 INFO - Crash dump filename: /tmp/tmpJtWOkZ/4f94e50a-cf6b-3b52-0cc55458-1e3c03c3.dmp
08:21:25 INFO - Operating system: Android
08:21:25 INFO - 0.0.0 Linux 3.2.0+ #2 SMP PREEMPT Thu Nov 29 08:06:57 EST 2012 armv7l pandaboard/pandaboard/pandaboard:4.0.4/IMM76I/5:eng/test-keys
08:21:25 INFO - CPU: arm
08:21:25 INFO - 2 CPUs
08:21:25 INFO - Crash reason: SIGILL
08:21:25 INFO - Crash address: 0x63372f58
08:21:25 INFO - Thread 66 (crashed)
08:21:25 INFO - 0 libxul.so!moz_speex_inner_product_single [resample_neon.c:f078e3258ffd : 124 + 0x0]
08:21:25 INFO - r4 = 0x6f3eadc0 r5 = 0x00000012 r6 = 0x0000003e r7 = 0x633721d9
08:21:25 INFO - r8 = 0x00000000 r9 = 0x00000032 r10 = 0x00000064 fp = 0x00000040
08:21:25 INFO - sp = 0x758ff980 lr = 0x6337226d pc = 0x63372f58
08:21:25 INFO - Found by: given as instruction pointer in context
08:21:25 INFO - 1 libxul.so!resampler_basic_direct_single [resample.c:f078e3258ffd : 369 + 0x7]
08:21:25 INFO - r4 = 0x6f3eadc0 r5 = 0x00000012 r6 = 0x0000003e r7 = 0x633721d9
08:21:25 INFO - r8 = 0x00000000 r9 = 0x00000032 r10 = 0x00000064 fp = 0x00000040
08:21:25 INFO - sp = 0x758ff988 pc = 0x6337226d
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 2 libxul.so!speex_resampler_process_native [resample.c:f078e3258ffd : 883 + 0x3]
08:21:25 INFO - r4 = 0x6f3eadc0 r5 = 0x758ffa20 r6 = 0x00000000 r7 = 0x633721d9
08:21:25 INFO - r8 = 0x758ffa24 r9 = 0x0000003f r10 = 0x68c73640 fp = 0x68c736c0
08:21:25 INFO - sp = 0x758ff9d8 pc = 0x63372107
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 3 libxul.so!moz_speex_resampler_process_int [resample.c:f078e3258ffd : 946 + 0x13]
08:21:25 INFO - r4 = 0x6f3eadc0 r5 = 0x000001b9 r6 = 0x736bdc08 r7 = 0x000000a0
08:21:25 INFO - r8 = 0x00000000 r9 = 0x00000001 r10 = 0x758ffa78 fp = 0x68c736c0
08:21:25 INFO - sp = 0x758ffa00 pc = 0x633729a3
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 4 libxul.so!void mozilla::AudioSegment::Resample<short>(SpeexResamplerState_*, unsigned int, unsigned int) [AudioSegment.h:f078e3258ffd : 211 + 0x3]
08:21:25 INFO - r4 = 0x701f1618 r5 = 0x00000000 r6 = 0x00000001 r7 = 0x758ffb88
08:21:25 INFO - r8 = 0x000001b9 r9 = 0x74057408 r10 = 0x0000ac44 fp = 0x00003e80
08:21:25 INFO - sp = 0x758ffa50 pc = 0x62bd29d5
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 5 libxul.so!mozilla::AudioSegment::ResampleChunks(SpeexResamplerState_*, unsigned int, unsigned int) [AudioSegment.cpp:f078e3258ffd : 141 + 0xb]
08:21:25 INFO - r4 = 0x758ffb88 r5 = 0x6f3eadc0 r6 = 0x00000000 r7 = 0x00003e80
08:21:25 INFO - r8 = 0x0000ac44 r9 = 0x00000001 r10 = 0x63c6278e fp = 0x63c63134
08:21:25 INFO - sp = 0x758ffad8 pc = 0x62bd2b9d
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 6 libxul.so!mozilla::SourceMediaStream::AppendToTrack(int, mozilla::MediaSegment*, mozilla::MediaSegment*) [MediaStreamGraph.cpp:f078e3258ffd : 2342 + 0x9]
08:21:25 INFO - r4 = 0x701c5cd0 r5 = 0x758ffb88 r6 = 0x6f0e1708 r7 = 0x00000000
08:21:25 INFO - r8 = 0x00000001 r9 = 0x00000001 r10 = 0x63c6278e fp = 0x63c63134
08:21:25 INFO - sp = 0x758ffaf8 pc = 0x62be4b7b
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 7 libxul.so!mozilla::MediaPipelineReceiveAudio::PipelineListener::NotifyPull(mozilla::MediaStreamGraph*, long long) [MediaPipeline.cpp:f078e3258ffd : 1461 + 0xb]
08:21:25 INFO - r4 = 0x00000000 r5 = 0x758ffb64 r6 = 0x6f0dd790 r7 = 0x701f1618
08:21:25 INFO - r8 = 0x00000001 r9 = 0x63c6310b r10 = 0x63c6278e fp = 0x63c63134
08:21:25 INFO - sp = 0x758ffb28 pc = 0x62515131
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 8 libxul.so!mozilla::MediaStreamGraphImpl::ExtractPendingInput(mozilla::SourceMediaStream*, long long, bool*) [MediaStreamGraph.cpp:f078e3258ffd : 187 + 0xf]
08:21:25 INFO - r4 = 0x00000002 r5 = 0x6f0dd790 r6 = 0x701c5cd0 r7 = 0x701c5de8
08:21:25 INFO - r8 = 0x63b7eb70 r9 = 0x63b698e0 r10 = 0x63b69892 fp = 0x69466890
08:21:25 INFO - sp = 0x758ffc68 pc = 0x62be921f
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 9 libxul.so!mozilla::MediaStreamGraphImpl::UpdateGraph(long long) [MediaStreamGraph.cpp:f078e3258ffd : 1274 + 0xf]
08:21:25 INFO - r4 = 0x003c8000 r5 = 0x00000000 r6 = 0x69466890 r7 = 0x00000006
08:21:25 INFO - r8 = 0x758ffcf7 r9 = 0x701c5cd0 r10 = 0x694668b4 fp = 0x00000000
08:21:25 INFO - sp = 0x758ffce8 pc = 0x62be9fcd
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 10 libxul.so!mozilla::MediaStreamGraphImpl::OneIteration(long long, long long, long long, long long) [MediaStreamGraph.cpp:f078e3258ffd : 1408 + 0x9]
08:21:25 INFO - r4 = 0x003c76ff r5 = 0x694668dc r6 = 0x69466890 r7 = 0x69abc800
08:21:25 INFO - r8 = 0x00000402 r9 = 0x69abcc48 r10 = 0x003c76ff fp = 0x00000000
08:21:25 INFO - sp = 0x758ffd20 pc = 0x62bfdbdf
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 11 libxul.so!mozilla::AudioCallbackDriver::DataCallback(short*, long) [GraphDriver.cpp:f078e3258ffd : 918 + 0x21]
08:21:25 INFO - r4 = 0x003c7aff r5 = 0x00000000 r6 = 0x00000017 r7 = 0x69abc800
08:21:25 INFO - r8 = 0x00000402 r9 = 0x69abcc48 r10 = 0x003c76ff fp = 0x00000000
08:21:25 INFO - sp = 0x758ffd68 pc = 0x62bd161b
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 12 libxul.so!noop_resampler::fill(void*, long) [cubeb_resampler.cpp:f078e3258ffd : 89 + 0xb]
08:21:25 INFO - r4 = 0x69a6bfc0 r5 = 0x00000402 r6 = 0x62bd1759 r7 = 0x69af3e20
08:21:25 INFO - r8 = 0x00000000 r9 = 0x6ef56000 r10 = 0x6415ecdd fp = 0x63ea9bed
08:21:25 INFO - sp = 0x758ffdc8 pc = 0x6333ea45
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 13 libxul.so!RDFXMLDataSourceImpl::GetInterface(nsID const&, void**) [nsRDFXMLDataSource.cpp:f078e3258ffd : 472 + 0x5]
08:21:25 INFO - r4 = 0x69af3e24 r5 = 0x00001008 r6 = 0x00000003 r7 = 0x69af3e20
08:21:25 INFO - r8 = 0x00000000 r9 = 0x6ef56000 r10 = 0x6415ecdd fp = 0x63ea9bed
08:21:25 INFO - sp = 0x758ffdd8 pc = 0x621a38dd
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 14 libxul.so!bufferqueue_callback [cubeb_opensl.c:f078e3258ffd : 116 + 0x11]
08:21:25 INFO - r4 = 0x69af3e24 r5 = 0x00001008 r6 = 0x00000003 r7 = 0x69af3e20
08:21:25 INFO - r8 = 0x00000000 r9 = 0x6ef56000 r10 = 0x6415ecdd fp = 0x63ea9bed
08:21:25 INFO - sp = 0x758ffde0 pc = 0x6333e1c3
08:21:25 INFO - Found by: call frame info
08:21:25 INFO - 15 libwilhelm.so + 0x8c1b
08:21:25 INFO - r4 = 0x012dacc8 r5 = 0x6333e13d r6 = 0x69af3e20 r7 = 0x758ffe50
08:21:25 INFO - r8 = 0x758ffe50 r9 = 0x00000001 r10 = 0x00100000 fp = 0x00000001
08:21:25 INFO - sp = 0x758ffe18 pc = 0x6fb08c1d
08:21:25 INFO - Found by: call frame info

Comment 1

[Andrew McCreight [:mccr8]]

Karl, do you have some idea what might be going wrong here?  Thanks.

Comment 2

[Karl Tomlinson (:karlt)]

Not really.

The debug info for the stack trace suggests that the instruction pointer is
on code returning or about to return from inner_product_single().

http://hg.mozilla.org/integration/mozilla-inbound/annotate/f078e3258ffd/media/libspeex_resampler/src/resample_neon.c#l124

If the return address had been corrupted then I would have expected a
different instruction pointer before the SIGILL and different stack trace.
The next frame of the stack (resampler_basic_direct_single) is as expected.

SIGILL seems to suggest the CPU may not be capable of processing neon assembly
instructions, but mozilla::supports_neon() has returned true, and I don't know why such a failure would be intermittent.

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Perhaps just the return address was corrupted without corrupting the rest of the stack?  

Odd...

Someone with more ARM knowledge might help untangle this

Comment 4

[Daniel Veditz [:dveditz]]

Jeff: any ARM insights you could add to this audio crash?

Comment 5

[Jeff Muizelaar [:jrmuizel]]

So the crash doesn't make any sense to me:

Here's the beginning of moz_speex_inner_product_single:
            0x01567f56    10b5         push {r4, lr}
            0x01567f58    0346         mov r3, r0
            0x01567f5a    02f00f04     and r4, r2, 0xf
            0x01567f5e    22f00f02     bic r2, r2, 0xf

The illegal instruction is apparently:
0346         mov r3, r0

which makes no sense.

Do we have more of these crashes? Are they at the same address?

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Hard to say definitely at this point, since we can't star failures from Treeherder to s-s bugs. However, this is the only bug that comes up when I search for moz_speex_inner_product_single and there haven't been any other log links added manually to the bug. So it appears not from what I can see.

Comment 7

[Jeff Muizelaar [:jrmuizel]]

One idea I tried was thinking that we somehow switched out of the 16 bit thumb mode and into the 32 bit arm mode. This would cause illegal instructions, but I can't see how it could happen.

Comment 8

[Matt Wobensmith [:mwobensmith][:matt:]]

If this happened just once, we will treat it as a fluke, as there doesn't appear to be much traction here. Reopen if it begins to rear its ugly head again.