Closed Bug 847903 Opened 13 years ago Closed 10 years ago

Intermittent crashtests/691096-1.html | Exited with code 1 during test run | application crashed [@ CoreAudio + 0x1c7fe]

Categories

(Core :: Audio/Video, defect)

Product:
Core
Component:
Audio/Video
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla40
Tracking Flags:
Tracking Status
firefox29 --- wontfix
firefox30 --- wontfix
firefox31 --- wontfix
firefox32 --- wontfix
firefox33 --- wontfix
firefox34 --- wontfix
firefox35 --- wontfix
firefox36 --- wontfix
firefox37 --- wontfix
firefox38 --- fixed
firefox39 --- fixed
firefox40 --- fixed
firefox-esr24 --- unaffected
b2g-v1.2 --- wontfix
b2g-v1.3 --- wontfix
b2g-v1.3T --- wontfix
b2g-v1.4 --- wontfix
b2g-v2.0 --- wontfix
b2g-v2.0M --- wontfix
b2g-v2.1 --- wontfix
b2g-v2.2 --- fixed
b2g-master --- fixed

People

(Reporter: RyanVM, Assigned: RyanVM)

References

Details

(Keywords: crash, intermittent-failure)

Crash Data

Attachments

(1 file)

bug847903_v0.patch
7.15 KB, patch
padenot
: review+
Sylvestre
: approval-mozilla-aurora+
Sylvestre
: approval-mozilla-beta+
Details | Diff | Splinter Review
https://tbpl.mozilla.org/php/getParsedLog.php?id=20319569&tree=Mozilla-Inbound Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound debug test crashtest on 2013-03-04 17:45:04 PST for push c5a453f499b2 slave: talos-r4-snow-042 17:48:42 INFO - REFTEST TEST-START | file:///Users/cltbld/talos-slave/test/build/tests/reftest/tests/content/media/test/crashtests/691096-1.html | 469 / 2305 (20%) 17:48:42 INFO - ++DOMWINDOW == 29 (0x12b8b70a8) [serial = 1051] [outer = 0x109db7508] 17:48:45 WARNING - TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/tests/reftest/tests/content/media/test/crashtests/691096-1.html | Exited with code 1 during test run 17:48:45 WARNING - This is a harness error. 17:48:45 INFO - INFO | automation.py | Application ran for: 0:01:39.716888 17:48:45 INFO - INFO | automation.py | Reading PID log: /var/folders/Hs/HsDn6a9SG8idoIya6p9mtE+++TI/-Tmp-/tmpnOPJN6pidlog 17:49:03 INFO - PROCESS-CRASH | file:///Users/cltbld/talos-slave/test/build/tests/reftest/tests/content/media/test/crashtests/691096-1.html | application crashed [@ CoreAudio + 0x1c7fe] 17:49:03 INFO - Crash dump filename: /var/folders/Hs/HsDn6a9SG8idoIya6p9mtE+++TI/-Tmp-/tmpSdbhxW/minidumps/506E489E-8390-42FD-9F4F-D341B03FB2F6.dmp 17:49:03 INFO - Operating system: Mac OS X 17:49:03 INFO - 10.6.8 10K549 17:49:03 INFO - CPU: amd64 17:49:03 INFO - family 6 model 23 stepping 10 17:49:03 INFO - 2 CPUs 17:49:03 INFO - Crash reason: EXC_BAD_ACCESS / 0x0000000d 17:49:03 INFO - Crash address: 0x0 17:49:03 INFO - Thread 114 (crashed) 17:49:03 INFO - 0 CoreAudio + 0x1c7fe 17:49:03 INFO - rbx = 0x000000012b827cb0 r12 = 0x000000012b827cb0 17:49:03 INFO - r13 = 0x0000000000000000 r14 = 0x00000001487d4d10 17:49:03 INFO - r15 = 0x000000012b871e5c rip = 0x00007fff80ac87fe 17:49:03 INFO - rsp = 0x000000014ad415e0 rbp = 0x000000014ad415e0 17:49:03 INFO - Found by: given as instruction pointer in context 17:49:03 INFO - 1 CoreAudio + 0x1c762 17:49:03 INFO - rip = 0x00007fff80ac8763 rsp = 0x000000014ad415f0 17:49:03 INFO - rbp = 0x000000014ad41630 17:49:03 INFO - Found by: stack scanning 17:49:03 INFO - 2 CoreAudio + 0x17b78 17:49:03 INFO - rip = 0x00007fff80ac3b79 rsp = 0x000000014ad41640 17:49:03 INFO - rbp = 0x000000014ad416a0 17:49:03 INFO - Found by: stack scanning 17:49:03 INFO - 3 libSystem.B.dylib + 0x4d1f 17:49:03 INFO - rip = 0x00007fff81130d20 rsp = 0x000000014ad41650 17:49:03 INFO - rbp = 0x000000014ad416a0 17:49:03 INFO - Found by: stack scanning 17:49:03 INFO - 4 libSystem.B.dylib + 0x3846 17:49:03 INFO - rip = 0x00007fff8112f847 rsp = 0x000000014ad41670 17:49:03 INFO - rbp = 0x000000014ad416a0 17:49:03 INFO - Found by: stack scanning 17:49:03 INFO - 5 CoreAudio + 0x1797f 17:49:03 INFO - rip = 0x00007fff80ac3980 rsp = 0x000000014ad416b0 17:49:03 INFO - rbp = 0x000000014ad41720 17:49:03 INFO - Found by: stack scanning 17:49:03 INFO - 6 CoreAudio + 0x18298 17:49:03 INFO - rip = 0x00007fff80ac4299 rsp = 0x000000014ad416c0 17:49:03 INFO - rbp = 0x000000014ad41720 17:49:03 INFO - Found by: stack scanning
(OrangeWFM bugs not touched for 2 months)
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Closing inactive keywords:intermittent-failure bugs where the TBPLbot has previously commented and the test isn't marked as disabled; filter on orange-cleanup-201401.
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Further down the stack: 05:45:54 INFO - 10 AudioUnit + 0xcf7 05:45:54 INFO - rip = 0x00007fff81821cf8 rsp = 0x0000000149191bf0 05:45:54 INFO - rbp = 0x0000000149191c00 05:45:54 INFO - Found by: stack scanning 05:45:54 INFO - 11 XUL!nsGlobalWindow::SetOnsubmit(JSContext*, JS::Handle<JS::Value>) [HoldDropJSObjects.h:e68b4acdc873 : 48 + 0x4] 05:45:54 INFO - rip = 0x0000000102010000 rsp = 0x0000000149191bf8 05:45:54 INFO - rbp = 0x0000000149191c00 05:45:54 INFO - Found by: stack scanning 05:45:54 INFO - 12 XUL!audiounit_stream_start [cubeb_audiounit.c:e68b4acdc873 : 480 + 0x8] 05:45:54 INFO - rbx = 0x0000000000000001 r12 = 0x0000000149191c00 05:45:54 INFO - r13 = 0x00007fff81821cf8 r14 = 0x0000000102010000 05:45:54 INFO - r15 = 0x0000000000820001 rip = 0x0000000102ee0c42 05:45:54 INFO - rsp = 0x0000000149191c10 rbp = 0x0000000149191c20 05:45:54 INFO - Found by: call frame info 05:45:54 INFO - 13 XUL!mozilla::AudioStream::Start() [AudioStream.cpp:e68b4acdc873 : 561 + 0x4] 05:45:54 INFO - rbx = 0x0000000128553700 r12 = 0x0000000149191c00 05:45:54 INFO - r13 = 0x00007fff81821cf8 r14 = 0x0000000102010000 05:45:54 INFO - r15 = 0x0000000000820001 rip = 0x000000010243ea39 05:45:54 INFO - rsp = 0x0000000149191c30 rbp = 0x0000000149191c40 05:45:54 INFO - Found by: call frame info 05:45:54 INFO - 14 XUL!mozilla::MediaDecoderStateMachine::PlayFromAudioQueue(unsigned long long, unsigned int) [MediaDecoderStateMachine.cpp:e68b4acdc873 : 722 + 0x7] 05:45:54 INFO - rbx = 0x0000000128553700 r12 = 0x0000000149191c00 05:45:54 INFO - r13 = 0x00007fff81821cf8 r14 = 0x0000000000000001 05:45:54 INFO - r15 = 0x0000000000820001 rip = 0x00000001024529c6 05:45:54 INFO - rsp = 0x0000000149191c50 rbp = 0x0000000149191ca0 05:45:54 INFO - Found by: call frame info 05:45:54 INFO - 15 XUL!mozilla::MediaDecoderStateMachine::AudioLoop() [MediaDecoderStateMachine.cpp:e68b4acdc873 : 883 + 0xa] 05:45:54 INFO - rbx = 0x0000000000000000 r12 = 0x0000000000001f40 05:45:54 INFO - r13 = 0x0000000000000600 r14 = 0x0000000128bd9180 05:45:54 INFO - r15 = 0x0000000000000101 rip = 0x000000010245242e 05:45:54 INFO - rsp = 0x0000000149191cb0 rbp = 0x0000000149191d20 05:45:54 INFO - Found by: call frame info 05:45:54 INFO - 16 XUL!nsRunnableMethodImpl<void (mozilla::MediaDecoderStateMachine::*)(), void, true>::Run() [nsThreadUtils.h:e68b4acdc873 : 383 + 0x15] 05:45:54 INFO - rbx = 0x0000000129e31060 r12 = 0x0000000000000000 05:45:54 INFO - r13 = 0x0000000129e3110c r14 = 0x0000000129e31088 05:45:54 INFO - r15 = 0x0000000129e310c8 rip = 0x00000001024576f6 05:45:54 INFO - rsp = 0x0000000149191d30 rbp = 0x0000000149191d30 05:45:54 INFO - Found by: call frame info 05:45:54 INFO - 17 XUL!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:e68b4acdc873 : 694 + 0x5] 05:45:54 INFO - rbx = 0x0000000129e31060 r12 = 0x0000000000000000 05:45:54 INFO - r13 = 0x0000000129e3110c r14 = 0x0000000129e31088 05:45:54 INFO - r15 = 0x0000000129e310c8 rip = 0x000000010107cf9f 05:45:54 INFO - rsp = 0x0000000149191d40 rbp = 0x0000000149191e00 05:45:54 INFO - Found by: call frame info 05:45:54 INFO - 18 XUL!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:e68b4acdc873 : 263 + 0xc] 05:45:54 INFO - rbx = 0x0000000000000000 r12 = 0x00000001281c3958 05:45:54 INFO - r13 = 0x00000001281c3940 r14 = 0x000000010b6d7780 05:45:54 INFO - r15 = 0x0000000000000001 rip = 0x0000000101017a1e 05:45:54 INFO - rsp = 0x0000000149191e10 rbp = 0x0000000149191e20 05:45:54 INFO - Found by: call frame info 05:45:54 INFO - 19 XUL!mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [MessagePump.cpp:e68b4acdc873 : 303 + 0xa] 05:45:54 INFO - rbx = 0x0000000129e31001 r12 = 0x00000001281c3958 05:45:54 INFO - r13 = 0x00000001281c3940 r14 = 0x000000010b6d7780 05:45:54 INFO - r15 = 0x0000000000000001 rip = 0x0000000101314326 05:45:54 INFO - rsp = 0x0000000149191e30 rbp = 0x0000000149191e80 05:45:54 INFO - Found by: call frame info 05:45:54 INFO - 20 XUL!MessageLoop::Run() [message_loop.cc:e68b4acdc873 : 226 + 0x8] 05:45:54 INFO - rbx = 0x0000000129e31060 r12 = 0x0000000000000000 05:45:54 INFO - r13 = 0x0000000000014c03 r14 = 0x000000010b6d7780 05:45:54 INFO - r15 = 0x0000000000000000 rip = 0x00000001012dfc9e 05:45:54 INFO - rsp = 0x0000000149191e90 rbp = 0x0000000149191eb0 05:45:54 INFO - Found by: call frame info Matthew, is this something you could look at?
Flags: needinfo?(kinetik)
It looks like it's crashing deep inside AudioOutputUnitStart. There's one other thread inside audiounit_stream_start, and a lot (15) of other threads inside audiounit_stream_init. crashtests/691096-1.html is trying to play 250 audio elements. This also only seems to crash on 10.6, and not 10.8. In some libcubeb backends, we needed to add a hardcoded stream limit to avoid triggering (what looked like) bugs in the OS audio code. I'll try doing that for the AudioUnit backend when it's running on 10.6. https://tbpl.mozilla.org/?tree=Try&rev=2d1375b5389a
Flags: needinfo?(kinetik)
Apparently I screwed up editing the list of reftests to run, so revert that change and try again: https://tbpl.mozilla.org/?tree=Try&rev=5fe15585c416
Looks like I picked the wrong week to quit sniffing glue. I was running the reftests rather than the crashtests on try. And again: https://tbpl.mozilla.org/?tree=Try&rev=912905c4f270
That looks good. Version with test for 10.6 only: https://tbpl.mozilla.org/?tree=Try&rev=94e0aeea9a1f
Attached patch bug847903_v0.patch — — Splinter Review
Limit active streams on 10.6 and older.
Assignee: nobody → kinetik
Status: REOPENED → ASSIGNED
Attachment #8425238 - Flags: review?(paul)
Comment on attachment 8425238 [details] [diff] [review] bug847903_v0.patch Review of attachment 8425238 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/crashtest/crashtests.list @@ -71,5 @@ > -include ../../widget/crashtests/crashtests.list > - > -include ../../media/libpng/crashtests/crashtests.list > - > -include ../../xpcom/string/crashtests/crashtests.list I assume you'll remove this part before landing. (maybe it's worth mentionning that `./mach crashtest content/media/test/crashtests/` has, I think, the same behaviour of this patch).
Attachment #8425238 - Flags: review?(paul) → review+
(In reply to Paul Adenot (:padenot) from comment #101) > Comment on attachment 8425238 [details] [diff] [review] > bug847903_v0.patch > > Review of attachment 8425238 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: testing/crashtest/crashtests.list > @@ -71,5 @@ > > -include ../../widget/crashtests/crashtests.list > > - > > -include ../../media/libpng/crashtests/crashtests.list > > - > > -include ../../xpcom/string/crashtests/crashtests.list > > I assume you'll remove this part before landing. Oops, yes. Thanks for pointing out that I'd left it there. > (maybe it's worth mentionning that `./mach crashtest > content/media/test/crashtests/` has, I think, the same behaviour of this > patch). Yeah, this was for running the tests on try. I don't know of an easy way to run a specific dir/test there other than hacking the manifests. :-/
https://hg.mozilla.org/mozilla-central/rev/32ae940ce5d1
Status: ASSIGNED → RESOLVED
Closed: 12 years ago → 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla32
When you feel reasonably confident about this fix, please request Aurora/Beta approval :)
status-b2g-v1.2: --- → wontfix
status-b2g-v1.3: --- → wontfix
status-b2g-v1.3T: --- → wontfix
status-b2g-v1.4: --- → affected
status-b2g-v2.0: --- → fixed
status-firefox29: --- → wontfix
status-firefox30: --- → affected
status-firefox31: --- → affected
status-firefox32: --- → fixed
status-firefox-esr24: --- → unaffected
Comment on attachment 8425238 [details] [diff] [review] bug847903_v0.patch [Approval Request Comment] Bug caused by (feature/regressing bug #): bug 742160, possibly existed in the old audio backend too User impact if declined: crash with many streams active on older OS X versions Testing completed (on m-c, etc.): many try runs, local testing, landed on m-c Risk to taking this patch (and alternatives if risky): very low, easy to back out String or IDL/UUID changes made by this patch: none
Attachment #8425238 - Flags: approval-mozilla-beta?
Attachment #8425238 - Flags: approval-mozilla-aurora?
Attachment #8425238 - Flags: approval-mozilla-beta?
Attachment #8425238 - Flags: approval-mozilla-beta+
Attachment #8425238 - Flags: approval-mozilla-aurora?
Attachment #8425238 - Flags: approval-mozilla-aurora+
Booo :(
Status: RESOLVED → REOPENED
status-b2g-v1.4: fixed → affected
status-b2g-v2.0: fixed → affected
status-firefox30: fixed → affected
status-firefox31: fixed → affected
status-firefox32: fixed → affected
Resolution: FIXED → ---
Target Milestone: mozilla32 → ---
Sorry. I'll take another look at this soon, but I've got other stuff on my plate right now.
Flags: needinfo?(kinetik)
See Also: → 1009992, 1021174
Flags: needinfo?(kinetik)
Attachment #8425238 - Attachment is obsolete: true
Attachment #8425238 - Attachment is obsolete: false
(In reply to Matthew Gregan [:kinetik] from comment #166) > https://hg.mozilla.org/integration/mozilla-inbound/rev/50e33b2149ff Based on JW's research in bug 1059265, the limit is still too high. This drops the limit to 8 on 10.6, let's see if this helps...
https://hg.mozilla.org/mozilla-central/rev/50e33b2149ff
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla35
(In reply to TBPL Robot from comment #169) Sadly, it is back. Let's keep an eye on how 1059265 progresses. There might multiple causes for this bug...
Status: RESOLVED → REOPENED
status-b2g-v2.0: affected → wontfix
status-b2g-v2.0M: --- → wontfix
status-b2g-v2.1: --- → affected
status-b2g-v2.2: --- → affected
status-firefox31: affected → wontfix
status-firefox32: affected → wontfix
status-firefox33: --- → affected
status-firefox34: --- → affected
status-firefox35: --- → affected
Resolution: FIXED → ---
Target Milestone: mozilla35 → ---
(In reply to TBPL Robot from comment #169) 17:13:09 INFO - 0 CoreAudio + 0x1c7fe 17:13:09 INFO - rbx = 0x0000000125ed4800 r12 = 0x0000000125ed4800 17:13:09 INFO - r13 = 0x0000000000000000 r14 = 0x00000001261f9e00 17:13:09 INFO - r15 = 0x00000001538b754c rip = 0x00007fff878247fe 17:13:09 INFO - rsp = 0x00000001488d5360 rbp = 0x00000001488d5360 17:13:09 INFO - Found by: given as instruction pointer in context 17:13:09 INFO - 1 CoreAudio + 0x1c762 17:13:09 INFO - rip = 0x00007fff87824763 rsp = 0x00000001488d5370 17:13:09 INFO - rbp = 0x00000001488d53b0 17:13:09 INFO - Found by: stack scanning 17:13:09 INFO - 2 libmozglue.dylib!arena_malloc [_string.h : 83 + 0xf] 17:13:09 INFO - rip = 0x00000001000108b0 rsp = 0x00000001488d5380 17:13:09 INFO - rbp = 0x00000001488d53b0 17:13:09 INFO - Found by: stack scanning I am curious how arena_malloc could call into CoreAudio? I start to wonder if this issue is about some kind of OOM.
Assignee: kinetik → nobody
It looks to me that all failures since Jan. 15th 2014 is from Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound opt test crashtest. It doesn't appear that the bug affects b2g anymore? Also, comment 167 seems to indicate that mac os tests should have been disabled. Or am I misinterpreting the comments?
> that the bug affects b2g anymore? Also, comment 167 seems to indicate that > mac os tests should have been disabled. Or am I misinterpreting the comments? Sorry, that was Comment 197.
QA Whiteboard: QAExclude
I am following up on Comments 221 and 222 to see if there are action items for this issue. Do the tests for mac still need to be disabled? NI :RyanVM
Flags: needinfo?(ryanvm)
Looks like we disabled our way to victory here. Ultimately, we should probably just disable any tests on 10.6 that hit [@ CoreAudio + 0x1c7fe] crashes, since the basic gist of the problem is that it simply can't handle them reliably. Matt, would you agree with that statement?
Flags: needinfo?(ryanvm) → needinfo?(kinetik)
Yeah, sadly, there's nothing better we can do for now.
Flags: needinfo?(kinetik)
https://hg.mozilla.org/mozilla-central/rev/4824c5c9cb38 Going to go ahead and resolve this under the assumption that this will be the last we think of this until after 10.6 is shut off permanently.
Assignee: nobody → ryanvm
Status: REOPENED → RESOLVED
Closed: 11 years ago → 10 years ago
status-firefox40: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla40
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: