Closed Bug 847903 Opened 10 years ago Closed 8 years ago

Intermittent crashtests/691096-1.html | Exited with code 1 during test run | application crashed [@ CoreAudio + 0x1c7fe]

Categories

(Core :: Audio/Video, defect)

Product:
Core
Component:
Audio/Video
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla40
Tracking Flags:
Tracking Status
firefox29 --- wontfix
firefox30 --- wontfix
firefox31 --- wontfix
firefox32 --- wontfix
firefox33 --- wontfix
firefox34 --- wontfix
firefox35 --- wontfix
firefox36 --- wontfix
firefox37 --- wontfix
firefox38 --- fixed
firefox39 --- fixed
firefox40 --- fixed
firefox-esr24 --- unaffected
b2g-v1.2 --- wontfix
b2g-v1.3 --- wontfix
b2g-v1.3T --- wontfix
b2g-v1.4 --- wontfix
b2g-v2.0 --- wontfix
b2g-v2.0M --- wontfix
b2g-v2.1 --- wontfix
b2g-v2.2 --- fixed
b2g-master --- fixed

People

(Reporter: RyanVM, Assigned: RyanVM)

References

Details

(Keywords: crash, intermittent-failure)

Crash Data

Attachments

(1 file)

bug847903_v0.patch
7.15 KB, patch
padenot
: review+
Sylvestre
: approval-mozilla-aurora+
Sylvestre
: approval-mozilla-beta+
Details | Diff | Splinter Review 
https://tbpl.mozilla.org/php/getParsedLog.php?id=20319569&tree=Mozilla-Inbound

Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound debug test crashtest on 2013-03-04 17:45:04 PST for push c5a453f499b2
slave: talos-r4-snow-042

17:48:42     INFO -  REFTEST TEST-START | file:///Users/cltbld/talos-slave/test/build/tests/reftest/tests/content/media/test/crashtests/691096-1.html | 469 / 2305 (20%)
17:48:42     INFO -  ++DOMWINDOW == 29 (0x12b8b70a8) [serial = 1051] [outer = 0x109db7508]
17:48:45  WARNING -  TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/tests/reftest/tests/content/media/test/crashtests/691096-1.html | Exited with code 1 during test run
17:48:45  WARNING -  This is a harness error.
17:48:45     INFO -  INFO | automation.py | Application ran for: 0:01:39.716888
17:48:45     INFO -  INFO | automation.py | Reading PID log: /var/folders/Hs/HsDn6a9SG8idoIya6p9mtE+++TI/-Tmp-/tmpnOPJN6pidlog
17:49:03     INFO -  PROCESS-CRASH | file:///Users/cltbld/talos-slave/test/build/tests/reftest/tests/content/media/test/crashtests/691096-1.html | application crashed [@ CoreAudio + 0x1c7fe]
17:49:03     INFO -  Crash dump filename: /var/folders/Hs/HsDn6a9SG8idoIya6p9mtE+++TI/-Tmp-/tmpSdbhxW/minidumps/506E489E-8390-42FD-9F4F-D341B03FB2F6.dmp
17:49:03     INFO -  Operating system: Mac OS X
17:49:03     INFO -                    10.6.8 10K549
17:49:03     INFO -  CPU: amd64
17:49:03     INFO -       family 6 model 23 stepping 10
17:49:03     INFO -       2 CPUs
17:49:03     INFO -  Crash reason:  EXC_BAD_ACCESS / 0x0000000d
17:49:03     INFO -  Crash address: 0x0
17:49:03     INFO -  Thread 114 (crashed)
17:49:03     INFO -   0  CoreAudio + 0x1c7fe
17:49:03     INFO -      rbx = 0x000000012b827cb0   r12 = 0x000000012b827cb0
17:49:03     INFO -      r13 = 0x0000000000000000   r14 = 0x00000001487d4d10
17:49:03     INFO -      r15 = 0x000000012b871e5c   rip = 0x00007fff80ac87fe
17:49:03     INFO -      rsp = 0x000000014ad415e0   rbp = 0x000000014ad415e0
17:49:03     INFO -      Found by: given as instruction pointer in context
17:49:03     INFO -   1  CoreAudio + 0x1c762
17:49:03     INFO -      rip = 0x00007fff80ac8763   rsp = 0x000000014ad415f0
17:49:03     INFO -      rbp = 0x000000014ad41630
17:49:03     INFO -      Found by: stack scanning
17:49:03     INFO -   2  CoreAudio + 0x17b78
17:49:03     INFO -      rip = 0x00007fff80ac3b79   rsp = 0x000000014ad41640
17:49:03     INFO -      rbp = 0x000000014ad416a0
17:49:03     INFO -      Found by: stack scanning
17:49:03     INFO -   3  libSystem.B.dylib + 0x4d1f
17:49:03     INFO -      rip = 0x00007fff81130d20   rsp = 0x000000014ad41650
17:49:03     INFO -      rbp = 0x000000014ad416a0
17:49:03     INFO -      Found by: stack scanning
17:49:03     INFO -   4  libSystem.B.dylib + 0x3846
17:49:03     INFO -      rip = 0x00007fff8112f847   rsp = 0x000000014ad41670
17:49:03     INFO -      rbp = 0x000000014ad416a0
17:49:03     INFO -      Found by: stack scanning
17:49:03     INFO -   5  CoreAudio + 0x1797f
17:49:03     INFO -      rip = 0x00007fff80ac3980   rsp = 0x000000014ad416b0
17:49:03     INFO -      rbp = 0x000000014ad41720
17:49:03     INFO -      Found by: stack scanning
17:49:03     INFO -   6  CoreAudio + 0x18298
17:49:03     INFO -      rip = 0x00007fff80ac4299   rsp = 0x000000014ad416c0
17:49:03     INFO -      rbp = 0x000000014ad41720
17:49:03     INFO -      Found by: stack scanning
(OrangeWFM bugs not touched for 2 months)
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Closing inactive keywords:intermittent-failure bugs where the TBPLbot has previously commented and the test isn't marked as disabled; filter on orange-cleanup-201401.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Further down the stack:

05:45:54     INFO -  10  AudioUnit + 0xcf7
05:45:54     INFO -      rip = 0x00007fff81821cf8   rsp = 0x0000000149191bf0
05:45:54     INFO -      rbp = 0x0000000149191c00
05:45:54     INFO -      Found by: stack scanning
05:45:54     INFO -  11  XUL!nsGlobalWindow::SetOnsubmit(JSContext*, JS::Handle<JS::Value>) [HoldDropJSObjects.h:e68b4acdc873 : 48 + 0x4]
05:45:54     INFO -      rip = 0x0000000102010000   rsp = 0x0000000149191bf8
05:45:54     INFO -      rbp = 0x0000000149191c00
05:45:54     INFO -      Found by: stack scanning
05:45:54     INFO -  12  XUL!audiounit_stream_start [cubeb_audiounit.c:e68b4acdc873 : 480 + 0x8]
05:45:54     INFO -      rbx = 0x0000000000000001   r12 = 0x0000000149191c00
05:45:54     INFO -      r13 = 0x00007fff81821cf8   r14 = 0x0000000102010000
05:45:54     INFO -      r15 = 0x0000000000820001   rip = 0x0000000102ee0c42
05:45:54     INFO -      rsp = 0x0000000149191c10   rbp = 0x0000000149191c20
05:45:54     INFO -      Found by: call frame info
05:45:54     INFO -  13  XUL!mozilla::AudioStream::Start() [AudioStream.cpp:e68b4acdc873 : 561 + 0x4]
05:45:54     INFO -      rbx = 0x0000000128553700   r12 = 0x0000000149191c00
05:45:54     INFO -      r13 = 0x00007fff81821cf8   r14 = 0x0000000102010000
05:45:54     INFO -      r15 = 0x0000000000820001   rip = 0x000000010243ea39
05:45:54     INFO -      rsp = 0x0000000149191c30   rbp = 0x0000000149191c40
05:45:54     INFO -      Found by: call frame info
05:45:54     INFO -  14  XUL!mozilla::MediaDecoderStateMachine::PlayFromAudioQueue(unsigned long long, unsigned int) [MediaDecoderStateMachine.cpp:e68b4acdc873 : 722 + 0x7]
05:45:54     INFO -      rbx = 0x0000000128553700   r12 = 0x0000000149191c00
05:45:54     INFO -      r13 = 0x00007fff81821cf8   r14 = 0x0000000000000001
05:45:54     INFO -      r15 = 0x0000000000820001   rip = 0x00000001024529c6
05:45:54     INFO -      rsp = 0x0000000149191c50   rbp = 0x0000000149191ca0
05:45:54     INFO -      Found by: call frame info
05:45:54     INFO -  15  XUL!mozilla::MediaDecoderStateMachine::AudioLoop() [MediaDecoderStateMachine.cpp:e68b4acdc873 : 883 + 0xa]
05:45:54     INFO -      rbx = 0x0000000000000000   r12 = 0x0000000000001f40
05:45:54     INFO -      r13 = 0x0000000000000600   r14 = 0x0000000128bd9180
05:45:54     INFO -      r15 = 0x0000000000000101   rip = 0x000000010245242e
05:45:54     INFO -      rsp = 0x0000000149191cb0   rbp = 0x0000000149191d20
05:45:54     INFO -      Found by: call frame info
05:45:54     INFO -  16  XUL!nsRunnableMethodImpl<void (mozilla::MediaDecoderStateMachine::*)(), void, true>::Run() [nsThreadUtils.h:e68b4acdc873 : 383 + 0x15]
05:45:54     INFO -      rbx = 0x0000000129e31060   r12 = 0x0000000000000000
05:45:54     INFO -      r13 = 0x0000000129e3110c   r14 = 0x0000000129e31088
05:45:54     INFO -      r15 = 0x0000000129e310c8   rip = 0x00000001024576f6
05:45:54     INFO -      rsp = 0x0000000149191d30   rbp = 0x0000000149191d30
05:45:54     INFO -      Found by: call frame info
05:45:54     INFO -  17  XUL!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:e68b4acdc873 : 694 + 0x5]
05:45:54     INFO -      rbx = 0x0000000129e31060   r12 = 0x0000000000000000
05:45:54     INFO -      r13 = 0x0000000129e3110c   r14 = 0x0000000129e31088
05:45:54     INFO -      r15 = 0x0000000129e310c8   rip = 0x000000010107cf9f
05:45:54     INFO -      rsp = 0x0000000149191d40   rbp = 0x0000000149191e00
05:45:54     INFO -      Found by: call frame info
05:45:54     INFO -  18  XUL!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:e68b4acdc873 : 263 + 0xc]
05:45:54     INFO -      rbx = 0x0000000000000000   r12 = 0x00000001281c3958
05:45:54     INFO -      r13 = 0x00000001281c3940   r14 = 0x000000010b6d7780
05:45:54     INFO -      r15 = 0x0000000000000001   rip = 0x0000000101017a1e
05:45:54     INFO -      rsp = 0x0000000149191e10   rbp = 0x0000000149191e20
05:45:54     INFO -      Found by: call frame info
05:45:54     INFO -  19  XUL!mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [MessagePump.cpp:e68b4acdc873 : 303 + 0xa]
05:45:54     INFO -      rbx = 0x0000000129e31001   r12 = 0x00000001281c3958
05:45:54     INFO -      r13 = 0x00000001281c3940   r14 = 0x000000010b6d7780
05:45:54     INFO -      r15 = 0x0000000000000001   rip = 0x0000000101314326
05:45:54     INFO -      rsp = 0x0000000149191e30   rbp = 0x0000000149191e80
05:45:54     INFO -      Found by: call frame info
05:45:54     INFO -  20  XUL!MessageLoop::Run() [message_loop.cc:e68b4acdc873 : 226 + 0x8]
05:45:54     INFO -      rbx = 0x0000000129e31060   r12 = 0x0000000000000000
05:45:54     INFO -      r13 = 0x0000000000014c03   r14 = 0x000000010b6d7780
05:45:54     INFO -      r15 = 0x0000000000000000   rip = 0x00000001012dfc9e
05:45:54     INFO -      rsp = 0x0000000149191e90   rbp = 0x0000000149191eb0
05:45:54     INFO -      Found by: call frame info

Matthew, is this something you could look at?
Flags: needinfo?(kinetik)
It looks like it's crashing deep inside AudioOutputUnitStart.  There's one other thread inside audiounit_stream_start, and a lot (15) of other threads inside audiounit_stream_init.  crashtests/691096-1.html is trying to play 250 audio elements.  This also only seems to crash on 10.6, and not 10.8.

In some libcubeb backends, we needed to add a hardcoded stream limit to avoid triggering (what looked like) bugs in the OS audio code.  I'll try doing that for the AudioUnit backend when it's running on 10.6.

https://tbpl.mozilla.org/?tree=Try&rev=2d1375b5389a
Flags: needinfo?(kinetik)
Apparently I screwed up editing the list of reftests to run, so revert that change and try again:

https://tbpl.mozilla.org/?tree=Try&rev=5fe15585c416
Looks like I picked the wrong week to quit sniffing glue.  I was running the reftests rather than the crashtests on try.  And again:

https://tbpl.mozilla.org/?tree=Try&rev=912905c4f270
That looks good.  Version with test for 10.6 only:
https://tbpl.mozilla.org/?tree=Try&rev=94e0aeea9a1f
Attached patch bug847903_v0.patch — — Splinter Review 
Limit active streams on 10.6 and older.
Assignee: nobody → kinetik
Status: REOPENED → ASSIGNED
Attachment #8425238 - Flags: review?(paul)
Comment on attachment 8425238 [details] [diff] [review]
bug847903_v0.patch

Review of attachment 8425238 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/crashtest/crashtests.list
@@ -71,5 @@
> -include ../../widget/crashtests/crashtests.list
> -
> -include ../../media/libpng/crashtests/crashtests.list
> -
> -include ../../xpcom/string/crashtests/crashtests.list

I assume you'll remove this part before landing.

(maybe it's worth mentionning that `./mach crashtest content/media/test/crashtests/` has, I think, the same behaviour of this patch).
Attachment #8425238 - Flags: review?(paul) → review+
(In reply to Paul Adenot (:padenot) from comment #101)
> Comment on attachment 8425238 [details] [diff] [review]
> bug847903_v0.patch
> 
> Review of attachment 8425238 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: testing/crashtest/crashtests.list
> @@ -71,5 @@
> > -include ../../widget/crashtests/crashtests.list
> > -
> > -include ../../media/libpng/crashtests/crashtests.list
> > -
> > -include ../../xpcom/string/crashtests/crashtests.list
> 
> I assume you'll remove this part before landing.

Oops, yes.  Thanks for pointing out that I'd left it there.

> (maybe it's worth mentionning that `./mach crashtest
> content/media/test/crashtests/` has, I think, the same behaviour of this
> patch).

Yeah, this was for running the tests on try.  I don't know of an easy way to run a specific dir/test there other than hacking the manifests. :-/
https://hg.mozilla.org/mozilla-central/rev/32ae940ce5d1
Status: ASSIGNED → RESOLVED
Closed: 10 years ago → 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla32
When you feel reasonably confident about this fix, please request Aurora/Beta approval :)
status-b2g-v1.2: --- → wontfix
status-b2g-v1.3: --- → wontfix
status-b2g-v1.3T: --- → wontfix
status-b2g-v1.4: --- → affected
status-b2g-v2.0: --- → fixed
status-firefox29: --- → wontfix
status-firefox30: --- → affected
status-firefox31: --- → affected
status-firefox32: --- → fixed
status-firefox-esr24: --- → unaffected
Comment on attachment 8425238 [details] [diff] [review]
bug847903_v0.patch

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 742160, possibly existed in the old audio backend too
User impact if declined: crash with many streams active on older OS X versions
Testing completed (on m-c, etc.): many try runs, local testing, landed on m-c
Risk to taking this patch (and alternatives if risky): very low, easy to back out
String or IDL/UUID changes made by this patch: none
Attachment #8425238 - Flags: approval-mozilla-beta?
Attachment #8425238 - Flags: approval-mozilla-aurora?
Attachment #8425238 - Flags: approval-mozilla-beta?
Attachment #8425238 - Flags: approval-mozilla-beta+
Attachment #8425238 - Flags: approval-mozilla-aurora?
Attachment #8425238 - Flags: approval-mozilla-aurora+
Booo :(
Status: RESOLVED → REOPENED
status-b2g-v1.4: fixed → affected
status-b2g-v2.0: fixed → affected
status-firefox30: fixed → affected
status-firefox31: fixed → affected
status-firefox32: fixed → affected
Resolution: FIXED → ---
Target Milestone: mozilla32 → ---
Sorry.  I'll take another look at this soon, but I've got other stuff on my plate right now.
Flags: needinfo?(kinetik)
See Also: → 1009992, 1021174
Flags: needinfo?(kinetik)
Attachment #8425238 - Attachment is obsolete: true
Attachment #8425238 - Attachment is obsolete: false
(In reply to Matthew Gregan [:kinetik] from comment #166)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/50e33b2149ff

Based on JW's research in bug 1059265, the limit is still too high.  This drops the limit to 8 on 10.6, let's see if this helps...
https://hg.mozilla.org/mozilla-central/rev/50e33b2149ff
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla35
(In reply to TBPL Robot from comment #169)
Sadly, it is back. Let's keep an eye on how 1059265 progresses. There might multiple causes for this bug...
Status: RESOLVED → REOPENED
status-b2g-v2.0: affected → wontfix
status-b2g-v2.0M: --- → wontfix
status-b2g-v2.1: --- → affected
status-b2g-v2.2: --- → affected
status-firefox31: affected → wontfix
status-firefox32: affected → wontfix
status-firefox33: --- → affected
status-firefox34: --- → affected
status-firefox35: --- → affected
Resolution: FIXED → ---
Target Milestone: mozilla35 → ---
(In reply to TBPL Robot from comment #169)
17:13:09     INFO -   0  CoreAudio + 0x1c7fe
17:13:09     INFO -      rbx = 0x0000000125ed4800   r12 = 0x0000000125ed4800
17:13:09     INFO -      r13 = 0x0000000000000000   r14 = 0x00000001261f9e00
17:13:09     INFO -      r15 = 0x00000001538b754c   rip = 0x00007fff878247fe
17:13:09     INFO -      rsp = 0x00000001488d5360   rbp = 0x00000001488d5360
17:13:09     INFO -      Found by: given as instruction pointer in context
17:13:09     INFO -   1  CoreAudio + 0x1c762
17:13:09     INFO -      rip = 0x00007fff87824763   rsp = 0x00000001488d5370
17:13:09     INFO -      rbp = 0x00000001488d53b0
17:13:09     INFO -      Found by: stack scanning
17:13:09     INFO -   2  libmozglue.dylib!arena_malloc [_string.h : 83 + 0xf]
17:13:09     INFO -      rip = 0x00000001000108b0   rsp = 0x00000001488d5380
17:13:09     INFO -      rbp = 0x00000001488d53b0
17:13:09     INFO -      Found by: stack scanning

I am curious how arena_malloc could call into CoreAudio? I start to wonder if this issue is about some kind of OOM.
Assignee: kinetik → nobody
It looks to me that all failures since Jan. 15th 2014 is from Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound opt test crashtest.  It doesn't appear that the bug affects b2g anymore? Also, comment 167 seems to indicate that mac os tests should have been disabled. Or am I misinterpreting the comments?
> that the bug affects b2g anymore? Also, comment 167 seems to indicate that
> mac os tests should have been disabled. Or am I misinterpreting the comments?
Sorry, that was Comment 197.
QA Whiteboard: QAExclude
I am following up on Comments 221 and 222 to see if there are action items for this issue. Do the tests for mac still need to be disabled? NI :RyanVM
Flags: needinfo?(ryanvm)
Looks like we disabled our way to victory here. Ultimately, we should probably just disable any tests on 10.6 that hit [@ CoreAudio + 0x1c7fe] crashes, since the basic gist of the problem is that it simply can't handle them reliably.

Matt, would you agree with that statement?
Flags: needinfo?(ryanvm) → needinfo?(kinetik)
Yeah, sadly, there's nothing better we can do for now.
Flags: needinfo?(kinetik)
https://hg.mozilla.org/mozilla-central/rev/4824c5c9cb38

Going to go ahead and resolve this under the assumption that this will be the last we think of this until after 10.6 is shut off permanently.
Assignee: nobody → ryanvm
Status: REOPENED → RESOLVED
Closed: 9 years ago → 8 years ago
status-firefox40: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla40
You need to log in before you can comment on or make changes to this bug.