Closed Bug 1456071 Opened 2 years ago Closed 2 years ago

Permanent false-positive /webrtc/RTCDTMFSender-insertDTMF.https.html | application crashed [@ mozalloc_abort][@ webrtc::voe::ChannelProxy::SendTelephoneEventOutband]

Categories

(Core :: WebRTC: Audio/Video, defect, P2, critical)

defect

Tracking

()

RESOLVED FIXED
mozilla62
Tracking Status
firefox-esr52 --- unaffected
firefox-esr60 --- fixed
firefox59 --- wontfix
firefox60 --- wontfix
firefox61 --- fixed
firefox62 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: dminor)

References

Details

(Keywords: crash, intermittent-failure)

Crash Data

Attachments

(2 files)

Filed by: hskupin [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=174091558&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/A-0l1cxiSUy11AfdRjvPUQ/runs/0/artifacts/public/logs/live_backing.log

Here the first 10 frames from the stack:

[task 2018-04-17T14:02:23.533Z] 14:02:23     INFO - Crash reason:  SIGSEGV
[task 2018-04-17T14:02:23.534Z] 14:02:23     INFO - Crash address: 0x0
[task 2018-04-17T14:02:23.534Z] 14:02:23     INFO - Process uptime: not available
[task 2018-04-17T14:02:23.535Z] 14:02:23     INFO - 
[task 2018-04-17T14:02:23.535Z] 14:02:23     INFO - Thread 7 (crashed)
[task 2018-04-17T14:02:23.536Z] 14:02:23     INFO -  0  firefox!mozalloc_abort [mozalloc_abort.cpp:a5aa0c6be992161aa110cfe91ad813a81a81767f : 34 + 0x5]
[task 2018-04-17T14:02:23.536Z] 14:02:23     INFO -     rax = 0x0000000000000000   rdx = 0x0000000000000000
[task 2018-04-17T14:02:23.537Z] 14:02:23     INFO -     rcx = 0x00007fc5caf972dd   rbx = 0x00007fc5cb265700
[task 2018-04-17T14:02:23.538Z] 14:02:23     INFO -     rsi = 0x00007fc5cb266770   rdi = 0x00007fc5cb265540
[task 2018-04-17T14:02:23.539Z] 14:02:23     INFO -     rbp = 0x00007fc5b2dfde30   rsp = 0x00007fc5b2dfde20
[task 2018-04-17T14:02:23.539Z] 14:02:23     INFO -      r8 = 0x00007fc5cb266770    r9 = 0x00007fc5b2dff700
[task 2018-04-17T14:02:23.540Z] 14:02:23     INFO -     r10 = 0x0000000000000058   r11 = 0x00007fc5caf0d7a0
[task 2018-04-17T14:02:23.540Z] 14:02:23     INFO -     r12 = 0x00007fc5cb265700   r13 = 0x00007fc5b2dfde58
[task 2018-04-17T14:02:23.540Z] 14:02:23     INFO -     r14 = 0x0000000000000064   r15 = 0x0000000000000000
[task 2018-04-17T14:02:23.540Z] 14:02:23     INFO -     rip = 0x000000000041bd6a
[task 2018-04-17T14:02:23.540Z] 14:02:23     INFO -     Found by: given as instruction pointer in context
[task 2018-04-17T14:02:23.541Z] 14:02:23     INFO -  1  firefox!abort [mozalloc_abort.cpp:a5aa0c6be992161aa110cfe91ad813a81a81767f : 81 + 0x5]
[task 2018-04-17T14:02:23.541Z] 14:02:23     INFO -     rbx = 0x00007fc5b2dfde90   rbp = 0x00007fc5b2dfde40
[task 2018-04-17T14:02:23.542Z] 14:02:23     INFO -     rsp = 0x00007fc5b2dfde40   r12 = 0x00007fc5cb265700
[task 2018-04-17T14:02:23.542Z] 14:02:23     INFO -     r13 = 0x00007fc5b2dfde58   r14 = 0x0000000000000064
[task 2018-04-17T14:02:23.542Z] 14:02:23     INFO -     r15 = 0x0000000000000000   rip = 0x000000000041bd40
[task 2018-04-17T14:02:23.543Z] 14:02:23     INFO -     Found by: call frame info
[task 2018-04-17T14:02:23.544Z] 14:02:23     INFO -  2  libxul.so!rtc::FatalMessage::~FatalMessage [checks.cc:a5aa0c6be992161aa110cfe91ad813a81a81767f : 109 + 0x5]
[task 2018-04-17T14:02:23.544Z] 14:02:23     INFO -     rbx = 0x00007fc5b2dfde90   rbp = 0x00007fc5b2dfde80
[task 2018-04-17T14:02:23.544Z] 14:02:23     INFO -     rsp = 0x00007fc5b2dfde50   r12 = 0x00007fc5cb265700
[task 2018-04-17T14:02:23.545Z] 14:02:23     INFO -     r13 = 0x00007fc5b2dfde58   r14 = 0x0000000000000064
[task 2018-04-17T14:02:23.546Z] 14:02:23     INFO -     r15 = 0x0000000000000000   rip = 0x00007fc5bc1a45d2
[task 2018-04-17T14:02:23.546Z] 14:02:23     INFO -     Found by: call frame info
[task 2018-04-17T14:02:23.547Z] 14:02:23     INFO -  3  libxul.so!webrtc::voe::ChannelProxy::SendTelephoneEventOutband [channel_proxy.cc:a5aa0c6be992161aa110cfe91ad813a81a81767f : 189 + 0x17]
[task 2018-04-17T14:02:23.547Z] 14:02:23     INFO -     rbp = 0x00007fc5b2dfe010   rsp = 0x00007fc5b2dfde90
[task 2018-04-17T14:02:23.547Z] 14:02:23     INFO -     rip = 0x00007fc5bc2c92fd
[task 2018-04-17T14:02:23.548Z] 14:02:23     INFO -     Found by: previous frame's frame pointer
[task 2018-04-17T14:02:23.548Z] 14:02:23     INFO -  4  libxul.so!mozilla::WebrtcAudioConduit::InsertDTMFTone [AudioConduit.cpp:a5aa0c6be992161aa110cfe91ad813a81a81767f : 277 + 0xf]
[task 2018-04-17T14:02:23.549Z] 14:02:23     INFO -     rbp = 0x00007fc5b2dfe040   rsp = 0x00007fc5b2dfe020
[task 2018-04-17T14:02:23.549Z] 14:02:23     INFO -     rip = 0x00007fc5ba1bfb27
[task 2018-04-17T14:02:23.549Z] 14:02:23     INFO -     Found by: previous frame's frame pointer
[task 2018-04-17T14:02:23.550Z] 14:02:23     INFO -  5  libxul.so!mozilla::runnable_args_func<mozilla::TransceiverImpl::InsertDTMFTone(int, uint32_t)::<lambda()> >::Run [TransceiverImpl.cpp:a5aa0c6be992161aa110cfe91ad813a81a81767f : 582 + 0x26]
[task 2018-04-17T14:02:23.550Z] 14:02:23     INFO -     rbp = 0x00007fc5b2dfe070   rsp = 0x00007fc5b2dfe050
[task 2018-04-17T14:02:23.551Z] 14:02:23     INFO -     rip = 0x00007fc5ba1ea1d7
[task 2018-04-17T14:02:23.551Z] 14:02:23     INFO -     Found by: previous frame's frame pointer
[task 2018-04-17T14:02:23.552Z] 14:02:23     INFO -  6  libxul.so!nsThread::ProcessNextEvent [nsThread.cpp:a5aa0c6be992161aa110cfe91ad813a81a81767f : 1096 + 0x15]
[task 2018-04-17T14:02:23.552Z] 14:02:23     INFO -     rbp = 0x00007fc5b2dfe650   rsp = 0x00007fc5b2dfe080
[task 2018-04-17T14:02:23.552Z] 14:02:23     INFO -     rip = 0x00007fc5b9ab5ab6
[task 2018-04-17T14:02:23.553Z] 14:02:23     INFO -     Found by: previous frame's frame pointer
[task 2018-04-17T14:02:23.553Z] 14:02:23     INFO -  7  libxul.so!NS_ProcessNextEvent [nsThreadUtils.cpp:a5aa0c6be992161aa110cfe91ad813a81a81767f : 519 + 0x11]
[task 2018-04-17T14:02:23.553Z] 14:02:23     INFO -     rbp = 0x00007fc5b2dfe680   rsp = 0x00007fc5b2dfe660
[task 2018-04-17T14:02:23.554Z] 14:02:23     INFO -     rip = 0x00007fc5b9ab78d1
[task 2018-04-17T14:02:23.554Z] 14:02:23     INFO -     Found by: previous frame's frame pointer
[task 2018-04-17T14:02:23.555Z] 14:02:23     INFO -  8  libxul.so!mozilla::net::nsSocketTransportService::Run [nsSocketTransportService2.cpp:a5aa0c6be992161aa110cfe91ad813a81a81767f : 1007 + 0x11]
[task 2018-04-17T14:02:23.556Z] 14:02:23     INFO -     rbp = 0x00007fc5b2dfe770   rsp = 0x00007fc5b2dfe690
[task 2018-04-17T14:02:23.556Z] 14:02:23     INFO -     rip = 0x00007fc5b9b640db
[task 2018-04-17T14:02:23.556Z] 14:02:23     INFO -     Found by: previous frame's frame pointer
[task 2018-04-17T14:02:23.557Z] 14:02:23     INFO -  9  libxul.so!nsThread::ProcessNextEvent [nsThread.cpp:a5aa0c6be992161aa110cfe91ad813a81a81767f : 1096 + 0x15]
[task 2018-04-17T14:02:23.557Z] 14:02:23     INFO -     rbp = 0x00007fc5b2dfed50   rsp = 0x00007fc5b2dfe780
[task 2018-04-17T14:02:23.558Z] 14:02:23     INFO -     rip = 0x00007fc5b9ab5ab6
[task 2018-04-17T14:02:23.558Z] 14:02:23     INFO -     Found by: previous frame's frame pointer
[task 2018-04-17T14:02:23.559Z] 14:02:23     INFO - 10  libxul.so!NS_ProcessNextEvent [nsThreadUtils.cpp:a5aa0c6be992161aa110cfe91ad813a81a81767f : 519 + 0x11]
[task 2018-04-17T14:02:23.559Z] 14:02:23     INFO -     rbp = 0x00007fc5b2dfed80   rsp = 0x00007fc5b2dfed60
[task 2018-04-17T14:02:23.560Z] 14:02:23     INFO -     rip = 0x00007fc5b9ab78d1
This is failing the thread check at: https://searchfox.org/mozilla-central/source/media/webrtc/trunk/webrtc/voice_engine/channel_proxy.cc#189, which is not expected. No reports in the wild yet, but that could be because DTMF is not very widely used.
Rank: 25
Priority: P5 → P3
Duplicate of this bug: 1456339
Duplicate of this bug: 1456640
Conveniently, it's not an intermittent failure, it happens 100% of the time. The wpt harness is just weird about 100% failure, annotating crashes as expected and not turning the run orange because they are expected. It'll only be starred when someone ignores the actual other failure which turned a run orange (which in the case from comment 0 was a timeout redownloading symbols for the second identical 100% thread check failure in RTCDTMFSender-ontonechange-long.https.html), ignores the "Permanent false-positive", and stars it anyway, but it's still 100% reproducible.
Summary: Intermittent /webrtc/RTCDTMFSender-insertDTMF.https.html | application crashed [@ mozalloc_abort][@ webrtc::voe::ChannelProxy::SendTelephoneEventOutband] → Permanent false-positive /webrtc/RTCDTMFSender-insertDTMF.https.html | application crashed [@ mozalloc_abort][@ webrtc::voe::ChannelProxy::SendTelephoneEventOutband]
Duplicate of this bug: 1456967
I'll either fix this or get it disabled so we no longer waste everyone's time.
Assignee: nobody → dminor
It appears this is actually a regression from the branch 57 update (Bug 1341285). We switched from using webrtc::VoiceEngineImpl to the ChannelProxy to send the event. We deliberately use the STS thread to send, but all of the ChannelProxy methods expect to be called from the main thread.

The DTMF mochitest completes before the crashing code actually runs, but the web-platform test is slow enough that it will hit the assertion and crash. The example at https://webrtc.github.io/samples/src/content/peerconnection/dtmf/ crashes at the first tone in a debug build.

Since we end up enqueueing the data anyway [1], I think we can safely remove the dispatch to the STS thread to call this method.

[1] https://searchfox.org/mozilla-central/rev/78dbe34925f04975f16cb9a5d4938be714d41897/media/webrtc/trunk/webrtc/modules/rtp_rtcp/source/rtp_sender_audio.cc#294
Blocks: 1341285
Status: NEW → ASSIGNED
Rank: 25 → 15
Priority: P3 → P2
Comment on attachment 8971322 [details]
Bug 1456071 - Do not dispatch InsertDTMFTone call to sts thread;

https://reviewboard.mozilla.org/r/240088/#review245874
Attachment #8971322 - Flags: review?(docfaraday) → review+
Blocks: 1456339
Try run here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bfedf210501a2242dd907548da8e28fb0738f258

Forgot to update the test expectations on the original patch.
Comment on attachment 8972033 [details]
Bug 1456071 - Update web-platform test expectations;

https://reviewboard.mozilla.org/r/240768/#review246626
Attachment #8972033 - Flags: review?(docfaraday) → review+
Pushed by dminor@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4f2ea3e9aa4d
Do not dispatch InsertDTMFTone call to sts thread; r=bwc
https://hg.mozilla.org/integration/mozilla-inbound/rev/fb2b32cae6e8
Update web-platform test expectations; r=bwc
https://hg.mozilla.org/mozilla-central/rev/4f2ea3e9aa4d
https://hg.mozilla.org/mozilla-central/rev/fb2b32cae6e8
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Please request Beta and ESR60 approval on this when you get a chance (it would be nice to reduce the noise on a branch we'll be starring for the next year).
Flags: needinfo?(dminor)
Comment on attachment 8971322 [details]
Bug 1456071 - Do not dispatch InsertDTMFTone call to sts thread;

Approval Request Comment
[Feature/Bug causing the regression]:
Bug 1341285
[User impact if declined]:
We're dispatching the call on the incorrect thread. This could potentially cause crashes or deadlocks, but as far as I know none have been seen on release.
[Is this code covered by automated tests?]:
Yes.
[Has the fix been verified in Nightly?]:
Yes.
[Needs manual test from QE? If yes, steps to reproduce]: 
No.
[List of other uplifts needed for the feature/fix]:
None.
[Is the change risky?]:
No.
[Why is the change risky/not risky?]:
This just removes an incorrect thread dispatch call.
[String changes made/needed]:
None.
Flags: needinfo?(dminor)
Attachment #8971322 - Flags: approval-mozilla-beta?
Comment on attachment 8971322 [details]
Bug 1456071 - Do not dispatch InsertDTMFTone call to sts thread;

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration:
This ends up being starred as an intermittent failure so having this on ESR would reduce workload in sheriffing. The code is also incorrect although as far as I know, we have not seen any problems as a result of using the wrong thread in release.
User impact if declined: 
Could potentially cause crashes or deadlock.
Fix Landed on Version:
62.
Risk to taking this patch (and alternatives if risky): 
Very low.
String or UUID changes made by this patch: 
None.

See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more info.
Attachment #8971322 - Flags: approval-mozilla-esr60?
Comment on attachment 8971322 [details]
Bug 1456071 - Do not dispatch InsertDTMFTone call to sts thread;

Fixes noise in our test automation. Approved for Beta and ESR60.
Attachment #8971322 - Flags: approval-mozilla-esr60?
Attachment #8971322 - Flags: approval-mozilla-esr60+
Attachment #8971322 - Flags: approval-mozilla-beta?
Attachment #8971322 - Flags: approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.