Closed
Bug 1741118
Opened 3 years ago
Closed 2 years ago
Intermittent gtest | application crashed [@ webrtc::internal::Call::OnTargetTransferRate(webrtc::TargetTransferRate)]
Categories
(Core :: WebRTC: Signaling, defect, P1)
Tracking
()
RESOLVED
FIXED
96 Branch
| Tracking | Status | |
|---|---|---|
| firefox-esr91 | --- | unaffected |
| firefox94 | --- | unaffected |
| firefox95 | --- | unaffected |
| firefox96 | --- | fixed |
People
(Reporter: imoraru, Assigned: pehrsons)
References
Details
(Keywords: csectype-uaf, intermittent-failure, sec-high, Whiteboard: [sec-survey])
Crash Data
Attachments
(3 files)
[task 2021-11-15T04:43:50.174Z] 04:43:50 INFO - TEST-START | MediaPipelineTest.TestAudioSendBundle
[task 2021-11-15T04:43:50.516Z] 04:43:50 INFO - gtest INFO | gtest | process wait complete, returncode=1
[task 2021-11-15T04:43:50.517Z] 04:43:50 INFO - mozcrash checking Z:\task_163695058515494\build\tests\gtest for minidumps...
[task 2021-11-15T04:43:50.518Z] 04:43:50 INFO - mozcrash INFO | Copy/paste: Z:/task_163695058515494/fetches\minidump_stackwalk\minidump_stackwalk.exe Z:\task_163695058515494\build\tests\gtest\ee1828c3-3342-4f5f-bf76-95306a3d7b2d.dmp Z:\task_163695058515494\build\symbols https://symbols.mozilla.org/
[task 2021-11-15T04:44:03.077Z] 04:44:03 INFO - mozcrash INFO | Saved minidump as Z:\task_163695058515494\build\blobber_upload_dir\ee1828c3-3342-4f5f-bf76-95306a3d7b2d.dmp
[task 2021-11-15T04:44:03.087Z] 04:44:03 INFO - mozcrash INFO | Saved app info as Z:\task_163695058515494\build\blobber_upload_dir\ee1828c3-3342-4f5f-bf76-95306a3d7b2d.extra
[task 2021-11-15T04:44:03.089Z] 04:44:03 WARNING - PROCESS-CRASH | gtest | application crashed [@ webrtc::internal::Call::OnTargetTransferRate(webrtc::TargetTransferRate)]
[task 2021-11-15T04:44:03.090Z] 04:44:03 INFO - Crash dump filename: Z:\task_163695058515494\build\tests\gtest\ee1828c3-3342-4f5f-bf76-95306a3d7b2d.dmp
[task 2021-11-15T04:44:03.090Z] 04:44:03 INFO - Operating system: Windows NT
[task 2021-11-15T04:44:03.090Z] 04:44:03 INFO - 10.0.19041
[task 2021-11-15T04:44:03.090Z] 04:44:03 INFO - CPU: x86
[task 2021-11-15T04:44:03.090Z] 04:44:03 INFO - GenuineIntel family 6 model 85 stepping 4
[task 2021-11-15T04:44:03.091Z] 04:44:03 INFO - 8 CPUs
[task 2021-11-15T04:44:03.091Z] 04:44:03 INFO - GPU: UNKNOWN
[task 2021-11-15T04:44:03.091Z] 04:44:03 INFO - Crash reason: EXCEPTION_ACCESS_VIOLATION_READ
[task 2021-11-15T04:44:03.091Z] 04:44:03 INFO - Crash address: 0xe5e5e605
[task 2021-11-15T04:44:03.091Z] 04:44:03 INFO - Process uptime: 306 seconds
[task 2021-11-15T04:44:03.091Z] 04:44:03 INFO - Thread 63 tid 8972 (crashed) - WebrtcWorker #2 0 xul.dll!mozilla::TaskQueueWrapper::PostTask(std::unique_ptr<webrtc::QueuedTask,std::default_delete<webrtc::QueuedTask> >) [TaskQueueWrapper.h:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 99 + 0x9]
[task 2021-11-15T04:44:03.092Z] 04:44:03 INFO - eip = 0x64418b06 esp = 0x16aaf48c ebp = 0x16aaf4b4 ebx = 0x0cbdc080
[task 2021-11-15T04:44:03.092Z] 04:44:03 INFO - esi = 0x0cb71040 edi = 0x16aaf48c eax = 0xe5e5e5e5 ecx = 0xdd43f622
[task 2021-11-15T04:44:03.092Z] 04:44:03 INFO - edx = 0x0cb4d200 efl = 0x00010202
[task 2021-11-15T04:44:03.092Z] 04:44:03 INFO - Found by: given as instruction pointer in context
[task 2021-11-15T04:44:03.093Z] 04:44:03 INFO - 1 xul.dll!webrtc::internal::Call::OnTargetTransferRate(webrtc::TargetTransferRate) [call.cc:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 1190 + 0x11]
[task 2021-11-15T04:44:03.093Z] 04:44:03 INFO - eip = 0x69e02b80 esp = 0x16aaf4bc ebp = 0x16aaf550 ebx = 0x0cb721c0
[task 2021-11-15T04:44:03.093Z] 04:44:03 INFO - esi = 0x16aaf4c0 edi = 0x64418a70
[task 2021-11-15T04:44:03.093Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.094Z] 04:44:03 INFO - 2 xul.dll!webrtc::RtpTransportControllerSend::UpdateControlState() [rtp_transport_controller_send.cc:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 185 + 0x4c]
[task 2021-11-15T04:44:03.094Z] 04:44:03 INFO - eip = 0x69e1e305 esp = 0x16aaf558 ebp = 0x16aaf614 ebx = 0x00000001
[task 2021-11-15T04:44:03.094Z] 04:44:03 INFO - esi = 0x0ff1e40c edi = 0x69e02a20
[task 2021-11-15T04:44:03.094Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.095Z] 04:44:03 INFO - 3 xul.dll!webrtc::RtpTransportControllerSend::OnNetworkAvailability::<lambda_3>::operator()() const [rtp_transport_controller_send.cc:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 372 + 0x7]
[task 2021-11-15T04:44:03.095Z] 04:44:03 INFO - eip = 0x69e2aba1 esp = 0x16aaf61c ebp = 0x16aaf650 ebx = 0x69db61e0
[task 2021-11-15T04:44:03.096Z] 04:44:03 INFO - esi = 0x07203900 edi = 0x0fcc8810
[task 2021-11-15T04:44:03.096Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.096Z] 04:44:03 INFO - 4 xul.dll!webrtc::webrtc_new_closure_impl::ClosureTask<`lambda at /builds/worker/checkouts/gecko/third_party/libwebrtc/call/rtp_transport_controller_send.cc:357:24'>::Run() [to_queued_task.h:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 32 + 0x8]
[task 2021-11-15T04:44:03.096Z] 04:44:03 INFO - eip = 0x69e2aa8b esp = 0x16aaf658 ebp = 0x16aaf658 ebx = 0x69e2aa80
[task 2021-11-15T04:44:03.097Z] 04:44:03 INFO - esi = 0x0fc94208 edi = 0x0fcc8800
[task 2021-11-15T04:44:03.097Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.097Z] 04:44:03 INFO - 5 xul.dll!mozilla::detail::RunnableFunction<`lambda at /builds/worker/checkouts/gecko/dom/media/webrtc/libwebrtcglue/TaskQueueWrapper.h:72:35'>::Run() [nsThreadUtils.h:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 531 + 0x34]
[task 2021-11-15T04:44:03.097Z] 04:44:03 INFO - eip = 0x6441906d esp = 0x16aaf660 ebp = 0x16aaf674
[task 2021-11-15T04:44:03.098Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.098Z] 04:44:03 INFO - 6 xul.dll!mozilla::TaskQueue::Runner::Run() [TaskQueue.cpp:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 208 + 0xe]
[task 2021-11-15T04:44:03.098Z] 04:44:03 INFO - eip = 0x650eb70f esp = 0x16aaf67c ebp = 0x16aaf6e4 ebx = 0x00000000
[task 2021-11-15T04:44:03.099Z] 04:44:03 INFO - esi = 0x0fcc8700 edi = 0x16aaf68c
[task 2021-11-15T04:44:03.099Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.099Z] 04:44:03 INFO - 7 xul.dll!nsThreadPool::Run() [nsThreadPool.cpp:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 305 + 0xe]
[task 2021-11-15T04:44:03.099Z] 04:44:03 INFO - eip = 0x651048c2 esp = 0x16aaf6ec ebp = 0x16aaf824 ebx = 0x16aaf7a0
[task 2021-11-15T04:44:03.099Z] 04:44:03 INFO - esi = 0x0fcc8500 edi = 0x00000000
[task 2021-11-15T04:44:03.100Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.100Z] 04:44:03 INFO - 8 xul.dll!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 1169 + 0xe]
[task 2021-11-15T04:44:03.100Z] 04:44:03 INFO - eip = 0x650fb925 esp = 0x16aaf82c ebp = 0x16aaf960 ebx = 0x0ffd1bb0
[task 2021-11-15T04:44:03.100Z] 04:44:03 INFO - esi = 0x0ec13160 edi = 0x0ffd1c98
[task 2021-11-15T04:44:03.100Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.101Z] 04:44:03 INFO - 9 xul.dll!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 467 + 0x16]
[task 2021-11-15T04:44:03.101Z] 04:44:03 INFO - eip = 0x65102491 esp = 0x16aaf968 ebp = 0x16aaf984 ebx = 0x658bf500
[task 2021-11-15T04:44:03.101Z] 04:44:03 INFO - esi = 0x0ffd1bb0 edi = 0x16aafa50
[task 2021-11-15T04:44:03.101Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.102Z] 04:44:03 INFO - 10 xul.dll!mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [MessagePump.cpp:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 300 + 0xa]
[task 2021-11-15T04:44:03.102Z] 04:44:03 INFO - eip = 0x65929c91 esp = 0x16aaf98c ebp = 0x16aaf9b8 ebx = 0x658bf550
[task 2021-11-15T04:44:03.102Z] 04:44:03 INFO - esi = 0x16b42580
[task 2021-11-15T04:44:03.102Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.102Z] 04:44:03 INFO - 11 xul.dll!MessageLoop::RunInternal() [message_loop.cc:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 331 + 0x11]
[task 2021-11-15T04:44:03.102Z] 04:44:03 INFO - eip = 0x658be6a3 esp = 0x16aaf9c0 ebp = 0x16aaf9e4 ebx = 0x65929bd0
[task 2021-11-15T04:44:03.103Z] 04:44:03 INFO - esi = 0x16aafa50 edi = 0x16b42580
[task 2021-11-15T04:44:03.103Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.103Z] 04:44:03 INFO - 12 xul.dll!MessageLoop::RunHandler() [message_loop.cc:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 324 + 0x7]
[task 2021-11-15T04:44:03.103Z] 04:44:03 INFO - eip = 0x658be5a3 esp = 0x16aaf9ec ebp = 0x16aafa18 ebx = 0x102e6220
[task 2021-11-15T04:44:03.103Z] 04:44:03 INFO - esi = 0x16aafa50 edi = 0x16aafa50
[task 2021-11-15T04:44:03.103Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.104Z] 04:44:03 INFO - 13 xul.dll!MessageLoop::Run() [message_loop.cc:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 306 + 0x5]
[task 2021-11-15T04:44:03.104Z] 04:44:03 INFO - eip = 0x658be456 esp = 0x16aafa20 ebp = 0x16aafa38 ebx = 0x102e6220
[task 2021-11-15T04:44:03.104Z] 04:44:03 INFO - esi = 0x0ffd1bb0 edi = 0x16aafa50
[task 2021-11-15T04:44:03.104Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.104Z] 04:44:03 INFO - 14 xul.dll!static nsThread::ThreadFunc(void*) [nsThread.cpp:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 391 + 0x7]
[task 2021-11-15T04:44:03.105Z] 04:44:03 INFO - eip = 0x650f79d7 esp = 0x16aafa40 ebp = 0x16aafb44
[task 2021-11-15T04:44:03.105Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.105Z] 04:44:03 INFO - 15 nss3.dll!_PR_NativeRunThread(void*) [pruthr.c:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 399 + 0xf]
[task 2021-11-15T04:44:03.105Z] 04:44:03 INFO - eip = 0x71c7bff8 esp = 0x16aafb4c ebp = 0x16aafb6c ebx = 0x1757d258
[task 2021-11-15T04:44:03.105Z] 04:44:03 INFO - esi = 0x102e6220 edi = 0x0cb82eb0
[task 2021-11-15T04:44:03.106Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.106Z] 04:44:03 INFO - 16 nss3.dll!pr_root(void*) [w95thred.c:64bbfa36deb940e7a04409a4f8a5f178baa7a5b1 : 139 + 0xf]
[task 2021-11-15T04:44:03.106Z] 04:44:03 INFO - eip = 0x71c6d8b6 esp = 0x16aafb74 ebp = 0x16aafb7c ebx = 0x1757d258
[task 2021-11-15T04:44:03.106Z] 04:44:03 INFO - esi = 0x0cb82eb0 edi = 0x1757d258
[task 2021-11-15T04:44:03.106Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.106Z] 04:44:03 INFO - 17 ucrtbase.dll + 0x44f9f
[task 2021-11-15T04:44:03.107Z] 04:44:03 INFO - eip = 0x77094f9f esp = 0x16aafb84 ebp = 0x16aafbb4 esi = 0x71c6d8a0
[task 2021-11-15T04:44:03.107Z] 04:44:03 INFO - Found by: call frame info
[task 2021-11-15T04:44:03.107Z] 04:44:03 INFO - 18 kernel32.dll + 0x1fa29
[task 2021-11-15T04:44:03.107Z] 04:44:03 INFO - eip = 0x75c3fa29 esp = 0x16aafbbc ebp = 0x16aafbc4
[task 2021-11-15T04:44:03.107Z] 04:44:03 INFO - Found by: previous frame's frame pointer
[task 2021-11-15T04:44:03.107Z] 04:44:03 INFO - 19 ntdll.dll + 0x67a9e
[task 2021-11-15T04:44:03.108Z] 04:44:03 INFO - eip = 0x77707a9e esp = 0x16aafbcc ebp = 0x16aafc20
[task 2021-11-15T04:44:03.108Z] 04:44:03 INFO - Found by: previous frame's frame pointer
[task 2021-11-15T04:44:03.108Z] 04:44:03 INFO - 20 ntdll.dll + 0x67a6e
[task 2021-11-15T04:44:03.108Z] 04:44:03 INFO - eip = 0x77707a6e esp = 0x16aafc28 ebp = 0x16aafc30
[task 2021-11-15T04:44:03.108Z] 04:44:03 INFO - Found by: previous frame's frame pointer
|
Reporter | |
Updated•3 years ago
|
Crash Signature: [@ webrtc::internal::Call::OnTargetTransferRate(webrtc::TargetTransferRate)]
|
||
Updated•3 years ago
|
Group: core-security → media-core-security
|
|
||
Updated•2 years ago
|
Keywords: csectype-uaf,
sec-high
|
||
Comment 1•2 years ago
|
||
Byron, is this the one you thought might be harness related?
Flags: needinfo?(docfaraday)
|
||
Updated•2 years ago
|
Blocks: webrtc-triage
|
Assignee | |
Comment 2•2 years ago
|
||
I suspect this happens because TaskQueueWrapper::Delete in some cases (one being when deleting the MainAsCurrent test-helper in MediaPipelineTest) doesn't block until the TaskQueue has been drained.
So it has to guarantee its own lifetime, but it also has to block the caller so the caller doesn't go away before it has finished shutting down.
S3 because we have no proof this happens in the wild.
Assignee: nobody → apehrson
Severity: -- → S3
Priority: -- → P1
|
|
Assignee | |
Updated•2 years ago
|
Status: NEW → ASSIGNED
|
|
Assignee | |
Comment 3•2 years ago
|
||
When writing the patches for this, I first of all disallowed destroying the TaskQueueWrapper on its own thread, and the only path on m-c that led there was when destroying GmpPluginNotifiers and disconnecting their MediaEvents async on the call thread (which released refs to their target thread, i.e., the call thread). In this case the Call instance is guaranteed to be torn down so we're free from UAFs.
Actually we can generalize this. WebrtcCallWrapper guarantees that the const call thread outlives the Call instance. The Call instance has a raw ptr to the call thread, all Mozilla uses the ref-counted AbstractThread interface of it.
In conclusion this is a test-only UAF that MediaPipelineTest can hit.
|
|
Assignee | |
Comment 4•2 years ago
|
||
|
|
Assignee | |
Comment 5•2 years ago
|
||
This is prohibited for regular TaskQueueWrappers, as Delete must join (i.e.
block) the thread after initiating shutdown. But MainAsCurrent is special as
main thread doesn't go away just because MainAsCurrent goes away. It is also
gtest-only, and thus tightly coupled with MediaPipelineTest behavior.
|
|
Assignee | |
Comment 6•2 years ago
|
||
Since bug 1727262 these are fine to release off-target.
|
||
Comment 7•2 years ago
|
||
(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #1)
Byron, is this the one you thought might be harness related?
No, that was a different bug, but it looks like this is test-only too based on comment 3.
Flags: needinfo?(docfaraday)
|
|
||
Comment 8•2 years ago
|
||
Backed out changeset 0920243940ad. r=ng
https://hg.mozilla.org/integration/autoland/rev/a5f8a8841c67ee6af474cda2bdb002a73a136824
https://hg.mozilla.org/mozilla-central/rev/a5f8a8841c67
Allow the MainAsCurrent TaskQueueWrapper to be deleted on itself. r=ng
https://hg.mozilla.org/integration/autoland/rev/a650d6d32d60f4c9c5888dfa8d5ab1fb8ef2a45f
https://hg.mozilla.org/mozilla-central/rev/a650d6d32d60
Don't release GmpPlugin MediaEvents, which hold on to the call thread, on the call thread. r=ng
https://hg.mozilla.org/integration/autoland/rev/a6b6b7f5fca01437af25eb1434dddfa20f0f4067
https://hg.mozilla.org/mozilla-central/rev/a6b6b7f5fca0
Group: media-core-security → core-security-release
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 96 Branch
|
||
Comment 9•2 years ago
|
||
As part of a security bug pattern analysis, we are requesting your help with a high level analysis of this bug. It is our hope to develop static analysis (or potentially runtime/dynamic analysis) in the future to identify classes of bugs.
Please visit this google form to reply.
Flags: needinfo?(apehrson)
Whiteboard: [sec-survey]
|
|
Assignee | |
Updated•2 years ago
|
Flags: needinfo?(apehrson)
|
||
Updated•2 years ago
|
Flags: qe-verify-
|
|
||
Updated•2 years ago
|
No longer blocks: webrtc-triage
|
|
||
Updated•2 years ago
|
Group: core-security-release
|
|
||
Comment 11•1 year ago
|
||
Copying crash signatures from duplicate bugs.
Crash Signature: [@ webrtc::internal::Call::OnTargetTransferRate(webrtc::TargetTransferRate)] → [@ webrtc::internal::Call::OnTargetTransferRate(webrtc::TargetTransferRate)]
[@ webrtc::webrtc_repeating_task_impl::RepeatingTaskImpl<`lambda at /builds/worker/checkouts/gecko/third_party/libwebrtc/call/rtp_transport_controller_send.cc:614:55'>::RunClosur…
You need to log in
before you can comment on or make changes to this bug.
Description
•