Closed Bug 1721443 Opened 3 years ago Closed 3 years ago

Intermittent gtest | application crashed [@ FrameListener::OnVideoFrameConverted(webrtc::VideoFrame const&)]

Categories

(Core :: WebRTC: Audio/Video, defect, P2)

Product:
Core
Component:
WebRTC: Audio/Video
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
96 Branch
Tracking Flags:
Tracking Status
firefox-esr91 --- wontfix
firefox93 --- wontfix
firefox94 --- wontfix
firefox95 --- wontfix
firefox96 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: pehrsons)

References

Details

(Keywords: crash, csectype-uaf, intermittent-failure, Whiteboard: [test only])

Crash Data

Attachments

(3 files, 3 obsolete files)

Bug 1721443 - Implement gtest helper TakeN that return a MozPromise wrapping the first N events from a MediaEventSource. r?bwc
48 bytes, text/x-phabricator-request
Details | Review
Bug 1721443 - Make WaitUntil handle multi-param MediaEvents. r?bwc!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1721443 - Use higher order functions for getting frames in TestVideoFrameConverter. r?bwc
48 bytes, text/x-phabricator-request
Details | Review

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=345772107&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/L_QI7PqoQN2KkiuzpYcj3Q/runs/0/artifacts/public/logs/live_backing.log

[task 2021-07-20T17:30:45.860Z] 17:30:45     INFO -  TEST-PASS | libvpx.test_cases | test completed (time: 12ms)
[task 2021-07-20T17:30:45.860Z] 17:30:45     INFO -  TEST-START | VideoFrameConverterTest.BasicConversion
[task 2021-07-20T17:30:45.861Z] 17:30:45     INFO -  mozilla::detail::MutexImpl::~MutexImpl: pthread_mutex_destroy failed: Device or resource busy
[task 2021-07-20T17:30:45.863Z] 17:30:45     INFO -  ExceptionHandler::GenerateDump cloned child ExceptionHandler::WaitForContinueSignal waiting for continue signal...
[task 2021-07-20T17:30:45.863Z] 17:30:45     INFO -  18071
[task 2021-07-20T17:30:45.864Z] 17:30:45     INFO -  ExceptionHandler::SendContinueSignalToChild sent continue signal to child
[task 2021-07-20T17:30:45.984Z] 17:30:45     INFO -  gtest INFO | gtest | process wait complete, returncode=-11
[task 2021-07-20T17:30:45.984Z] 17:30:45     INFO -  mozcrash checking /builds/worker/workspace/build/tests/gtest for minidumps...
[task 2021-07-20T17:30:45.985Z] 17:30:45     INFO -  mozcrash INFO | Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/M4TxeE4qSrGA4chclPSEcA/artifacts/public/build/target.crashreporter-symbols.zip
[task 2021-07-20T17:30:51.514Z] 17:30:51     INFO -  mozcrash INFO | Copy/paste: /builds/worker/fetches/minidump_stackwalk/minidump_stackwalk /builds/worker/workspace/build/tests/gtest/47104658-7a67-3c86-1a9e-9d75707f5920.dmp /tmp/tmp9s9sbg6f
[task 2021-07-20T17:30:56.940Z] 17:30:56     INFO -  mozcrash INFO | Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/47104658-7a67-3c86-1a9e-9d75707f5920.dmp
[task 2021-07-20T17:30:56.941Z] 17:30:56     INFO -  mozcrash INFO | Saved app info as /builds/worker/workspace/build/blobber_upload_dir/47104658-7a67-3c86-1a9e-9d75707f5920.extra
[task 2021-07-20T17:30:56.946Z] 17:30:56  WARNING -  PROCESS-CRASH | gtest | application crashed [@ FrameListener::OnVideoFrameConverted(webrtc::VideoFrame const&)]
[task 2021-07-20T17:30:56.946Z] 17:30:56     INFO -  Crash dump filename: /builds/worker/workspace/build/tests/gtest/47104658-7a67-3c86-1a9e-9d75707f5920.dmp
[task 2021-07-20T17:30:56.946Z] 17:30:56     INFO -  Mozilla crash reason: MOZ_CRASH(mozilla::detail::MutexImpl::~MutexImpl: pthread_mutex_destroy failed)
[task 2021-07-20T17:30:56.946Z] 17:30:56     INFO -  Operating system: Linux
[task 2021-07-20T17:30:56.947Z] 17:30:56     INFO -                    0.0.0 Linux 4.4.0-1014-aws #14taskcluster1-Ubuntu SMP Tue Apr 3 10:27:00 UTC 2018 x86_64
[task 2021-07-20T17:30:56.947Z] 17:30:56     INFO -  CPU: x86
[task 2021-07-20T17:30:56.947Z] 17:30:56     INFO -       AuthenticAMD family 23 model 1 stepping 2
[task 2021-07-20T17:30:56.947Z] 17:30:56     INFO -       4 CPUs
[task 2021-07-20T17:30:56.947Z] 17:30:56     INFO -  GPU: UNKNOWN
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -  Crash reason:  SIGSEGV /SEGV_MAPERR
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -  Crash address: 0x0
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -  Process uptime: not available
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -  Thread 63 (crashed) 0  libxul.so!webrtc::VideoFrame::VideoFrame(webrtc::VideoFrame const&) [video_frame.cc:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 41 + 0x28]
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -      eip = 0xe9a5d2d8   esp = 0xe0a6cb70   ebp = 0xe0a6cb88   ebx = 0xf0d01000
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -      esi = 0xf86c15e0   edi = 0xe684f560   eax = 0xe5e5e5e5   ecx = 0x00000000
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -      edx = 0xe68e3040   efl = 0x00210282
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -      Found by: given as instruction pointer in context
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -   1  libxul.so!FrameListener::OnVideoFrameConverted(webrtc::VideoFrame const&) [TestVideoFrameConverter.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 67 + 0x488]
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -      eip = 0xec6a2420   esp = 0xe0a6cb90   ebp = 0xe0a6cc28   ebx = 0xf0d01000
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -      esi = 0xf86c15e0   edi = 0xe684f560
[task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -      Found by: call frame info
task 2021-07-20T17:30:56.948Z] 17:30:56     INFO -   2  libxul.so!mozilla::VideoFrameConverter::SameFrameTick(nsITimer*, void*) [VideoFrameConverter.h:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 236 + 0x12]
[task 2021-07-20T17:30:56.949Z] 17:30:56     INFO -      eip = 0xe73c74d3   esp = 0xe0a6cc30   ebp = 0xe0a6cc48   ebx = 0xf0d01000
[task 2021-07-20T17:30:56.949Z] 17:30:56     INFO -      esi = 0x00000000   edi = 0x00000000
[task 2021-07-20T17:30:56.949Z] 17:30:56     INFO -      Found by: call frame info
[task 2021-07-20T17:30:56.949Z] 17:30:56     INFO -   3  libxul.so!nsTimerImpl::Fire(int) [nsTimerImpl.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 618 + 0x17]
[task 2021-07-20T17:30:56.949Z] 17:30:56     INFO -      eip = 0xecb8aa01   esp = 0xe0a6cc50   ebp = 0xe0a6ccf8   ebx = 0xf0d01000
[task 2021-07-20T17:30:56.949Z] 17:30:56     INFO -      esi = 0x00000000   edi = 0xf987f100
[task 2021-07-20T17:30:56.949Z] 17:30:56     INFO -      Found by: call frame info
[task 2021-07-20T17:30:56.949Z] 17:30:56     INFO -   4  libxul.so!nsTimerEvent::Run() [TimerThread.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 248 + 0x12]
[task 2021-07-20T17:30:56.949Z] 17:30:56     INFO -      eip = 0xecb796b4   esp = 0xe0a6cd00   ebp = 0xe0a6cd68   ebx = 0xf0d01000
[task 2021-07-20T17:30:56.949Z] 17:30:56     INFO -      esi = 0xf1060678   edi = 0xf1060678
[task 2021-07-20T17:30:56.950Z] 17:30:56     INFO -      Found by: call frame info
[task 2021-07-20T17:30:56.950Z] 17:30:56     INFO -   5  libxul.so!mozilla::TaskQueue::Runner::Run() [TaskQueue.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 208 + 0x8]
[task 2021-07-20T17:30:56.950Z] 17:30:56     INFO -      eip = 0xecb76f1c   esp = 0xe0a6cd70   ebp = 0xe0a6cdc8   ebx = 0xf0d01000
[task 2021-07-20T17:30:56.950Z] 17:30:56     INFO -      esi = 0xf1060678   edi = 0x00000000
[task 2021-07-20T17:30:56.950Z] 17:30:56     INFO -      Found by: call frame info
[task 2021-07-20T17:30:56.950Z] 17:30:56     INFO -   6  libxul.so!nsThreadPool::Run() [nsThreadPool.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 303 + 0x8]
[task 2021-07-20T17:30:56.950Z] 17:30:56     INFO -      eip = 0xecb81394   esp = 0xe0a6cdd0   ebp = 0xe0a6cf58   ebx = 0xf0d01000
[task 2021-07-20T17:30:56.950Z] 17:30:56     INFO -      esi = 0xe6971380   edi = 0x00000000
[task 2021-07-20T17:30:56.950Z] 17:30:56     INFO -      Found by: call frame info
[task 2021-07-20T17:30:56.951Z] 17:30:56     INFO -   7  libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 1142 + 0x18]
[task 2021-07-20T17:30:56.951Z] 17:30:56     INFO -      eip = 0xeb2cfdc4   esp = 0xe0a6cf60   ebp = 0xe0a6d158   ebx = 0xf0d01000
[task 2021-07-20T17:30:56.951Z] 17:30:56     INFO -      esi = 0xe0a6d0d0   edi = 0xf9866040
[task 2021-07-20T17:30:56.951Z] 17:30:56     INFO -      Found by: call frame info
[task 2021-07-20T17:30:56.951Z] 17:30:56     INFO -   8  libxul.so!mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [MessagePump.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 300 + 0x30]
[task 2021-07-20T17:30:56.951Z] 17:30:56     INFO -      eip = 0xecd4fc46   esp = 0xe0a6d160   ebp = 0xe0a6d1b8   ebx = 0xf0d01000
[task 2021-07-20T17:30:56.951Z] 17:30:56     INFO -      esi = 0xe68e35b0   edi = 0xe0a6d220
[task 2021-07-20T17:30:56.951Z] 17:30:56     INFO -      Found by: call frame info
[task 2021-07-20T17:30:56.951Z] 17:30:56     INFO -   9  libxul.so!MessageLoop::Run() [message_loop.cc:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 306 + 0xc]
[task 2021-07-20T17:30:56.951Z] 17:30:56     INFO -      eip = 0xecd2880f   esp = 0xe0a6d1c0   ebp = 0xe0a6d1f8   ebx = 0xf0d01000
[task 2021-07-20T17:30:56.952Z] 17:30:56     INFO -      esi = 0xe0a6d1d8   edi = 0xe0a6d220
[task 2021-07-20T17:30:56.952Z] 17:30:56     INFO -      Found by: call frame info
[task 2021-07-20T17:30:56.952Z] 17:30:56     INFO -  10  libxul.so!nsThread::ThreadFunc(void*) [nsThread.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 390 + 0x8]
[task 2021-07-20T17:30:56.952Z] 17:30:56     INFO -      eip = 0xecb7ad8e   esp = 0xe0a6d200   ebp = 0xe0a6d318   ebx = 0xf0d01000
[task 2021-07-20T17:30:56.952Z] 17:30:56     INFO -      esi = 0xf9866040   edi = 0xe0a6d220
[task 2021-07-20T17:30:56.952Z] 17:30:56     INFO -      Found by: call frame info
[task 2021-07-20T17:30:56.952Z] 17:30:56     INFO -  11  libnspr4.so!_pt_root [ptthread.c:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 201 + 0x9]
[task 2021-07-20T17:30:56.952Z] 17:30:56     INFO -      eip = 0xf67e4892   esp = 0xe0a6d320   ebp = 0xe0a6d358   ebx = 0xf67fc000
[task 2021-07-20T17:30:56.952Z] 17:30:56     INFO -      esi = 0xf49c5958   edi = 0xf985c340
[task 2021-07-20T17:30:56.952Z] 17:30:56     INFO -      Found by: call frame info
[task 2021-07-20T17:30:56.953Z] 17:30:56     INFO -  12  libpthread.so.0 + 0x63bd
[task 2021-07-20T17:30:56.953Z] 17:30:56     INFO -      eip = 0xf76a53bd   esp = 0xe0a6d360   ebp = 0xe0a6d428   ebx = 0x00000000
[task 2021-07-20T17:30:56.953Z] 17:30:56     INFO -      esi = 0xe0a6db40   edi = 0xf76bb000
[task 2021-07-20T17:30:56.953Z] 17:30:56     INFO -      Found by: call frame info
[task 2021-07-20T17:30:56.953Z] 17:30:56     INFO -  13  libc.so.6 + 0xf8fe6
[task 2021-07-20T17:30:56.954Z] 17:30:56     INFO -      eip = 0xf7307fe6   esp = 0xe0a6d430   ebp = 0x00000000
[task 2021-07-20T17:30:56.954Z] 17:30:56     INFO -      Found by: previous frame's frame pointer
Group: core-security → media-core-security
Component: Audio/Video → WebRTC: Audio/Video
Keywords: csectype-uaf, sec-high

This is a gtest, so maybe sec-high is not accurate. The top frames of the stack don't look unlike what I'd expect from a browser stack, FWIW.

This looks to me like a UAF in VideoFrameConverterTest showing in the mMonitor member, i.e., test-only.

What I don't understand is how the crashing stack leads to ~MutexImpl(). Assuming the nature of a UAF leads us there.

What I see as a plausible order of events is:

  • The gtest runs to completion on main thread.
  • The test fixture TearDown method runs on main thread, VideoFrameConverter::Shutdown dispatches a clean up task to its worker thread, holding a strong ref to the converter itself.
  • The test fixture is destroyed on main thread.
  • On the worker thread, before the clean up task has run, the same frame timer ticks and runs its handler, calling into the test fixture's FrameListener.
  • mTest is the destroyed test fixture. UAF.

The non-test equivalent to the raw ptr causing problems above is not as naive.

Assignee: nobody → apehrson
Priority: -- → P2
Status: NEW → ASSIGNED

Comment on attachment 9247963 [details]
Bug 1721443 - Use higher order functions for getting frames in TestVideoFrameConverter. r?bwc

Security Approval Request

  • How easily could an exploit be constructed based on the patch?: So this is a gtest-only bug. How does that affect the rating? Docs were not explicit on this.
  • Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: No
  • Which older supported branches are affected by this flaw?: all
  • If not all supported branches, which bug introduced the flaw?: None
  • Do you have backports for the affected branches?: No
  • If not, how different, hard to create, and risky will they be?: Older branches than 85 will require more backports. Easiest would be to skip esr78.
  • How likely is this patch to cause regressions; how much testing does it need?: not likely
Attachment #9247963 - Flags: sec-approval?
Attachment #9247961 - Flags: sec-approval?
Blocks: 1529581

If this is test only, then it doesn't need to be a sec bug, and doesn't need sec-approval. Thanks for your caution, though.

Group: media-core-security
Keywords: sec-high
Whiteboard: [test only]
Attachment #9247961 - Flags: sec-approval?
Attachment #9247963 - Flags: sec-approval?
Attachment #9248196 - Attachment is obsolete: true
Attachment #9248197 - Attachment is obsolete: true
Attachment #9248198 - Attachment is obsolete: true
Pushed by pehrsons@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/df2612b2d343
Implement gtest helper TakeN that return a MozPromise wrapping the first N events from a MediaEventSource. r=bwc
https://hg.mozilla.org/integration/autoland/rev/0fec1fce1701
Make WaitUntil handle multi-param MediaEvents. r=bwc
https://hg.mozilla.org/integration/autoland/rev/8c52f162022b
Use higher order functions for getting frames in TestVideoFrameConverter. r=bwc
Flags: needinfo?(apehrson)
Pushed by pehrsons@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/d70290bc3fb5
Implement gtest helper TakeN that return a MozPromise wrapping the first N events from a MediaEventSource. r=bwc
https://hg.mozilla.org/integration/autoland/rev/a3eea7436bc2
Make WaitUntil handle multi-param MediaEvents. r=bwc
https://hg.mozilla.org/integration/autoland/rev/c38ef0933c9c
Use higher order functions for getting frames in TestVideoFrameConverter. r=bwc

https://hg.mozilla.org/mozilla-central/rev/d70290bc3fb5
https://hg.mozilla.org/mozilla-central/rev/a3eea7436bc2
https://hg.mozilla.org/mozilla-central/rev/c38ef0933c9c

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
status-firefox96: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 96 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: