Open Bug 1594141 Opened 5 months ago Updated 2 months ago

Assertion failure: mVideoEncoder->IsInitialized(), at /builds/worker/workspace/build/src/dom/media/encoder/MediaEncoder.cpp:682

Categories

(Core :: Audio/Video: Recording, defect, P2)

defect

Tracking

()

Tracking Status
firefox72 --- affected

People

(Reporter: jkratzer, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: assertion, testcase)

Attachments

(3 files)

Attached file testcase.html

Testcase found while fuzzing mozilla-central rev 4d585c7edc76.

Assertion failure: mVideoEncoder->IsInitialized(), at /builds/worker/workspace/build/src/dom/media/encoder/MediaEncoder.cpp:682

rax = 0x000055df47163340   rdx = 0x0000000000000000
rcx = 0x00007ff50af4f63a   rbx = 0x00007ff4fd2ab660
rsi = 0x00007ff5169a08b0   rdi = 0x00007ff51699f680
rbp = 0x00007ff4fb9a1410   rsp = 0x00007ff4fb9a1390
r8 = 0x00007ff5169a08b0    r9 = 0x00007ff4fb9a2700
r10 = 0x0000000000000002   r11 = 0x0000000000000000
r12 = 0x00007ff4fb9a13c0   r13 = 0x0000000000000000
r14 = 0x0000000000000000   r15 = 0x00007ff4fb9a1470
rip = 0x00007ff506ccd761
OS|Linux|0.0.0 Linux 5.0.0-31-generic #33~18.04.1-Ubuntu SMP Tue Oct 1 10:20:39 UTC 2019 x86_64
CPU|amd64|family 6 model 94 stepping 3|1
GPU|||
Crash|SIGSEGV|0x0|35
35|0|libxul.so|mozilla::MediaEncoder::GetEncodedData(nsTArray<nsTArray<unsigned char> >*)|hg:hg.mozilla.org/mozilla-central:dom/media/encoder/MediaEncoder.cpp:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|680|0x32
35|1|libxul.so|mozilla::dom::MediaRecorder::Session::Extract(mozilla::TimeStamp, bool)|hg:hg.mozilla.org/mozilla-central:dom/media/MediaRecorder.cpp:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|946|0x18
35|2|libxul.so|mozilla::dom::MediaRecorder::Session::MediaEncoderDataAvailable()|hg:hg.mozilla.org/mozilla-central:dom/media/MediaRecorder.cpp:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|1196|0x17
35|3|libxul.so|mozilla::MediaEncoder::NotifyDataAvailable()|hg:hg.mozilla.org/mozilla-central:dom/media/encoder/MediaEncoder.cpp:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|915|0x11
35|4|libxul.so|mozilla::MediaEncoder::EncoderListener::DataAvailableImpl()|hg:hg.mozilla.org/mozilla-central:dom/media/encoder/MediaEncoder.cpp:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|349|0x5
35|5|libxul.so|mozilla::detail::RunnableMethodImpl<mozilla::MediaEncoder::EncoderListener*, void (mozilla::MediaEncoder::EncoderListener::*)(), true, (mozilla::RunnableKind)0>::Run()|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThreadUtils.h:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|1176|0x13
35|6|libxul.so|mozilla::TaskQueue::Runner::Run()|hg:hg.mozilla.org/mozilla-central:xpcom/threads/TaskQueue.cpp:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|199|0x7
35|7|libxul.so|nsThreadPool::Run()|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThreadPool.cpp:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|256|0x15
35|8|libxul.so|nsThread::ProcessNextEvent(bool, bool*)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThread.cpp:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|1225|0x15
35|9|libxul.so|NS_ProcessNextEvent(nsIThread*, bool)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThreadUtils.cpp:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|486|0x11
35|10|libxul.so|mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessagePump.cpp:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|333|0xd
35|11|libxul.so|MessageLoop::RunInternal()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|315|0x17
35|12|libxul.so|MessageLoop::Run()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|290|0x8
35|13|libxul.so|nsThread::ThreadFunc(void*)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThread.cpp:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|458|0x38
35|14|libnspr4.so|_pt_root|hg:hg.mozilla.org/mozilla-central:nsprpub/pr/src/pthreads/ptthread.c:4d585c7edc7683e4b35eca6b18c9a646a1b8a78d|201|0x7
35|15|libpthread-2.27.so||||0x76db
35|16|libc-2.27.so||||0x12188f
Flags: in-testsuite?

:bryce, it looks like you were last in this code. Would you care to take a look?

Flags: needinfo?(bvandyk)
Priority: -- → P2
Assignee: nobody → bvandyk

Think I've found the source of the issue.

MediaEncoder expects the track encoders to be initialized by the time we're getting encoded data. This would be a safe assumption if an initialized encoder could never transition to becoming uninitialized, but for the vp8 encoder that is not the case.

There is an edge case that we're not handling:

The above results in us transitioning from an initialized encoder to an uninitialized one and hitting this assert as we get data.

Patch incoming.

Flags: needinfo?(bvandyk)

This is adapted from the original test on the bug. The only changes are:

  • Adding reftest-wait so the test plays nice with our harness.
  • Removing code that doesn't appear required to hit the assert.
    • There are a few lines that don't appear needed.
    • Simplified some arguments to make the test more human readable.

If the VP8 encoder fails to reconfigure in the middle of a stream it can
transition from a state where it was returning true for IsInitialized to
then returning false. This would trigger an assert in GetEncodedData in
MediaEncoder. This patch changes to instead handle this case by logging an
error and moving the encoder into an error state.

Depends on D51955

Try is currently closed. Getting patches up for review, and will push tomorrow/when the tree is open.

Pushed by bvandyk@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0516cc76b5b2
Add crashtest for if a Vp8 track encoder becomes uninitialized mid stream. r=pehrsons
https://hg.mozilla.org/integration/autoland/rev/6d4a958ff155
Handle failure to reconfigure video encoder in MediaEncoder. r=pehrsons

Backed out for media failures on test_mediarecorder_principals.html

backout: https://hg.mozilla.org/integration/autoland/rev/f9f1ad5a1dfa732bfbdc099db1cf5f7e1f931773

push: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedJob=275405673&searchStr=android%2C8.0%2Cpixel2%2Cdebug%2Cmochitests%2Ctest-android-hw-p2-8-0-arm7-api-16%2Fdebug-geckoview-mochitest-media-e10s-3%2Cm%28mda3%29&revision=6d4a958ff1554c7bc1258da680cb7c9327af529b

failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=275405673&repo=autoland&lineNumber=3126

[task 2019-11-08T20:54:27.958Z] 20:53:56 INFO - 286 INFO TEST-PASS | dom/media/test/test_mediarecorder_principals.html | mediaRecorder.start() must throw SecurityError
[task 2019-11-08T20:54:27.958Z] 20:53:56 INFO - 287 INFO TEST-PASS | dom/media/test/test_mediarecorder_principals.html | mediaRecorder.start() should not throw here, and didn't
[task 2019-11-08T20:54:27.958Z] 20:53:56 INFO - Buffered messages finished
[task 2019-11-08T20:54:27.958Z] 20:53:56 WARNING - 288 INFO TEST-UNEXPECTED-FAIL | dom/media/test/test_mediarecorder_principals.html | mediaRecorder.onerror must fire SecurityError - got "UnknownError", expected "SecurityError"
[task 2019-11-08T20:54:27.958Z] 20:53:56 INFO - SimpleTest.is@SimpleTest/SimpleTest.js:322:16
[task 2019-11-08T20:54:27.958Z] 20:53:56 INFO - testPrincipals2@dom/media/test/test_mediarecorder_principals.html:113:5
[task 2019-11-08T20:54:27.958Z] 20:53:56 INFO - 289 INFO TEST-PASS | dom/media/test/test_mediarecorder_principals.html | Events fired from onerror should include an error with a stack trace indicating an error in this test
[task 2019-11-08T20:54:27.958Z] 20:53:56 INFO - 290 INFO TEST-PASS | dom/media/test/test_mediarecorder_principals.html | Playback should not have reached end

Flags: needinfo?(bvandyk)

Acking. I don't appear able repro the failure on my local Android emulator, continuing to investigate.

My suspicion is that the failure is caused by the new calls to SetError in the patch are triggering a session end task due to the encode error before the security error the test expects. That indicates that the vp8 encoder is failing to encode data prior to our security error, which is likely a pre-existing issue being shown up by the patch.

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:bryce, could you have a look please?
For more information, please visit auto_nag documentation.

Flags: needinfo?(bvandyk)

Rework is pending me being able to debug this on Android, which is proving more fun than I expected. I'm currently soft blocked by bug 1598461.

Depends on: 1598461
Flags: needinfo?(bvandyk)

It seems fixed, maybe it's time to finish it?

Bryce can I help you to finish this one. FWIW, I have a local Android build and I can test stuff in my device.

(In reply to Alex Chronopoulos [:achronop] from comment #14)

Bryce can I help you to finish this one. FWIW, I have a local Android build and I can test stuff in my device.

That would be appreciated.

Flags: needinfo?(bvandyk)
Assignee: bvandyk → nobody
You need to log in before you can comment on or make changes to this bug.