Implement video output for media engine playback
Categories
(Core :: Audio/Video: Playback, task, P2)
Tracking
()
Tracking | Status | |
---|---|---|
firefox105 | --- | fixed |
People
(Reporter: alwu, Assigned: alwu)
References
(Blocks 1 open bug)
Details
Attachments
(10 files, 1 obsolete file)
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review | |
48 bytes,
text/x-phabricator-request
|
Details | Review |
In this bug, we will address how to display the video output via the media engine API.
See here for more details.
Assignee | ||
Comment 1•2 years ago
|
||
Assignee | ||
Comment 2•2 years ago
|
||
Depends on D149941
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Assignee | ||
Comment 3•2 years ago
|
||
Depends on D149941
Updated•2 years ago
|
Assignee | ||
Comment 4•2 years ago
•
|
||
Now the major problem I encounter is that sometime IDCompositionDevice::CreateSurfaceFromHandle()
would fail and returns E_HANDLE
error, which causes nothing showing on the screen.
This situation is intermittent, and have two situations. (1) can't create a surface in the beginning, and then the surface can be created successfully by using the same handle after a while (2) no matter how long we wait, it always fail to create the surface by using the same handle.
The implementation details can be viewed here. The brief introduction is that, (1) we duplicate it via IPC mechanism and sent it to the GPU process (2) call IDCompositionDevice::CreateSurfaceFromHandle()
(in RenderDcompSurfaceTextureHost::CreateSurfaceFromDevice()
) to construct the surface an use it in the DC layer tree.
I've tried following ways
(1) delay the timing of getting handle from the media engine
(2) duplicate the handle by using DUPLICATE_SAME_ACCESS
before sending the handle across IPC to ensure the handle alive
(3) ask for a new handle whenever the media time changes, and send the new handle to GPU process
I also tried to
(1) step in IDCompositionDevice::CreateSurfaceFromHandle()
in disassembly mode via Visual Studio
(2) use DebugView and Mftrace
But none of them works or gave me useful information.
Welcome for any help or suggestion because now I have no idea how to solve this problem :(
Updated•2 years ago
|
Assignee | ||
Comment 5•2 years ago
|
||
Depends on D151019
Assignee | ||
Comment 6•2 years ago
|
||
Depends on D151585
Assignee | ||
Comment 7•2 years ago
|
||
Depends on D151707
Updated•2 years ago
|
Assignee | ||
Comment 8•2 years ago
•
|
||
If anyone wants to try or debug the issue, currently the way to use media engine to do the video playback is
(1) apply above patches stack (bug 1778038 + bug 1771011)
(2) set the pref
media.utility-process.enabled=false
(there is a bug needed to be fixed)
media.wmf.media-engine.enabled=true
media.wmf.media-engine.video-output.enabled=true
(3) restart browser
(4) run firefox with MOZ_DISABLE_RDD_SANDBOX=1
(5) go to Youtube (or any sites using MSE), then the playback would be done through the media engine.
Assignee | ||
Comment 9•2 years ago
|
||
Depends on D151803
Assignee | ||
Comment 10•2 years ago
|
||
Depends on D151905
Comment 11•2 years ago
|
||
I think we're not transferring the handle properly between processes.
Comment 12•2 years ago
|
||
The theory I had about this is probably not true. It looks like the message is supposed to be relayed through a privileged broker process, I haven't looked at the details of how this would be happening. It's still possible something is broken there.
Comment 13•2 years ago
|
||
The problem is that HANDLE gets closed in DCompSurfaceHandleHost:
mHandle = const_cast<ipc::FileDescriptor&>(aDescriptor.handle()).TakePlatformHandle().get();
TakePlatformHandle() returns a
UniqueFileHandlewhich gets dropped after
.get()` is called.
I suspect the reason that it sometimes works is that we were continually passing the handle over to the GPU process and there was a small window between it being sent and being closed. If we called CreateSurfaceFromHandle()
during this window it would succeed.
Changing DCompSurfaceHandleHost
to have mozilla::UniqueFileHandle mHandle
instead of HANDLE mHandle
seems to fix the problem.
Assignee | ||
Comment 14•2 years ago
|
||
When the media config change monitor detects inband format change, it
would recreate the video decoder along with passing a new config.
This results in a new call of CreateVideoDecoder()
in which we will
wrap existing video stream in a new wrapper in MFMediaEngineParent::GetMediaEngineStream()
.
Therefore, if this path get called again, that implicitly means the
inband config change happens and we should dispatch a format change
event and regenerate the stream descriptor.
H264 is an exception, we follow Chromimum's implementation to disable
the format change event for that and explained the reason in the code.
Depends on D151906
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Assignee | ||
Comment 15•2 years ago
|
||
When we use the media engine, we have to use the media engine decoder
for both audio and video in order to let media engine manage a/v sync.
We assign the wrong location for RemoteAudioChild which causes using
audio decoder in the utility process.
Depends on D152251
Assignee | ||
Updated•2 years ago
|
Assignee | ||
Comment 16•2 years ago
|
||
Assignee | ||
Comment 17•2 years ago
|
||
FYI, updated the diagram for the current architecture.
Comment 18•2 years ago
|
||
Pushed by alwu@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/c3f7229289f6 part1 : setup some video playback related attributes on the media engine.r=jolin https://hg.mozilla.org/integration/autoland/rev/39cbad610c66 part2 : wrap media engine's dcomp handle, and use it on our gfx pipeline in the GPU process.r=jolin,sotaro https://hg.mozilla.org/integration/autoland/rev/3ff1b211a793 part3 : support changing playback rate on the media engine playback.r=jolin https://hg.mozilla.org/integration/autoland/rev/4a9d4b84bba8 part4 : process any pending sample request when a stream starts.r=jolin https://hg.mozilla.org/integration/autoland/rev/c431eda74191 part5 : report playback statistic data.r=jolin https://hg.mozilla.org/integration/autoland/rev/e85111774e42 part6 : convert h264 samples to annexB before sending them to the media engine.r=jolin https://hg.mozilla.org/integration/autoland/rev/d4a6f861a2ce part7 : implement MFT check for video codecs and cache all results for future usage.r=jolin https://hg.mozilla.org/integration/autoland/rev/cfe2f5232167 part8 : dispatch format change event for video when detecting inband config change.r=jolin https://hg.mozilla.org/integration/autoland/rev/58a60aa8d5b8 part9 : fix the issue of not being able to create media engine audio decoder correctly when enabling the utility audio decoder.r=jolin https://hg.mozilla.org/integration/autoland/rev/1ff210178343 part10 : prevent adding a redundant refcount which causes leaking streams. r=jolin
Comment 19•2 years ago
|
||
Backed out 10 changesets (Bug 1771011) for causing bustages on DcompSurfaceImage.obj.
Backout link
Push with failures <--> Bp-hybrid
Failure Log
Also WMC64 Bd Failure Log
Also WMC32 Bd Failure Log
Comment 20•2 years ago
|
||
Pushed by alwu@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/d461236d84b8 part1 : setup some video playback related attributes on the media engine.r=jolin https://hg.mozilla.org/integration/autoland/rev/0b78c56a75b6 part2 : wrap media engine's dcomp handle, and use it on our gfx pipeline in the GPU process.r=jolin,sotaro https://hg.mozilla.org/integration/autoland/rev/e2eaae053320 part3 : support changing playback rate on the media engine playback.r=jolin https://hg.mozilla.org/integration/autoland/rev/9800e1146c4f part4 : process any pending sample request when a stream starts.r=jolin https://hg.mozilla.org/integration/autoland/rev/e09f282a1759 part5 : report playback statistic data.r=jolin https://hg.mozilla.org/integration/autoland/rev/11fc59bc723c part6 : convert h264 samples to annexB before sending them to the media engine.r=jolin https://hg.mozilla.org/integration/autoland/rev/92b3e8bbbd93 part7 : implement MFT check for video codecs and cache all results for future usage.r=jolin https://hg.mozilla.org/integration/autoland/rev/64f52a6de5f3 part8 : dispatch format change event for video when detecting inband config change.r=jolin https://hg.mozilla.org/integration/autoland/rev/50407de8f01f part9 : fix the issue of not being able to create media engine audio decoder correctly when enabling the utility audio decoder.r=jolin https://hg.mozilla.org/integration/autoland/rev/2305d2cb269b part10 : prevent adding a redundant refcount which causes leaking streams. r=jolin
Comment 21•2 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/d461236d84b8
https://hg.mozilla.org/mozilla-central/rev/0b78c56a75b6
https://hg.mozilla.org/mozilla-central/rev/e2eaae053320
https://hg.mozilla.org/mozilla-central/rev/9800e1146c4f
https://hg.mozilla.org/mozilla-central/rev/e09f282a1759
https://hg.mozilla.org/mozilla-central/rev/11fc59bc723c
https://hg.mozilla.org/mozilla-central/rev/92b3e8bbbd93
https://hg.mozilla.org/mozilla-central/rev/64f52a6de5f3
https://hg.mozilla.org/mozilla-central/rev/50407de8f01f
https://hg.mozilla.org/mozilla-central/rev/2305d2cb269b
Assignee | ||
Updated•2 years ago
|
Description
•