Closed Bug 1771011 Opened 5 months ago Closed 2 months ago

Implement video output for media engine playback

Categories

(Core :: Audio/Video: Playback, task, P2)

Unspecified
Windows
task

Tracking

()

RESOLVED FIXED
105 Branch
Tracking Status
firefox105 --- fixed

People

(Reporter: alwu, Assigned: alwu)

References

(Blocks 2 open bugs)

Details

Attachments

(10 files, 1 obsolete file)

48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review

In this bug, we will address how to display the video output via the media engine API.

See here for more details.

Depends on D149941

Attachment #9282299 - Attachment description: WIP: Bug 1771011 - part1 : setup some video playback related attributes on the media engine. → Bug 1771011 - part1 : setup some video playback related attributes on the media engine.
Attachment #9282300 - Attachment description: WIP: Bug 1771011 - part2 : implement Dcomp Texture and image. → Bug 1771011 - part2 : implement Dcomp Texture and image.
Attachment #9282299 - Attachment description: Bug 1771011 - part1 : setup some video playback related attributes on the media engine. → WIP: Bug 1771011 - part1 : setup some video playback related attributes on the media engine.
Attachment #9282300 - Attachment is obsolete: true

Now the major problem I encounter is that sometime IDCompositionDevice::CreateSurfaceFromHandle() would fail and returns E_HANDLE error, which causes nothing showing on the screen.

This situation is intermittent, and have two situations. (1) can't create a surface in the beginning, and then the surface can be created successfully by using the same handle after a while (2) no matter how long we wait, it always fail to create the surface by using the same handle.

The implementation details can be viewed here. The brief introduction is that, (1) we duplicate it via IPC mechanism and sent it to the GPU process (2) call IDCompositionDevice::CreateSurfaceFromHandle() (in RenderDcompSurfaceTextureHost::CreateSurfaceFromDevice()) to construct the surface an use it in the DC layer tree.


I've tried following ways
(1) delay the timing of getting handle from the media engine
(2) duplicate the handle by using DUPLICATE_SAME_ACCESS before sending the handle across IPC to ensure the handle alive
(3) ask for a new handle whenever the media time changes, and send the new handle to GPU process

I also tried to
(1) step in IDCompositionDevice::CreateSurfaceFromHandle() in disassembly mode via Visual Studio
(2) use DebugView and Mftrace

But none of them works or gave me useful information.
Welcome for any help or suggestion because now I have no idea how to solve this problem :(

Flags: needinfo?(jmuizelaar)
Attachment #9285446 - Attachment description: WIP: Bug 1771011 - part5 : report statistic data. → WIP: Bug 1771011 - part5 : report playback statistic data.

If anyone wants to try or debug the issue, currently the way to use media engine to do the video playback is
(1) apply above patches stack (bug 1778038 + bug 1771011)
(2) set the pref
media.utility-process.enabled=false (there is a bug needed to be fixed)
media.wmf.media-engine.enabled=true
media.wmf.media-engine.video-output.enabled=true
(3) restart browser
(4) run firefox with MOZ_DISABLE_RDD_SANDBOX=1
(5) go to Youtube (or any sites using MSE), then the playback would be done through the media engine.

I think we're not transferring the handle properly between processes.

The theory I had about this is probably not true. It looks like the message is supposed to be relayed through a privileged broker process, I haven't looked at the details of how this would be happening. It's still possible something is broken there.

The problem is that HANDLE gets closed in DCompSurfaceHandleHost:

mHandle = const_cast<ipc::FileDescriptor&>(aDescriptor.handle()).TakePlatformHandle().get();

TakePlatformHandle() returns aUniqueFileHandlewhich gets dropped after.get()` is called.

I suspect the reason that it sometimes works is that we were continually passing the handle over to the GPU process and there was a small window between it being sent and being closed. If we called CreateSurfaceFromHandle() during this window it would succeed.

Changing DCompSurfaceHandleHost to have mozilla::UniqueFileHandle mHandle instead of HANDLE mHandle seems to fix the problem.

Flags: needinfo?(jmuizelaar)

When the media config change monitor detects inband format change, it
would recreate the video decoder along with passing a new config.

This results in a new call of CreateVideoDecoder() in which we will
wrap existing video stream in a new wrapper in MFMediaEngineParent::GetMediaEngineStream().

Therefore, if this path get called again, that implicitly means the
inband config change happens and we should dispatch a format change
event and regenerate the stream descriptor.

H264 is an exception, we follow Chromimum's implementation to disable
the format change event for that and explained the reason in the code.

Depends on D151906

Attachment #9282299 - Attachment description: WIP: Bug 1771011 - part1 : setup some video playback related attributes on the media engine. → Bug 1771011 - part1 : setup some video playback related attributes on the media engine.r=jolin!
Attachment #9284126 - Attachment description: WIP: Bug 1771011 - part2 : wrap media engine's dcomp handle, and use it on our gfx pipeline in the GPU process. → Bug 1771011 - part2 : wrap media engine's dcomp handle, and use it on our gfx pipeline in the GPU process.r=jolin!,sotaro!
Attachment #9285074 - Attachment description: WIP: Bug 1771011 - part3 : support changing playback rate on the media engine playback. → Bug 1771011 - part3 : support changing playback rate on the media engine playback.r=jolin!
Attachment #9285279 - Attachment description: WIP: Bug 1771011 - part4 : process any pending sample request when a stream starts. → Bug 1771011 - part4 : process any pending sample request when a stream starts.r=jolin!
Attachment #9285446 - Attachment description: WIP: Bug 1771011 - part5 : report playback statistic data. → Bug 1771011 - part5 : report playback statistic data.r=jolin!
Attachment #9285600 - Attachment description: WIP: Bug 1771011 - part6 : convert h264 samples to annexB before sending them to the media engine. → Bug 1771011 - part6 : convert h264 samples to annexB before sending them to the media engine.r=jolin!
Attachment #9285601 - Attachment description: WIP: Bug 1771011 - part7 : implement MFT check for video codecs and cache all results for future usage. → Bug 1771011 - part7 : implement MFT check for video codecs and cache all results for future usage.r=jolin!
Attachment #9286189 - Attachment description: WIP: Bug 1771011 - part8 : dispatch format change event for video when detecting inband config change. → Bug 1771011 - part8 : dispatch format change event for video when detecting inband config change.r=jolin!

When we use the media engine, we have to use the media engine decoder
for both audio and video in order to let media engine manage a/v sync.

We assign the wrong location for RemoteAudioChild which causes using
audio decoder in the utility process.

Depends on D152251

FYI, updated the diagram for the current architecture.

Pushed by alwu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c3f7229289f6
part1 : setup some video playback related attributes on the media engine.r=jolin
https://hg.mozilla.org/integration/autoland/rev/39cbad610c66
part2 : wrap media engine's dcomp handle, and use it on our gfx pipeline in the GPU process.r=jolin,sotaro
https://hg.mozilla.org/integration/autoland/rev/3ff1b211a793
part3 : support changing playback rate on the media engine playback.r=jolin
https://hg.mozilla.org/integration/autoland/rev/4a9d4b84bba8
part4 : process any pending sample request when a stream starts.r=jolin
https://hg.mozilla.org/integration/autoland/rev/c431eda74191
part5 : report playback statistic data.r=jolin
https://hg.mozilla.org/integration/autoland/rev/e85111774e42
part6 : convert h264 samples to annexB before sending them to the media engine.r=jolin
https://hg.mozilla.org/integration/autoland/rev/d4a6f861a2ce
part7 : implement MFT check for video codecs and cache all results for future usage.r=jolin
https://hg.mozilla.org/integration/autoland/rev/cfe2f5232167
part8 : dispatch format change event for video when detecting inband config change.r=jolin
https://hg.mozilla.org/integration/autoland/rev/58a60aa8d5b8
part9 : fix the issue of not being able to create media engine audio decoder correctly when enabling the utility audio decoder.r=jolin
https://hg.mozilla.org/integration/autoland/rev/1ff210178343
part10 : prevent adding a redundant refcount which causes leaking streams. r=jolin

Backed out 10 changesets (Bug 1771011) for causing bustages on DcompSurfaceImage.obj.
Backout link
Push with failures <--> Bp-hybrid
Failure Log
Also WMC64 Bd Failure Log
Also WMC32 Bd Failure Log

Flags: needinfo?(alwu)
Pushed by alwu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d461236d84b8
part1 : setup some video playback related attributes on the media engine.r=jolin
https://hg.mozilla.org/integration/autoland/rev/0b78c56a75b6
part2 : wrap media engine's dcomp handle, and use it on our gfx pipeline in the GPU process.r=jolin,sotaro
https://hg.mozilla.org/integration/autoland/rev/e2eaae053320
part3 : support changing playback rate on the media engine playback.r=jolin
https://hg.mozilla.org/integration/autoland/rev/9800e1146c4f
part4 : process any pending sample request when a stream starts.r=jolin
https://hg.mozilla.org/integration/autoland/rev/e09f282a1759
part5 : report playback statistic data.r=jolin
https://hg.mozilla.org/integration/autoland/rev/11fc59bc723c
part6 : convert h264 samples to annexB before sending them to the media engine.r=jolin
https://hg.mozilla.org/integration/autoland/rev/92b3e8bbbd93
part7 : implement MFT check for video codecs and cache all results for future usage.r=jolin
https://hg.mozilla.org/integration/autoland/rev/64f52a6de5f3
part8 : dispatch format change event for video when detecting inband config change.r=jolin
https://hg.mozilla.org/integration/autoland/rev/50407de8f01f
part9 : fix the issue of not being able to create media engine audio decoder correctly when enabling the utility audio decoder.r=jolin
https://hg.mozilla.org/integration/autoland/rev/2305d2cb269b
part10 : prevent adding a redundant refcount which causes leaking streams. r=jolin
Blocks: 1784909
Flags: needinfo?(alwu)
You need to log in before you can comment on or make changes to this bug.