1804134 - [wmfme] Only display black frames when watching Youtube short/Youtube preview (on homepage)

Status: NEW

(Core :: Audio/Video: Playback, defect, P3)

Description

[Alastor Wu [:alwu]]

STR.

1. enable the pref media.wmf.media-engine.enabled
2. go to https://www.youtube.com/shorts/ontm1vqONVY

Expected.
3. video frames get displayed correctly

Actual.
4. Only see black frames

Note, if you watch video via Pip window, then the video can be displayed correctly (?!)

Comment 1

[Alastor Wu [:alwu]]

When video is displaying black frames, I can see there are some web render errors "Invalid ext-image". But after you switch video into Pip window, after a while, when video starts displaying correctly, that error would dismiss.

By checking the code, the web render code calls lock to get an external image, which seems related with this default function. As RenderDcompSurfaceTextureHost didn't implement that function, calling that would result in an invalid type which causes showing that error.

As playing the video in the PiP window won't trigger that error, my question is whether we have different code paths for displaying video in the element and displaying video in the PiP window? Should I implement lock() for RenderDcompSurfaceTextureHost as well?

---

Satoro, I wonder if you know the answer of my questions, do you have any thought about this issue? Thank you so much.

Comment 2

[Alastor Wu [:alwu]]

There are also some logs I captured

• When the external image got created

2022-12-12 23:54:43.374000 UTC - [GPU 33072: Compositor]: D/DcompSurface DcompSurfaceHandleHost 20310113570 CreateRenderTexture, ext-id=1
2022-12-12 23:54:43.374000 UTC - [GPU 33072: Compositor]: D/DcompSurface DcompSurfaceHandleHost 20310113570 PushResourceUpdate, exi-id=64424509442, type=ext
2022-12-12 23:54:43.374000 UTC - [GPU 33072: Compositor]: D/DcompSurface DcompSurfaceHandleHost 20310113570 PushDisplayItems

• When video frames couldn't be displayed (black frames)

2022-12-12 23:54:52.705000 UTC - [GPU 33072: Renderer]: D/RenderThread RenderThread::UpdateAndRender() aWindowId 1 aRender 1
[2022-12-12T23:54:52Z WARN  webrender::renderer] Invalid ext-image
[2022-12-12T23:54:52Z WARN  webrender::renderer] For ext_id:ExternalImageId(64424509442), channel:0.
2022-12-12 23:54:52.709000 UTC - [GPU 33072: Renderer]: D/RenderThread ---- done RenderThread::UpdateAndRender() aWindowId 1 aRender 1

• When video frames are displayed correctly in the Pip Window

2022-12-12 23:54:52.721000 UTC - [GPU 33072: Renderer]: D/RenderThread RenderThread::UpdateAndRender() aWindowId 3 aRender 1
2022-12-12 23:54:52.721000 UTC - [GPU 33072: Renderer]: D/DcompSurface DCSurfaceHandle=2031fe32480, PresentSurfaceHandle
2022-12-12 23:54:52.721000 UTC - [GPU 33072: Renderer]: D/DcompSurface DCSurfaceHandle=2031fe32480, Set surface 20311c80290 to visual
2022-12-12 23:54:52.721000 UTC - [GPU 33072: Renderer]: D/RenderThread ---- done RenderThread::UpdateAndRender() aWindowId 3 aRender 1

Comment 3

[Sotaro Ikeda [:sotaro]]

Satoro, I wonder if you know the answer of my questions, do you have any thought about this issue? Thank you so much.

This seemed to happen when external image was used for rendering without using external compositing by WebRender. And it seemed to fallback to wr::ExternalImageType::TextureHandle(wr::ImageBufferKind::TextureExternal).

I am not sure if we could force external compositing to external image.

• https://searchfox.org/mozilla-central/rev/a0d4f8f112c5c792ae272bf6ce50763ddd23ffa2/gfx/layers/DcompSurfaceImage.cpp#99
• https://searchfox.org/mozilla-central/rev/a0d4f8f112c5c792ae272bf6ce50763ddd23ffa2/gfx/layers/DcompSurfaceImage.h#116

Comment 4

[Sotaro Ikeda [:sotaro]]

:gw, can we force force external compositing to external image? DcompSurfaceHandleHost/RenderDcompSurfaceTextureHost always need to use external compositing.

Comment 5

[Glenn Watson [:gw]]

We can't force external compositing, it's only a hint for WR to use if possible. It will disable it if there is a feature required not supported by external compositing (e.g. border-radius, non-axis-aligned transform, too many compositor surfaces etc). It may be worth looking in to why it's not using external compositing in this case, perhaps it's something simple we could add support for, but we can't guarantee it in all cases.

Comment 6

[Alastor Wu [:alwu]]

After doing more debugging and discussion with gw, this is the place preventing us from using external compositing because we need to apply mask. Youtube applies rounding on those shorts video, same as video preview when you're in their homepage.

Gw told me that "drawing this video requires a clip mask (e.g. for a rounded corner mask), which native compositors can't handle - so we have to composite it ourselves within WR", which means we MUST know pixels for these kinds of video. So if we want to solve this issue, we have to support frame server mode in the media engine, which allows us to access the decoded video frames.

I am not sure if this is something we want to support, because our goal is only using this for encrypted playback, where no one (except OS) should know the content of decoded video. I will put this issue on hold for now.

Comment 7

[Alastor Wu [:alwu]]

There are some references for the implementation of the frame server mode in the chromium. The implementation seems more complicated as I originally thought.

Bug : https://bugs.chromium.org/p/chromium/issues/detail?id=1258887
Design doc 1 : https://docs.google.com/document/d/1GnE-9WWVJdqHEDq3gX4eRCbH-gjIj40oICEFVZg3ywQ/edit#heading=h.4z9asym9m3g0
Design doc 2 : https://docs.google.com/document/d/1GmxT0XlBQH24vQIQWudHqcJ9guLmMniF0tNKEiNt2RM/edit#heading=h.tu1x1w8tswqq

Comment 8

[Jim Mathies [:jimm]]

Note this is related to general playback of non-encrypted video using the new pipeline, which we do not have a shipping roadmap for yet.