1760369 - PIP Captions with Crunchyroll, other video players that use canvases

Status: NEW

(Toolkit :: Picture-in-Picture, enhancement, P3)

Description

[Callum Collins]

Steps to reproduce:

I tried creating a new video wrapper using the new PictureInPictureVideoWrapper feature for Crunchyroll as it's something I wanted to use PiP with for a while but needs captions to work to be useful to me. But found it uses a canvas and not a textual overlay for subtitles which the wrapper currently needs for caption support

Actual results:

I implemented a hacky work around but found the following issues along the way

• Creating a fixed size canvas in setupPlayer that scales to the pip window size ends up with a 150x300 canvas but very stretched, I work around this with a fixed size high res canvas that was just set to 100% height and width. I think this is because the video player starts that size and resizes after creation, I imagine making the canvas later or rebuilding it is probably a better solution that the fixed size work around I used
• The MutationObserver doesn't work for canvas's I think? So I used a setInterval instead to just repeatedly copy out the canvas content and pass it out of the wrapper as a work around, taking the whole canvas or the image data seems to work and it just needs scaling to the new canvas over the PIP video

This ended up working pretty well and the canvas rendered nicely and updated, resized well enough for my usage.

Expected results:

I think either having a more general way of overlaying content with different HTML elements, or if text and canvas elements are the only common cases being able to use either in the wrapper could be good.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Toolkit::Picture-in-Picture' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Molly Howell (she/her) (no longer active)]

This is extremely interesting, thank you for writing up this report, and also for working on a site wrapper!

I don't think we would want to support overlaying arbitrary HTML, or even just arbitrary canvases, over the PIP window, if we can avoid it; I'm afraid there would be security implications there that would become quite involved to work around.

I do think we want to support captions on Crunchyroll if at all possible though; certainly there's a relatively wider audience for the feature there. I'd much rather find a way to obtain the text itself and render that directly though, even if it means digging a bit uncomfortably deep into the workings of the app.

Comment 3

[Callum Collins]

Interesting I didn't think about looking deeper into the JS but they are using libass/SubtitlesOctopus as the subtitle renderer by the looks of it. It could be possible to get the text earlier. Still may be difficult to replicate the original layout (this may be somewhat necessary for when multiple separate text tracks are being shown?) and font, but I've seen other issues discussing the existing font differences already. I might play around some more and see what else I can find out.

Comment 4

[Molly Howell (she/her) (no longer active)]

Right, supporting multiple simultaneous cues well is a known limitation that we're carrying right now. Whatever you could do within that limitation would still be highly appreciated, certainly.