Movie playback (especially H.264 & AAC) acceleration is required for Yocto based devices. Although GStreamer is the de facto standard media framework in Yocto world, unfortunately GStreamer support of Gecko is already dropped (Bug 1234092). Some SoC vendors provides OpenMAX IL library for Yocto but we cannot expect that it's always available. For example, i.MX6 doesn't provide OpenMAX on Yocto. Which way we have to choose?: * Port to each SoC using it's native API * some (or most?) of them provides OpenMAX * Revive or re-implement GStreamer support
In bug1305341, we are now porting Gecko to following SoC: * Renesas RZ/G1 series * Native codecs: provided as OpenMAX IL library * GStreamer: available via wrapper library (gst-omx) * NXP i.MX6 series * Native codecs: original API (libvpu) * GStreamer: available Since our current work is based on ESR45 which has GStreamer support, I'm now trying to enable acceleration via GStreamer. Although it can enable native codecs, I caught high CPU usage than software codecs. I'm now investigating this issue...
On the other hand I'm also investigating the way to use OpenMAX. Although Gecko has OpenMAX support, it depends on Android. It seems that most codes are placed under "android" namespace. I'm not sure whether it can be easily port to Yocto (pure GNU/Linux) or not.
> I caught high CPU usage than software codecs. "software codecs" means FFmpeg.
> I caught high CPU usage than software codecs. I'm now investigating this issue... FYI: https://github.com/mozilla-japan/gecko-embedded/issues/3 (Sorry, written in Japanese)
Gstreamer won't be reintroduced , it's fundamentally incompatible with HTML5 media source extension. HW acceleration enabled via gstreamer wouldn't provide much benefit as you would need to use basic layer (gstreamer copies everything back to a a software buffer)
Thank you for your comment. I have a question about OpenMAX. I'm now reading dom/media/platform/omx. If I want to implement OpenMAX support, all I have to do is implementing child class of OmxPlatformLayer?
Alfredo - what is the current state of OMX support?
(In reply to Takuro Ashie from comment #6) > Thank you for your comment. > > I have a question about OpenMAX. I'm now reading dom/media/platform/omx. > If I want to implement OpenMAX support, all I have to do is implementing > child class of OmxPlatformLayer? You need to implement OmxPlatformLayer and OmxPromiseLayer::BufferData. OmxPlatformLayer is a wrapper for OMX api, it should be straight forward to implement it. BufferData could be little complicated depends on video buffer type. If you plan to use graphic accelerated buffer, BufferData acts as a wrapper of graphic buffer. Otherwise, it just likes the normal OMX buffer. You can refer to GonkOmxPlatformLayer, it should be able to provide you basic idea how to implement it. Let me know it you have other questions.
Thanks! I'll try to implement it. Memo: In addition OmxDecoderModule have to be initiated at dom/media/platforms/PDMFactory.cpp
Created attachment 8805487 [details] [diff] [review] Part1: Update the code base of dom/media/platform/omx in ESR45
Created attachment 8805489 [details] [diff] [review] Add the initial implementation of PureOmxPlatformLayer It's a concrete class of OmxPlatformLayer for accessing OpenMAX IL libraries directly. Although it's not stable yet, I confirmed that it works on Renesas's RZ/G and reduce CPU usage than FFmpeg.
FYI: I'm now developing it at the following repository: https://github.com/mozilla-japan/gecko-dev/tree/esr45-omx-linux
Created attachment 8811109 [details] [diff] [review] Part1: Update the code base of dom/media/platform/omx in ESR45 (v2) Fix the wrong commit comment.
Attachment #8805487 - Attachment is obsolete: true
Created attachment 8811111 [details] [diff] [review] Part2: Add the initial implementation of PureOmxPlatformLayer (v2) Improve stability: * Use TaskQueue to dispatch callbakcs from OpenMAX * Release OMX buffers correctly
Attachment #8805489 - Attachment is obsolete: true
TODO: * Port to mozilla-central * Port to Raspberry Pi * Implement zero-copy mode by using OMX_UseEGLImage()
Created attachment 8873305 [details] [diff] [review] Part1: Add the initial implementation of PureOmxPlatformLayer to ESR52 I pick the latest patch from https://github.com/mozilla-japan/meta-browser/tree/firefox-52.1esr. It's for ESR52, and I confirmed it only on Renesas's RZ/G1 series. It seems that it can't be applied against current trunk. I'll fix it.
Created attachment 8873307 [details] [diff] [review] Part2: Fix a crash on opening about:support (for ESR52)
Hi. Are you still interested in implementing this? We're considering removing the OMX support code, since it's currently unused.
Thank you for your notification. Yes, we are still interesting in it. Just now we are porting the code to m-c. https://github.com/KSR-Yasuda/meta-browser/tree/feature/firefox-60/OpenMAX/recipes-mozilla/firefox/firefox/openmax https://github.com/webdino/gecko-dev/tree/omx-linux-dev-wip It seems that the current code can decode a first frame of a H.264 movie but cannot decode following frames (on Renesas RZ/G1 series). We are now investigating this issue. Since the Wayland port is recently landed on m-c (bug635134), we are now catching up m-c (our main target is Wayland).
Ok, thanks for the update. Dan, it looks like we have downstream users of OMX, so we should keep the code as long as they're willing to help maintain it.
(In reply to Takuro Ashie from comment #19) > https://github.com/webdino/gecko-dev/tree/omx-linux-dev-wip Rebase & remove needless code: https://github.com/webdino/gecko-dev/tree/omx-linux-dev-wip-2
(In reply to Takuro Ashie from comment #19) > It seems that the current code can decode a first frame of a H.264 movie but > cannot decode following frames (on Renesas RZ/G1 series). We are now It's not correct. It can play when I specify a movie file directly. e.g.) $ firefox https://www.quirksmode.org/html5/videos/big_buck_bunny.mp4 But it can't play anymore when I press the play button after the first playback is finished. It seems that the decoder thread is stalled on shutting down. (When a HTML file with a video element is loaded, it will be stalled after decoding the preview picture. This is the reason I thought that it can decode only one frame.) The issue seems resolved by the following patch: * https://github.com/webdino/gecko-dev/commit/c8cd0ca2e914a2bd4c7f7fc15d49feffd6c1ae41 I'll attach it to this bug after I clean up all patches.
You need to log in before you can comment on or make changes to this bug.