1663844 - Allow using OpenH264 GMP decoder as fallback for video decoding (Mozilla could take Fedora's downstream fix: https://src.fedoraproject.org/rpms/firefox/blob/rawhide/f/mozilla-1663844.patch )

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, task, P5)

Description

[Jean-Yves Avenard [:jya]]

Currently the GMP OpenH264 decoder is disabled by default. Primary reason is that our version OpenH264 only supports the main profile.

Once the OpenH264 plugin is updated to 2.1.x we could theoretically use if for most h264 videos.
Though, there are likely performance problem as OpenH264 was geared toward low-latency and doesn't scale well speed-wise.

Comment 1

[Martin Stránský [:stransky] (ni? me)]

Some distros (Fedora for instance) ship system OpenH264 package which can be used by Firefox so this bug does not depend on the Mozilla copy.

So far (after some hacks) the GMP plugin (OpenH264 2.1.1) fails to play video content and the screen states

"Video can't be played because it's corrupted..."

GMP/Media log looks like:

[Parent 66952: GMPThread]: D/GMP GMPParent[0x7f9481e6a000|childPid=67420] LoadProcess: Sent StartPlugin to child process
[Child 67340: GMPThread]: D/GMP GMPContentParent::GMPContentParent(this=0x7fdcccd18400), aParent=(nil)
[Child 67340: GMPThread]: D/GMP GMPContentParent::GMPEventTarget(this=0x7fdcccd18400)
[Child 67340: GMPThread]: D/GMP GMPContentParent::AddCloseBlocker(this=0x7fdcccd18400) mCloseBlockerCount=1
[Child 67340: GMPThread]: D/GMP GMPContentParent::GetGMPVideoDecoder(this=0x7fdcccd18400)
[Child 67340, GMPThread] WARNING: Trying to use an dead GMP video decoder: file /home/komat/src/dom/media/gmp/GMPVideoDecoderParent.cpp, line 220
[Child 67340: GMPThread]: D/GMP GMPVideoDecoderParent[0x7fdcd0c63920]::InitDecode()
[Child 67340: GMPThread]: D/GMP GMPContentParent::RemoveCloseBlocker(this=0x7fdcccd18400) mCloseBlockerCount=0
[Child 67340: GMPThread]: D/GMP GMPContentParent::CloseIfUnused(this=0x7fdcccd18400) mVideoDecoders.IsEmpty=false, mVideoEncoders.IsEmpty=true, mChromiumCDMs.IsEmpty=true, mCloseBlockerCount=0
[Child 67340: GMPThread]: V/GMP GMPVideoDecoderParent[0x7fdcd0c63920]::Decode() timestamp=0 keyframe=1
[Child 67340: GMPThread]: V/GMP GMPVideoDecoderParent[0x7fdcd0c63920]::RecvInputDataExhausted()
[Child 67340: GMPThread]: V/GMP GMPVideoDecoderParent[0x7fdcd0c63920]::Decode() timestamp=133467 keyframe=0
[Child 67340: GMPThread]: V/GMP GMPVideoDecoderParent[0x7fdcd0c63920]::RecvDecoded() timestamp=133467 frameCount=1

and nothing more from GMP.

Note: Playback works when OpenH264 is used over ffvpx.

Comment 2

[Martin Stránský [:stransky] (ni? me)]

Hm, that may be caused by missing audio decoding.

Comment 3

[Martin Stránský [:stransky] (ni? me)]

Looks like the gmp-openh264.cpp module needs to be updated for the new codec version or gecko does something wrong. I'm getting called data exhausted callback from the module and it looks like gecko does not send frames to decoder.

Looks like https://github.com/cisco/openh264/issues/3079 ffmpeg has a similar bug but simple DecodeFrame2 -> DecodeFrameNoDelay replace in gmp-openh264.cpp does not work.

I wonder how the GMP plugin can signalize it needs more data to produce a frame.

Comment 4

[Martin Stránský [:stransky] (ni? me)]

Attachment: openh264-media-playback.patch

There are two issues with OpenH264/GMP used for media playback which causes the plugin is unusable:

• GMPDecoderModule::SupportsMimeType() returns always false so GMP is not used by PDMFactory
• mDecodePromise (created at GMPVideoDecoder::Decode()) is newer resolved so the decode queue is not fed and decoding is blocked after first frame decode.

Jean-Yves, the patch works I'm not sure the promise resolve is correct there, can you take a look please?
Thanks.

Comment 5

[Jean-Yves Avenard [:jya]]

Comment on attachment 9177627 [details] [diff] [review]
openh264-media-playback.patch

Review of attachment 9177627 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/media/platforms/agnostic/gmp/GMPDecoderModule.cpp
@@ +86,4 @@
>  
>  bool GMPDecoderModule::SupportsMimeType(
>      const nsACString& aMimeType, DecoderDoctorDiagnostics* aDiagnostics) const {
> +  return MP4Decoder::IsH264(aMimeType);

isn't there something available from the GMP API on what it does support?

that's kind of assume that the only plugin available is OpenH264 and that's it

::: dom/media/platforms/agnostic/gmp/GMPVideoDecoder.cpp
@@ +67,5 @@
>    RefPtr<GMPVideoDecoder> self = this;
>    if (v) {
>      mDecodedData.AppendElement(std::move(v));
> +    mDecodePromise.Resolve(std::move(mDecodedData), __func__);
> +    mDecodedData = DecodedData();

the normal flow of operation should be:
Decoded() - Decoded() ... - InputExhausted()/DrainCompleted()

So resolving a single frame as soon as it's returned will potentially means me missed a few. Particularly when draining, which will typically return more than a single frame.

The scenario you describe indicates that InputExhausted isn't called as it should. That's what should be fixed.

Comment 6

[Jean-Yves Avenard [:jya]]

(In reply to Martin Stránský [:stransky] from comment #3)

Looks like the gmp-openh264.cpp module needs to be updated for the new codec version or gecko does something wrong. I'm getting called data exhausted callback from the module and it looks like gecko does not send frames to decoder.

Looks like https://github.com/cisco/openh264/issues/3079 ffmpeg has a similar bug but simple DecodeFrame2 -> DecodeFrameNoDelay replace in gmp-openh264.cpp does not work.

I wonder how the GMP plugin can signalize it needs more data to produce a frame.

InputExhausted is called when that happens.

The API mimics the Windows Media Foundation (WMF) decoder.

So currently when InputExhausted is called, the promise is resolved, if the DecodedData returned is empty, the MediaFormatReader will call the decoder with more compressed frames.
If the DecodedData returned a frame, the GMP decoder will be called again once a new frame is needed.

Comment 7

[Martin Stránský [:stransky] (ni? me)]

(In reply to Jean-Yves Avenard [:jya] from comment #6)

So currently when InputExhausted is called, the promise is resolved, if the DecodedData returned is empty, the MediaFormatReader will call the decoder with more compressed frames.
If the DecodedData returned a frame, the GMP decoder will be called again once a new frame is needed.

I tried to fiddle with InputExhausted on plugin side but without any luck. When InputExhausted is utilized then codec keep to decode the stream but nothing in shown on Firefox side. I'll investigate it more when Bug 1619988 (plugin update to 2.1.1) is fixed.

Comment 9

[tallyo]

has this been fixed yet with latest releases of firefox?

Comment 10

[Arthur K. (he/him)]

Any improvement here with v2.3.0?

Comment 11

[Andrew Osmond [:aosmond] (he/him)]

I have done local testing using 2.3.1. I have had success playing some H264 videos that I found randomly that the 1.8.1.2 plugin fails with today. I needed to incorporate the patch attached in order to get WebRTC/video playback to correct detect the plugin support.

Comment 12

[Andrew Osmond [:aosmond] (he/him)]

I will fix the patch based on jya's comments and get something landable.

Comment 13

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1663844 - Part 1. Expose GMP video decoder codec support if enabled.

By default, decoding via GMP plugins is disabled by the pref
media.gmp.decoder.enabled. However if it is set to true, we never
actually exposed what codecs were supposed by the plugins (EME was
special cased). This patch corrects that.

Comment 14

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1663844 - Part 2. Ensure we resolve decode promises with GMPVideoDecoder.

This patch allows us to track multiple outstanding decodes and fulfill
the decode promise when they have all been completed. This fixes how the
media state machine would get stuck waiting for the decode promise to
fulfill when using the OpenH264 decoder.

Depends on D174354

Comment 15

[Andrew Osmond [:aosmond] (he/him)]

Comment on attachment 9177627 [details] [diff] [review]
openh264-media-playback.patch

>diff --git a/dom/media/platforms/agnostic/gmp/GMPDecoderModule.cpp b/dom/media/platforms/agnostic/gmp/GMPDecoderModule.cpp
>--- a/dom/media/platforms/agnostic/gmp/GMPDecoderModule.cpp
>+++ b/dom/media/platforms/agnostic/gmp/GMPDecoderModule.cpp
>@@ -86,7 +86,7 @@ bool GMPDecoderModule::SupportsMimeType(
> 
> bool GMPDecoderModule::SupportsMimeType(
>     const nsACString& aMimeType, DecoderDoctorDiagnostics* aDiagnostics) const {
>-  return false;
>+  return MP4Decoder::IsH264(aMimeType);
> }
> 
> }  // namespace mozilla
>diff --git a/dom/media/platforms/agnostic/gmp/GMPVideoDecoder.cpp b/dom/media/platforms/agnostic/gmp/GMPVideoDecoder.cpp
>--- a/dom/media/platforms/agnostic/gmp/GMPVideoDecoder.cpp
>+++ b/dom/media/platforms/agnostic/gmp/GMPVideoDecoder.cpp
>@@ -67,6 +67,8 @@ void GMPVideoDecoder::Decoded(GMPVideoi4
>   RefPtr<GMPVideoDecoder> self = this;
>   if (v) {
>     mDecodedData.AppendElement(std::move(v));
>+    mDecodePromise.Resolve(std::move(mDecodedData), __func__);
>+    mDecodedData = DecodedData();
>   } else {
>     mDecodedData.Clear();
>     mDecodePromise.RejectIfExists(

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0da411d6fadb
Part 1. Expose GMP video decoder codec support if enabled. r=media-playback-reviewers,azebrowski
https://hg.mozilla.org/integration/autoland/rev/c948f6a27e6d
Part 2. Ensure we resolve decode promises with GMPVideoDecoder. r=media-playback-reviewers,azebrowski

Comment 17

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/0da411d6fadb
https://hg.mozilla.org/mozilla-central/rev/c948f6a27e6d

Comment 18

[Darkspirit]

Thanks! Verified on Debian Testing:

• Downloaded and extracted http://ciscobinary.openh264.org/mozilla-openh264-2.3.1-2.fc39.x86_64.rpm from https://codecs.fedoraproject.org/openh264/39/x86_64/os/Packages/m/

• MOZ_GMP_PATH="~/Downloads/openh264/usr/lib64/mozilla/plugins/gmp-gmpopenh264/system-installed" MOZ_LOG="GMP:5,PlatformDecoderModule:5" mozregression --launch ddba2e10fb26 --pref media.gmp.decoder.enabled:true media.gmp.decoder.preferred:true media.gmp-gmpopenh264.autoupdate:false media.gmp-gmpopenh264.version:"system-installed" -P stdout -a https://bug1619882.bmoattachments.org/attachment.cgi?id=9149605

• MOZ_GMP_PATH="~/Downloads/openh264/usr/lib64/mozilla/plugins/gmp-gmpopenh264/system-installed" MOZ_LOG="GMP:5,PlatformDecoderModule:5" mozregression --launch ddba2e10fb26 --pref media.av1.enabled:false media.webm.enabled:false media.ffvpx.enabled:false media.rdd-vpx.enabled:false media.gmp.decoder.enabled:true media.gmp.decoder.preferred:true media.gmp-gmpopenh264.autoupdate:false media.gmp-gmpopenh264.version:"system-installed" media.utility-process.enabled:false -P stdout -a https://www.youtube.com/watch?v=aIVsxSB8HKk