Open Bug 1663844 Opened 2 years ago Updated 5 days ago

Allow using OpenH264 GMP decoder as fallback for video decoding (Mozilla could take Fedora's downstream fix: https://src.fedoraproject.org/rpms/firefox/blob/rawhide/f/mozilla-1663844.patch )

Categories

(Core :: Audio/Video: Playback, task, P5)

Desktop
Linux
task

Tracking

()

People

(Reporter: jya, Assigned: stransky)

References

(Depends on 2 open bugs)

Details

Attachments

(1 file)

Currently the GMP OpenH264 decoder is disabled by default. Primary reason is that our version OpenH264 only supports the main profile.

Once the OpenH264 plugin is updated to 2.1.x we could theoretically use if for most h264 videos.
Though, there are likely performance problem as OpenH264 was geared toward low-latency and doesn't scale well speed-wise.

Some distros (Fedora for instance) ship system OpenH264 package which can be used by Firefox so this bug does not depend on the Mozilla copy.

So far (after some hacks) the GMP plugin (OpenH264 2.1.1) fails to play video content and the screen states

"Video can't be played because it's corrupted..."

GMP/Media log looks like:

[Parent 66952: GMPThread]: D/GMP GMPParent[0x7f9481e6a000|childPid=67420] LoadProcess: Sent StartPlugin to child process
[Child 67340: GMPThread]: D/GMP GMPContentParent::GMPContentParent(this=0x7fdcccd18400), aParent=(nil)
[Child 67340: GMPThread]: D/GMP GMPContentParent::GMPEventTarget(this=0x7fdcccd18400)
[Child 67340: GMPThread]: D/GMP GMPContentParent::AddCloseBlocker(this=0x7fdcccd18400) mCloseBlockerCount=1
[Child 67340: GMPThread]: D/GMP GMPContentParent::GetGMPVideoDecoder(this=0x7fdcccd18400)
[Child 67340, GMPThread] WARNING: Trying to use an dead GMP video decoder: file /home/komat/src/dom/media/gmp/GMPVideoDecoderParent.cpp, line 220
[Child 67340: GMPThread]: D/GMP GMPVideoDecoderParent[0x7fdcd0c63920]::InitDecode()
[Child 67340: GMPThread]: D/GMP GMPContentParent::RemoveCloseBlocker(this=0x7fdcccd18400) mCloseBlockerCount=0
[Child 67340: GMPThread]: D/GMP GMPContentParent::CloseIfUnused(this=0x7fdcccd18400) mVideoDecoders.IsEmpty=false, mVideoEncoders.IsEmpty=true, mChromiumCDMs.IsEmpty=true, mCloseBlockerCount=0
[Child 67340: GMPThread]: V/GMP GMPVideoDecoderParent[0x7fdcd0c63920]::Decode() timestamp=0 keyframe=1
[Child 67340: GMPThread]: V/GMP GMPVideoDecoderParent[0x7fdcd0c63920]::RecvInputDataExhausted()
[Child 67340: GMPThread]: V/GMP GMPVideoDecoderParent[0x7fdcd0c63920]::Decode() timestamp=133467 keyframe=0
[Child 67340: GMPThread]: V/GMP GMPVideoDecoderParent[0x7fdcd0c63920]::RecvDecoded() timestamp=133467 frameCount=1

and nothing more from GMP.

Note: Playback works when OpenH264 is used over ffvpx.

Hm, that may be caused by missing audio decoding.

Looks like the gmp-openh264.cpp module needs to be updated for the new codec version or gecko does something wrong. I'm getting called data exhausted callback from the module and it looks like gecko does not send frames to decoder.

Looks like https://github.com/cisco/openh264/issues/3079 ffmpeg has a similar bug but simple DecodeFrame2 -> DecodeFrameNoDelay replace in gmp-openh264.cpp does not work.

I wonder how the GMP plugin can signalize it needs more data to produce a frame.

There are two issues with OpenH264/GMP used for media playback which causes the plugin is unusable:

  • GMPDecoderModule::SupportsMimeType() returns always false so GMP is not used by PDMFactory
  • mDecodePromise (created at GMPVideoDecoder::Decode()) is newer resolved so the decode queue is not fed and decoding is blocked after first frame decode.

Jean-Yves, the patch works I'm not sure the promise resolve is correct there, can you take a look please?
Thanks.

Assignee: nobody → stransky
Attachment #9177627 - Flags: feedback?(jyavenard)
Comment on attachment 9177627 [details] [diff] [review]
openh264-media-playback.patch

Review of attachment 9177627 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/media/platforms/agnostic/gmp/GMPDecoderModule.cpp
@@ +86,4 @@
>  
>  bool GMPDecoderModule::SupportsMimeType(
>      const nsACString& aMimeType, DecoderDoctorDiagnostics* aDiagnostics) const {
> +  return MP4Decoder::IsH264(aMimeType);

isn't there something available from the GMP API on what it does support?

that's kind of assume that the only plugin available is OpenH264 and that's it

::: dom/media/platforms/agnostic/gmp/GMPVideoDecoder.cpp
@@ +67,5 @@
>    RefPtr<GMPVideoDecoder> self = this;
>    if (v) {
>      mDecodedData.AppendElement(std::move(v));
> +    mDecodePromise.Resolve(std::move(mDecodedData), __func__);
> +    mDecodedData = DecodedData();

the normal flow of operation should be:
Decoded() - Decoded() ... - InputExhausted()/DrainCompleted()

So resolving a single frame as soon as it's returned will potentially means me missed a few. Particularly when draining, which will typically return more than a single frame.

The scenario you describe indicates that InputExhausted isn't called as it should. That's what should be fixed.
Attachment #9177627 - Flags: feedback?(jyavenard) → feedback-

(In reply to Martin Stránský [:stransky] from comment #3)

Looks like the gmp-openh264.cpp module needs to be updated for the new codec version or gecko does something wrong. I'm getting called data exhausted callback from the module and it looks like gecko does not send frames to decoder.

Looks like https://github.com/cisco/openh264/issues/3079 ffmpeg has a similar bug but simple DecodeFrame2 -> DecodeFrameNoDelay replace in gmp-openh264.cpp does not work.

I wonder how the GMP plugin can signalize it needs more data to produce a frame.

InputExhausted is called when that happens.

The API mimics the Windows Media Foundation (WMF) decoder.

So currently when InputExhausted is called, the promise is resolved, if the DecodedData returned is empty, the MediaFormatReader will call the decoder with more compressed frames.
If the DecodedData returned a frame, the GMP decoder will be called again once a new frame is needed.

Depends on: 1670333

(In reply to Jean-Yves Avenard [:jya] from comment #6)

So currently when InputExhausted is called, the promise is resolved, if the DecodedData returned is empty, the MediaFormatReader will call the decoder with more compressed frames.
If the DecodedData returned a frame, the GMP decoder will be called again once a new frame is needed.

I tried to fiddle with InputExhausted on plugin side but without any luck. When InputExhausted is utilized then codec keep to decode the stream but nothing in shown on Firefox side. I'll investigate it more when Bug 1619988 (plugin update to 2.1.1) is fixed.

Severity: -- → S4
Priority: -- → P4
Duplicate of this bug: 1608252

has this been fixed yet with latest releases of firefox?

Blocks: 1764436
OS: Unspecified → Linux
Hardware: Unspecified → Desktop
Summary: Allow using OpenH264 GMP decoder as fallback for video decoding → Allow using OpenH264 GMP decoder as fallback for video decoding (Consider upstreaming Fedora's bug fix: https://src.fedoraproject.org/rpms/firefox/blob/rawhide/f/mozilla-1663844.patch )
Summary: Allow using OpenH264 GMP decoder as fallback for video decoding (Consider upstreaming Fedora's bug fix: https://src.fedoraproject.org/rpms/firefox/blob/rawhide/f/mozilla-1663844.patch ) → Allow using OpenH264 GMP decoder as fallback for video decoding (Would be fixed by upstreaming Fedora's bug fix: https://src.fedoraproject.org/rpms/firefox/blob/rawhide/f/mozilla-1663844.patch )
Summary: Allow using OpenH264 GMP decoder as fallback for video decoding (Would be fixed by upstreaming Fedora's bug fix: https://src.fedoraproject.org/rpms/firefox/blob/rawhide/f/mozilla-1663844.patch ) → Allow using OpenH264 GMP decoder as fallback for video decoding (Mozilla could take Fedora's downstream fix: https://src.fedoraproject.org/rpms/firefox/blob/rawhide/f/mozilla-1663844.patch )

Any improvement here with v2.3.0?

You need to log in before you can comment on or make changes to this bug.