Closed Bug 1619523 Opened 4 years ago Closed 4 years ago

[X11] Implement ffmpeg/VAAPI video playback

Categories

(Core :: Audio/Video: Playback, enhancement, P5)

Product:
Core
Component:
Audio/Video: Playback
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla80
Tracking Flags:
Tracking Status
relnote-firefox --- -
firefox80 --- fixed

People

(Reporter: stransky, Assigned: stransky, NeedInfo)

References

(Blocks 1 open bug)

Details

Attachments

(4 files, 4 obsolete files)

x11-dmabuf-wip.patch
8.28 KB, patch
Details | Diff | Splinter Review
Bug 1619523 [Linux] Remove Wayland check at gfxPlatformGtk::UseHardwareVideoDecoding(), make VA-API decoding possible on X11 r?jhorak
47 bytes, text/x-phabricator-request
Details | Review
Bug 1619523 [Linux] Remove Wayland check at gfxPlatformGtk::UseDMABufVideoTextures(), make VA-API decoding possible on X11 r?jhorak
47 bytes, text/x-phabricator-request
Details | Review
{ "application": { "name": "Firefox", "osVersion": "Linux 5.7.7-1-MANJARO #1 SMP PREEMPT Wed Jul 1 10:17:47 UTC 2020", "version": "80.0a1", "buildID": "20200703215022", "distributionID": "", "userAgent": "Mozilla/5.0 (X11; Linux
30.83 KB, text/plain
Details

Although I'm not personally interested to implement VA-API accelerated playback on X11 I can guide anyone who want's to work on that. There's short TODO list what needs to be done for it:

The ffmpeg/shared surface/webrender/compositor parts are mostly backend independent so there isn't needed any extensive work there and recent wayland path can be reused.

See Also: → 1210727
Depends on: 1210727
See Also: 1210727 →

Dmabuf for X11 is Bug 1580166

Depends on: 1580166

Hi.
I'd like to implement VA-API support on X11 backend. I use X11 Pixmaps for passing image data between content and browser processes. That approach is less disruptive as it does not require transition to EGL and/or dmabuf. (It should still work after EGL transition too, as long texture_from_pixmap extension is available.)

So the question is, should I use this bug as a tracker, or should I create a new one?

Flags: needinfo?(bvandyk)

Please don't. bug 788319 is anyway needed for bug 1620076. The goal is to reuse codepaths, not to add duplicate. GLX should be removed and X11 connections be prevented (bug 1129492 comment 24). Surfman already switched to EGL on X11. VAAPI decoding won't stay in the content process (bug 1595994).

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #3)

bug 788319 is anyway needed for bug 1620076.

I don't see how a working hw-decode on X11 prevents that. Both GLX and EGL support texture-from-pixmap. When EGL time comes, changes to my code should be almost trivial. BindTexImage/ReleaseTexImage are already in the codebase for both GLX and EGL.

The goal is to reuse codepaths, not to add duplicate.

I don't think I understand what duplication are you talking about. Most of the code paths are the same. Bindings are different, but one cannot merge X11 and Wayland bindings, since they have different APIs by design.

GLX should be removed and X11 connections be prevented (bug 1129492 comment 24).

That's great, but is switching to EGL already scheduled? Why can't we have a working hw-decode now on GLX, and gradually migrate to EGL later? I don't get it. With switching to EGL will come dropping of unneeded X11 connections. But again, is it coming anytime soon?

Surfman already switched to EGL on X11.

Isn't it a Servo-related project? Haven't found it mentioned in bugzilla's search results.

VAAPI decoding won't stay in the content process (bug 1595994).

I don't see why moving it to another process will make any difference. Decoding and rendering are already in separate processes. Images were transferred from a content process to the browser process. After moving, images will be transferred from RDD process to the browser process.

(In reply to Rinat from comment #4)

GLX should be removed and X11 connections be prevented (bug 1129492 comment 24).

That's great, but is switching to EGL already scheduled? Why can't we have a working hw-decode now on GLX, and gradually migrate to EGL later? I don't get it. With switching to EGL will come dropping of unneeded X11 connections. But again, is it coming anytime soon?

If I am not misunderstanding something:

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #5)

bug 1129492 comment 24 must harden the sandbox soon: "Remove X11 connections from content processes in all cases, for security"
Wouldn't pixmap VAAPI introduce another X11 connection?
(bug 1635451 and bug 1636493 anyway demand strong reduction of X11 connections.)

Yeah, but as you mentioned, video decoders are going to RDD process, which will relieve content processes from X11 connections.
Also, a single X11 connection is enough. It can be shared by multiple decoder instances living in RDD process.

DMABUF VAAPI should be used on X11 as well (bug 1580166 + comment 0).

Sure, that's a great idea, but that requires EGL. Which wasn't very great in my experience even with Intel hardware. When I was trying to get GLES2 for PPAPI-to-NPAPI Flash translation layer, EGL was the first choice. Unfortunately, that somehow causes a whole machine lock-up half of the time. Not an application crash or freeze, whole machine just froze. There wasn't any locking issues with GLX though, so hardware malfunction is unlikely.

Granted, it was about 5 years ago, but the grudge is still there. And systems with older drivers/software still there too.

EGL (bug 788319 blocks bug 1620076 + bug 1580166): As soon as possible. It would be great if you could try to tackle bug 788319 instead.

If you are going to switch to EGL without any fallback to GLX, I believe a lot of failures will arise. Even with GLX being deprecated for almost a decade now, EGL is still not tested widely. And I still expect EGL to fail more often than GLX. Also I believe P5 priority for this bug was set because transition to EGL is likely to cause issues.

VAAPI can't get enabled by default for security reasons as long as bug 1595994 and bug 1619585 comment 17 are not fixed.

I was never aiming for enabled-by-default. Hardware decoding is too failure-prone. Unfortunately, Linux is not tested as widely as Windows is.

WebRender/X11: bug 1614523

I added X11 Pixmap support for WebRender too. There are some issues with texture-from-pixmap, as its presense triggers untested paths in WebRender bindings even without my changes, but I still believe that's solvable.

I'm not saying that EGL+dmabuf approach is inferior to GLX+Pixmap. But I want a usable feature in the near future, not in another 5 years. Currently code uses GLX on X11. That's why I'm using GLX+Pixmap. It's less disruptive and a lot less likely to cause issues.

WebRender (which VAAPI depends on) is - on Nightly - currently enabled for Mesa 18+ (Ubuntu 16.04 LTS). If there are issues, they would be present on Wayland as well. Surfman switched to EGL because GLX and pixmaps turned out to be problematic ("because GLX is so busted").
Everyone wants bug 1620076: bug 788319 is needed for it on X11 - it's not P5. VAAPI (on Wayland and X11) is P5 because it's community-maintained.

WebRender (which VAAPI depends on) is - on Nightly - currently enabled for Mesa 18+ (Ubuntu 16.04 LTS). If there are issues, they would be present on Wayland as well.

I believe that currently in Nightly WebRender still gets its GL context through GLX.
I was talking about EGL issues. You are currently not seeing them since EGL is not used.
Hope there will be no EGL issues at all. But can't force myself to believe that.

which VAAPI depends on (WebRender)

It's the current implementation of VAAPI-based decoder that depends on WebRender.

Here is a WIP version of the patch I was working on: https://github.com/i-rinat/firefox/commit/73ee73df4e09ff0a5cfd6c30d92d6070fe722c21
Decoding via VA-API works in basic, opengl, and WebRender compositor modes. Basic compositor works better with XRender enabled, that removes the necessity of reading image back to the CPU memory. OpenGL and WebRender compositors require gfx.use-glx-texture-from-pixmap enabled to render image from X11 Pixmaps.
The main point is that WebRender is not a requirement for VAAPI decoding and rendering.

I know that XRender is being deprecated by Firefox and all that, but must note that basic compositor with XRender and VA-API enabled gave the smallest power consumption during video playback.

because GLX is so busted

Wasn't GLX blamed for its thread unsafety? Xlib is also not thread-safe. That hurts a lot if you are trying to use foreign Display * in multiple threads. And as I can see, Surfman calls XInitThreads() which is generally not a good idea for a library. Unless one can call it before any XOpenDisplay() is called, it will strike back at some point. It's easier avoid Xlib locking functions and just use a mutex instead. Or open a connection per-thread.
But Firefox is not an library which is used by other applications. While only a single thread at a time uses Xlib or GLX functions, there should be no issues. Both Xlib and GLX are already working in Firefox with proper locking in place.

VAAPI (on Wayland and X11) is P5 because it's community-maintained.

Ah, OK.
I was thinking that P5 is a "no-go" because its says: "We basically never want this".

(In reply to Rinat from comment #8)

I believe that currently in Nightly WebRender still gets its GL context through GLX.

If there would be EGL bugs on recent Linux distributions, Wayland were already affected, as it's used by default on Fedora, Debian, etc.

Here is a WIP version of the patch I was working on: https://github.com/i-rinat/firefox/commit/73ee73df4e09ff0a5cfd6c30d92d6070fe722c21

Oh dear. Why didn't you ask before? Taking it would delay/prevent the outlined path of EGL partial present, DMABUF/X11 and sandbox hardening.
Please try to tackle bug 788319 and bug 1580166.

Basic compositor works better with XRender enabled, that removes the necessity of reading image back to the CPU memory. OpenGL and WebRender compositors require gfx.use-glx-texture-from-pixmap enabled to render image from X11 Pixmaps.

This pref caused crashes when I tested it last. I'm glad it's going to be replaced by DMABUF.

The main point is that WebRender is not a requirement for VAAPI decoding and rendering.

It was defined as one as VAAPI decoding should only be used if zero copy is possible.

I know that XRender is being deprecated by Firefox and all that, but must note that basic compositor with XRender and VA-API enabled gave the smallest power consumption during video playback.

Wayland should be used and improved to achieve that.

If there would be EGL bugs on recent Linux distributions, Wayland were already affected, as it's used by default on Fedora, Debian, etc.

OK. My experience is limited and I can't say anything other than what I've seen myself. You are right.
It doesn't change the fact that EGL is not used in Firefox yet though.

Oh dear. Why didn't you ask before?

It's not like I'm wandering around having nothing to do. I want a feature. Nobody seems to be working on that feature. I start to work on the feature. I don't mind sharing results of my work. And if upstream accepts patches, that's great. If not... Well, life's unfair.

Taking it would delay/prevent the outlined path of EGL partial present, DMABUF/X11

That's not true. How is that even possible? It's not like transitioning to EGL will break Pixmap-based image sharing. It's enough to change GLX-based texture-from-pixmap to EGL-based texture-from-pixmap to keep image sharing working. It's possible to then implement dmabuf-based image sharing and phase out Pixmap-based sharing. There is a clear migration path.

and sandbox hardening

Well, that's true, but not crucial. As you can see in the bugs you mentioned, it's the content process sandbox which is hardened. Hardening the content process sandbox have a lot of sense, since content processes execute Javascript code from the wild. I didn't see anything about RDD hardening. Its level of isolation is decreased instead.

And to reiterate, having a GLX-using decoder is not preventing implementation of a decoder that uses only DRM connections. The same way as existing of X11 backend does not prevent Wayland backend from being developed.

Please try to tackle bug 788319 and bug 1580166.

Bug 788319 is a task for a single developer. It won't be completed faster if two people start working on it. From that and your multiple propositions to work on bug 788319, I can conclude that nobody is seriously working on bug 788319. If despite the importance of EGL transition, nobody is working on the issue, there is no expected completion date. It may be in a month, may be in a year or five. Or may be never. Looks like a mirage to me.

Maybe later I'll try to work on that too, but definitely not today. I still have bugs to eliminate from my patch. Dropping it just to start to work on another feature which future is dubious to me, is not a sound idea. I doubt that I'm your final chance. But if I am, I'm not a good candidate. At least not at the moment.

This pref caused crashes when I tested it last. I'm glad it's going to be replaced by DMABUF.

Of course it crashed! But it's not that texture-from-pixmap is broken somehow. It's just WebRender bindings lack X11 Pixmap support. When you turn that pref on, content processes start to pass X11 Pixmaps which WebRender in the browser process have no idea how to deal with. One needs to implement methods CreateRenderTexture, PushResourceUpdates, PushDisplayItems for X11TextureHost. There are implementations in the patch I mentioned in the previous comment. Then enabling gfx.use-glx-texture-from-pixmap makes WebGL Aquarium (500 fish) go from about 45 fps to about 60 fps in Nightly with WebRender. It's not solid 60 fps like in Chrome unfortunately, but you must admit it's a solid boost.

DMABUF sharing is fundamentally the same operation as texture-from-pixmap. It's just different API approach. If GLX/EGL texture-from-pixmap is failing on some machine, DMABUF will fail too. Blaming texture-from-pixmap based on experience with software that is not using texture-from-pixmap is shady.

It was defined as one as VAAPI decoding should only be used if zero copy is possible.

One should chase lower power consumption, not lower number of copies. Zero copy operation may exhibit a lower performance and higher power consumption if, for example, source and target tiling is different. Again, it's not like having copying implementation is preventing somebody from improving it by lowering number of copies.

Wayland should be used and improved to achieve that.

Seriously? Another Wayland vs. X11 holywar? In a bug with "[X11]" in name?

I also gave this a look today and if I understand things correctly, all possibilities to go forward here require some serious effort. That is assuming the goal is shipping things enabled by default - because everything else I'd consider wasted time.

  • use EGL with DMABUF: EGL facilitates things quite a bit, but what's missing is GBM support on X11. So we'd need to reimplement things that are much easier on Wayland + potential bugs in drivers IIUC. No NVidia support, i.e. we'd only target platforms that could as well use Wayland.
  • use EGL with pixmaps: requires bug 1640147 and dealing with a potential bug galore that hasn't been dealt with in the past (e.g. bug 1193015, (1)). Also, probably undesirable for future sandboxing efforts.
  • use GLX with pixmaps: bug galore around pixmap enabling as above, also same sandboxing problems. But most importantly: would need it's own implementation for partial damage with Webrender as WR is a precondition. Or go without partial damage, which might be acceptable for non-battery systems with dedicated GPU (2). But that increases maintenance burden.
  • use GLX with DMABUF: should be out of question.

I personally concluded I don't want to go down any of these paths. Bug 1640048 so far turned out to be quite "easy", being more about enabling than implementing, and gives us some features for free. The things above in turn look like rabbit wholes to me :/

1: https://searchfox.org/mozilla-central/source/modules/libpref/init/StaticPrefList.yaml#4069-4070
2: https://mozillagfx.wordpress.com/2019/10/22/dramatically-reduced-power-usage-in-firefox-70-on-macos-with-core-animation/

Thanks for having a look at this, too! I think that's why bug 1580166 was created for contributers who really want to invest their time on bringing existing Wayland enhancements to EGL/X11/Mesa, even when they could log in to Wayland instead. It would still be great if someone could try to tackle it. Improvements for legacy X11 should be a byproduct of the Wayland work (to share code paths), for better long-term maintenance.

While Gnome supports Wayland well, it's not the case for other Desktop-/Window- managers, in particular KDE is still problematic with Wayland.

(not saying that it's not something that should be fixed on KDE side, just noting that it's not so simple to switch to Wayland)

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #12)

…I think that's why bug 1580166 was created for contributers who really want to invest their time on bringing existing Wayland enhancements to EGL/X11/Mesa, even when they could log in to Wayland instead…

(In reply to Hubert Kario from comment #13)

While Gnome supports Wayland well, it's not the case for other Desktop-/Window- managers, in particular KDE is still problematic with Wayland.
(not saying that it's not something that should be fixed on KDE side, just noting that it's not so simple to switch to Wayland)

Indeed. Just recently one of kwin maintainers and its most prolific developers had to fork entire kwin subsystem [1] to allow himself to actually work on Wayland support because "the committee", that doesn't seem to do anything substantial, was dragging their feet on reviews and blocked his changes for both X11 [2] and Wayland [3] versions. Which is also why we have kwin-lowlatency fork [4] for X11 only, despite how grumpy retired kwin creator is about forks [5].

Moreover, even after all these years, colour correction / display colour profiles application is nowhere to be found [6] on Wayland, at "best" there is some hipster "night" colour shift fad that smears everything. Which is especially bad now that we have 10-bit HDR displays proliferation that widely differ in their capabilities and require tone mapping, gamma, contrast corrections for a decent output.

So, I wouldn't count on Wayland for another 5-10 years at best.
Unless you're going to fix all that mess and coax all mainstream distros into making it the universal default.

1: https://subdiff.org/blog/2020/the-k-win-ft-project/
2: https://phabricator.kde.org/T11071
3: https://phabricator.kde.org/T5710
4: https://github.com/tildearrow/kwin-lowlatency#additional-options-menu
5: https://blog.martin-graesslin.com/blog/2020/04/on-forking/
6: https://phabricator.kde.org/T9232

I'll look at it as Bug 1640048 has a patch to switch X11 to EGL.

Assignee: nobody → stransky

(In reply to Martin Stránský [:stransky] from comment #15)

I'll look at it as Bug 1640048 has a patch to switch X11 to EGL.

I guess this path will be for the WebRender compositor?

I'll make support for the basic compositor then, in Bug 1644610. Release versions of Firefox still use basic by default on my platform (Intel Skylake, system libraries are from Debian testing). It would be nice to have a hardware decoding in that configuration too.

(In reply to Rinat from comment #16)

(In reply to Martin Stránský [:stransky] from comment #15)

I'll look at it as Bug 1640048 has a patch to switch X11 to EGL.

I guess this path will be for the WebRender compositor?

I'll make support for the basic compositor then, in Bug 1644610. Release versions of Firefox still use basic by default on my platform (Intel Skylake, system libraries are from Debian testing). It would be nice to have a hardware decoding in that configuration too.

Yes, that's for WebRender/GL compositor.

With patches from Bug 1580166 and Bug 1640048 it's relatively easy to implement X11 dmabuf vaapi playback. There's a rough WIP. Main difference is using drm display instead of the wayland one.

No longer blocks: 1210726
Depends on: 1640048
No longer depends on: 1210727

much thanks for your work Martin, and to be honest I will pay something for me, a self gift, with the bounty money, you deserved this.

will there be some way to check if firefox is using or not video acceleration?

(In reply to albertogomezmarin from comment #19)

much thanks for your work Martin, and to be honest I will pay something for me, a self gift, with the bounty money, you deserved this.

Thanks but I don't think I can get any gifts/rewards etc. as the work is paid by Red Hat and I do that as a RH employee which is perfectly fine for me. I'm going claim the bounty only because bountysource is going to discard it and I'll post it to a charity to avoid any RH code-of-conduct violation.

(In reply to albertogomezmarin from comment #20)

will there be some way to check if firefox is using or not video acceleration?

You need to use the log right now. We can look at it when it's going to be enabled for wider audience (depends on Bug 1595994).

As DMABuf is going to be used under X11 too remove Wayland keyword from the preferences.

Rename:

  • widget.wayland-dmabuf-textures.enabled to widget.dmabuf-textures.enabled
  • widget.wayland-dmabuf-webgl.enabled to widget.dmabuf-webgl.enabled
  • widget.wayland-dmabuf-vaapi.enabled to widget.dmabuf-vaapi.enabled
  • widget.wayland-dmabuf-video-textures.enabled to widget.dmabuf-video-textures.enabled

Also implement widget.dmabuf-use-drm-display preference to use DRM VA-API display
instead of the native one.

widget.wayland-dmabuf-vaapi.enabled to widget.dmabuf-vaapi.enabled

Since you are renaming it anyway, wouldn't it be better to rename it to media.ffmpeg.vaapi.enabled instead? Similarly, to how DXVA-related config options are under media.wmf.dxva.*.

VA-API part is concentrated around FFmpeg-related code. It's not that its configuration options really belong to widget.*

Flags: needinfo?(stransky)
Depends on: 1649120
Attachment #9159724 - Attachment is obsolete: true
Attachment #9159725 - Attachment is obsolete: true
Attachment #9159726 - Attachment is obsolete: true
Attachment #9159727 - Attachment is obsolete: true

(In reply to Rinat from comment #27)

widget.wayland-dmabuf-vaapi.enabled to widget.dmabuf-vaapi.enabled

Since you are renaming it anyway, wouldn't it be better to rename it to media.ffmpeg.vaapi.enabled instead? Similarly, to how DXVA-related config options are under media.wmf.dxva.*.

VA-API part is concentrated around FFmpeg-related code. It's not that its configuration options really belong to widget.*

Okay, implemented as Bug 1649120.

Flags: needinfo?(stransky)
Depends on: 1649675

So this is done now. Run Firefox with EGL on X11 (set MOZ_X11_EGL=1 env variable) and you get VA-API as well as on Wayland.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED

Release Note Request (optional, but appreciated)
[Why is this notable]: New cool feature for Linux
[Affects Firefox for Android]: No
[Suggested wording]: Hardware acceleration for video decoding on Firefox running on Linux/X11 enabled
[Links (documentation, blog post, etc)]:

status-firefox80: --- → fixed
relnote-firefox: --- → ?

I'm pretty sure a release note doesn't make much sense as long as this feature isn't enabled by default but behind an environment variable and an about:config pref.

Above original patch had it, but it wasn't merged: Everything is still gated by IsWaylandDisplay().
https://hg.mozilla.org/mozilla-central/file/tip/gfx/thebes/gfxPlatformGtk.cpp#l735

Flags: needinfo?(stransky)

(In reply to Jan Alexander Steffens [:heftig] from comment #31)

I'm pretty sure a release note doesn't make much sense as long as this feature isn't enabled by default but behind an environment variable and an about:config pref.

Wayland users already need to set an env var. Many users even enable WebRender or OpenGL by env var. Those who want it certainly want to be aware that's possible.

When IsWaylandDisplay() has been removed everyone will run into the following regressions, they are probably the same: bug 1645671, bug 1645673, bug 1643855. DMABuf video textures aren't finished yet: bug 1629788 comment 27. These bugs have 3 weeks to be fixed in 79 beta, otherwise they'll hit release and affect Wayland users who have manually enabled VAAPI.

Ahh, okay, let's remove the IsWaylandDisplay() leftover.

Status: RESOLVED → REOPENED
Flags: needinfo?(stransky)
Resolution: FIXED → ---
Pushed by ccoroiu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2b880e8d9143
[Linux] Remove Wayland check at gfxPlatformGtk::UseHardwareVideoDecoding(), make VA-API decoding possible on X11 r=jhorak

Martin: I noticed there are a few references to Wayland left, see e.g. https://searchfox.org/mozilla-central/source/widget/gtk/DMABufLibWrapper.cpp#203-206

Also, should WebGL with DMABUF work on X11/EGL now, given widget.dmabuf-webgl.enabled is enabled? It doesn't seem so on my local build.

Flags: needinfo?(stransky)

No, WebGL/DMABuf is still disabled on X11. It may need some work to create dmabuf surface from X11 display.

Flags: needinfo?(stransky)
Pushed by ccoroiu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3f4a0bc01840
[Linux] Remove Wayland check at gfxPlatformGtk::UseDMABufVideoTextures(), make VA-API decoding possible on X11 r=jhorak

https://hg.mozilla.org/mozilla-central/rev/2b880e8d9143

Status: REOPENED → RESOLVED
Closed: 4 years ago → 4 years ago
status-firefox80: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla80

(In reply to Martin Stránský [:stransky] (back on Jul 20) from comment #29)

So this is done now. Run Firefox with EGL on X11 (set MOZ_X11_EGL=1 env variable) and you get VA-API as well as on Wayland.

I have run it already, no problems that I detected yet, firefox-nightly(20200703215022). Btw, I have run with an env that I saw in a bug relevant to see the media log, I think I am not using vaapi. How can I check it?

Media log
[Child 5229: MediaPDecoder #3]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::operator(): sample[3021000,3041000] no trimming information
[Child 5229: MediaPDecoder #2]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::HandleDecodedResult: sample[3021000,3041000] (decoded[3014500,3034500] no trimming needed
[Child 5229: MediaPDecoder #2]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::operator(): sample[3041000,3061000] no trimming information
[Child 5229: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::HandleDecodedResult: sample[3041000,3061000] (decoded[3034500,3054500] no trimming needed
[Child 5229: MediaPDecoder #1]: D/PlatformDecoderModule Got one frame output with pts=1460000 dts=1460000 duration=42000 opaque=-9223372036854775808
[Child 5229: MediaPDecoder #1]: D/PlatformDecoderModule Got one frame output with pts=1502000 dts=1502000 duration=41000 opaque=-9223372036854775808
[Child 5229: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::operator(): sample[3061000,3081000] no trimming information
[Child 5229: MediaPDecoder #2]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::HandleDecodedResult: sample[3061000,3081000] (decoded[3054500,3074500] no trimming needed
[Child 5229: MediaPDecoder #2]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::operator(): sample[3081000,3101000] no trimming information
[Child 5229: MediaPDecoder #3]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::HandleDecodedResult: sample[3081000,3101000] (decoded[3074500,3094500] no trimming needed
[Child 5229: MediaPDecoder #3]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::operator(): sample[3101000,3121000] no trimming information
[Child 5229: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::HandleDecodedResult: sample[3101000,3121000] (decoded[3094500,3114500] no trimming needed
[Child 5229: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::operator(): sample[3121000,3141000] no trimming information
[Child 5229: MediaPDecoder #2]: D/PlatformDecoderModule AudioTrimmer[0x7f95bf01fa60] ::HandleDecodedResult: sample[3121000,3141000] (decoded[3114500,3134500] no trimming needed
[Child 5229: MediaPDecoder #2]: D/PlatformDecoderModule Got one frame output with pts=1543000 dts=1543000 duration=42000 opaque=-9223372036854775808
[Parent 4936: MediaTelemetry]: D/PlatformDecoderModule Sandbox decoder rejects requested type
[Parent 4936: MediaTelemetry]: D/PlatformDecoderModule Sandbox decoder rejects requested type

You also need to enable media.ffmpeg.vaapi.enabled in about:config.

(In reply to Ilya K from comment #45)

You also need to enable media.ffmpeg.vaapi.enabled in about:config.

it's already enabled by default I think in nightly. BTW you have all the relevanat info I think in the previous comment that attach the info of about:troubleshooting

(In reply to albertogomezmarin from comment #46)

(In reply to Ilya K from comment #45)

You also need to enable media.ffmpeg.vaapi.enabled in about:config.

it's already enabled by default I think in nightly. BTW you have all the relevanat info I think in the previous comment that attach the info of about:troubleshooting

it WAS, sorry, I don't know hot to edit a comment

definitely, with the latest build of firefox and amdgpu VAAPI is not working with EGL enabled. I have seen in the VCN status report of mine ryzen 7 3700U so... what can be happening?
Vaapi is enabled in about:config webrenderer too etc, you can see my config in my latest attachment.

Regressions: 1650748

What exactly do I need to enable to use it with X11?

I enabled these keys:

media.ffmpeg.dmabuf-textures.enabled
media.ffmpeg.vaapi-drm-display.enabled
media.ffmpeg.vaapi.enabled

WebRender is active for me, and I see this in the log:

Child 18134: MediaController #2]: D/PlatformDecoderModule DMA-Buf/VA-API can't be used, WebRender/DMA-Buf is disabled

(In reply to Shmerl from comment #50)

What exactly do I need to enable to use it with X11?

I enabled these keys:

media.ffmpeg.dmabuf-textures.enabled
media.ffmpeg.vaapi-drm-display.enabled
media.ffmpeg.vaapi.enabled

WebRender is active for me, and I see this in the log:

Child 18134: MediaController #2]: D/PlatformDecoderModule DMA-Buf/VA-API can't be used, WebRender/DMA-Buf is disabled

You need to set MOZ_X11_EGL=1 env variable. But note that VA-API works on Firefox 81 and later unless you use a patched version.

Looks like it's working for me (Firefox 81.0b1) with MOZ_X11_EGL=1

And these config keys set to true

media.ffmpeg.vaapi.enabled
media.ffmpeg.dmabuf-textures.enabled

However it worked only for videos with H.264. For VP9 / Opus (like on Youtube), the log doesn't show VA-API usage. The way I'm testing it:

MOZ_X11_EGL=1 MOZ_LOG="PlatformDecoderModule:5" firefox 2>&1 | grep 'VA-API'

Here is my vainfo:

libva info: VA-API version 1.8.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_8
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.8 (libva 2.8.0)
vainfo: Driver version: Mesa Gallium driver 20.1.5 for AMD Radeon RX 5700 XT (NAVI10, DRM 3.38.0, 5.8.3, LLVM 10.0.1)
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc

Does it mean VAProfileVP9Profile0 and VAProfileVP9Profile2 are not suitable for Youtube usage of VP9?

(In reply to Shmerl from comment #52)

Looks like it's working for me (Firefox 81.0b1) with MOZ_X11_EGL=1

And these config keys set to true

media.ffmpeg.vaapi.enabled
media.ffmpeg.dmabuf-textures.enabled

However it worked only for videos with H.264. For VP9 / Opus (like on Youtube), the log doesn't show VA-API usage. The way I'm testing it:

MOZ_X11_EGL=1 MOZ_LOG="PlatformDecoderModule:5" firefox 2>&1 | grep 'VA-API'

(Bash has the shorthand |&.)

Do you have media.ffvpx.enabled set to false, cf bug 1660336?

[…]

(In reply to Paul Menzel from comment #53)

Do you have media.ffvpx.enabled set to false, cf bug 1660336?

After setting it to false, it worked, thanks!

I think I tried all possible with my Ubuntu FF 80:
startet firefox by
MOZ_X11_EGL=1 MOZ_LOG="PlatformDecoderModule:5" firefox
media.ffvpx.enabled: false
media.ffmpeg.vaapi-drm-display.enabled: true
media.ffmpeg.vaapi.enabled: true
widget.wayland-dmabuf-vaapi.enabled: true

and I added h264ify to the active extensions. My core-m5 gives me still 50% load ....
Anything else I can do or debug? Where can I see which format the video stream is actually?

vainfo gives me a rather short list:
libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 20.1.1 ()
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSliceLP
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSliceLP
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointEncPicture
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
VAProfileVP8Version0_3 : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointVLD

(In reply to star from comment #55)

I think I tried all possible with my Ubuntu FF 80:

(Please create separate bug reports for problems.)

startet firefox by
MOZ_X11_EGL=1 MOZ_LOG="PlatformDecoderModule:5" firefox
media.ffvpx.enabled: false
media.ffmpeg.vaapi-drm-display.enabled: true
media.ffmpeg.vaapi.enabled: true
widget.wayland-dmabuf-vaapi.enabled: true

Please verify, that WebRender is used for compositing in about:support. Otherwise add MOZ_WEBRENDER=1 to your command, or configure it in about:config.

and I added h264ify to the active extensions. My core-m5 gives me still 50% load ....

Please paste the beginning of the log, once you start playing a video in Firefox. 50 % load is still reasonable. sudo intel_gpu_top can give some insight.

Anything else I can do or debug? Where can I see which format the video stream is actually?

In YouTube, right-click in the video frame and choose Stats for nerds.

vainfo gives me a rather short list:

[…]

libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so

[…]

vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSliceLP
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSliceLP
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointEncPicture
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
VAProfileVP8Version0_3 : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointVLD

So, H264 is supported, so the GPU should be used. (You can also verify with mpv --hwdec=vaapi … if VA-API is successfully used.

(In reply to star from comment #55)

I think I tried all possible with my Ubuntu FF 80:
startet firefox by
MOZ_X11_EGL=1 MOZ_LOG="PlatformDecoderModule:5" firefox

  • Required prefs are: media.ffmpeg.vaapi.enabled=true (use hw video decoding), media.ffvpx.enabled=false (use system ffmpeg), gfx.webrender.all=true (use OpenGL rendering)
  • env vars: MOZ_X11_EGL=1 is correct and required for VAAPI. MOZ_LOG="PlatformDecoderModule:5" is of course only needed if you want to check the log.
  • VAAPI is definitely broken in 80 (bug 1656436), technically the first VAAPI on X11 release. It was left broken for Wayland.
    Please try Beta 81 or Nightly 82.
  • https://addons.mozilla.org/firefox/addon/enhanced-h264ify/ only works on YouTube, but disabling VP9 means video resolutions above 1080p are not available.
  • With sudo apt install intel-gpu-tools; sudo intel_gpu_top you can see hardware video decoding workload ("Video").

Oh, setting gfx.webrender.all=true was a really bad idea. My ff gets so slow, I thought it is completely dead. Needed few minutes to revert that ;)

That log thing I didn't get (sorry, newbee). I dont get any information in command line with that (same with MOZ_X11_EGL=1 MOZ_LOG="PlatformDecoderModule:5" firefox 2>&1 )
If it is not working in v80 I have to wait for next one.

Thanks for background info to h264fy.

I know that mpv can work efficiently, load just around 5% watching TV stream. Via web (welt.de) it is around 50%. But not all work as a stream like that, thus efficient browser would be great.

(In reply to star from comment #58)
This sounds unexpected, it shouldn't slow down anything. Please file a bug about this. Also open about:support, click on "copy text to clipboard" and paste it into the attachment field of your bug report. Thanks! OpenGL rendering (WebRender) is required for VAAPI to work.

X11 VAAPI support seems to behind the MOZ_WAYLAND ifdefs and is not available when built in an environment which does not contain gtk+3 and mesa built with wayland support and further does not contain 'wayland' and 'wayland-protocols'. In other words, compiling Firefox on a Xorg only system does not result in the X11 VAAPI code being included the resulting binary. This code doesn't exist in the resulting binary as all the wayland stuff isn't present.

Will this be decoupled from MOZ_WAYLAND in the future? I find it odd that the X11 support requires Wayland.

(In reply to dylan.araps from comment #60)

X11 VAAPI support seems to behind the MOZ_WAYLAND ifdefs and is not available when built in an environment which does not contain gtk+3 and mesa built with wayland support and further does not contain 'wayland' and 'wayland-protocols'. In other words, compiling Firefox on a Xorg only system does not result in the X11 VAAPI code being included the resulting binary. This code doesn't exist in the resulting binary as all the wayland stuff isn't present.

Will this be decoupled from MOZ_WAYLAND in the future? I find it odd that the X11 support requires Wayland.

Did you file a bug for this? I know this will affect at least Gentoo.

Flags: needinfo?(dylan.araps)

Opened a bug report here just now: https://bugzilla.mozilla.org/show_bug.cgi?id=1662496

Thanks.

Flags: needinfo?(dylan.araps)

(In reply to Darkspirit, Servo QA from comment #59)

(In reply to star from comment #58)
This sounds unexpected, it shouldn't slow down anything. Please file a bug about this. Also open about:support, click on "copy text to clipboard" and paste it into the attachment field of your bug report. Thanks! OpenGL rendering (WebRender) is required for VAAPI to work.

It is filed here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1662521

After switching from intel to modeset driver FF doesn't slow down if gfx.webrender.all=true
So I repeated all that settings mentioned above and get this output (see below):

It looks not to bad, but:
CPU seems to have same load as before around 50% playing welt.de live tv with a core-m5.
Even worse, after some second video screen freezes and load keeps high.

Anyhow - there is reason for me to use intel instead of modeset - modeset has heavy tearing. :(
(https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1888810)

:~$ MOZ_X11_EGL=1 MOZ_LOG="PlatformDecoderModule:5" firefox 2>&1 | grep 'VA-API'
[Child 3410: MediaPDecoder #1]: D/PlatformDecoderModule Initialising VA-API FFmpeg decoder
libva info: VA-API version 1.7.0
[Child 3410: MediaPDecoder #1]: D/PlatformDecoderModule VA-API FFmpeg init successful
[Child 3410: MediaPDecoder #1]: D/PlatformDecoderModule Choosing FFmpeg pixel format for VA-API video decoding.
[Child 3410: MediaPDecoder #1]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1599054601792000dts=1599054601592000 duration=1400000 opaque=-9223372036854775808
[Child 3410: MediaPDecoder #2]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1599054603232000dts=1599054603049489 duration=40000 opaque=-9223372036854775808
[Child 3410: MediaPDecoder #2]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1599054603272000dts=1599054603170947 duration=40000 opaque=-9223372036854775808
[Child 3410: MediaPDecoder #2]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1599054603312000dts=1599054603211433 duration=40000 opaque=-9223372036854775808
[Child 3410: MediaPDecoder #3]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1599054603392000dts=-9223372036854775808 duration=40000 opaque=-9223372036854775808
[Child 3410: MediaPDecoder #3]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1599054603472000dts=-9223372036854775808 duration=40000 opaque=-9223372036854775808
[Child 3410: MediaPDecoder #3]: D/PlatformDecoderModule Initialising VA-API FFmpeg decoder
libva info: VA-API version 1.7.0
[Child 3410: MediaPDecoder #3]: D/PlatformDecoderModule VA-API FFmpeg init successful
[Child 3410: MediaPDecoder #1]: D/PlatformDecoderModule Choosing FFmpeg pixel format for VA-API video decoding.
[Child 3410: MediaPDecoder #1]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1599054603152000dts=1599054603072000 duration=40000 opaque=-9223372036854775808
[Child 3410: MediaPDecoder #1]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1599054603232000dts=1599054603152000 duration=40000 opaque=-9223372036854775808
[Child 3410: MediaPDecoder #1]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1599054603272000dts=1599054603272000 duration=40000 opaque=-9223372036854775808
[Child 3410: MediaPDecoder #1]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1599054603312000dts=1599054603312000 duration=40000 opaque=-9223372036854775808

(In reply to star from comment #64)
...

Please do not clutter this bug with individual issues - you already opened a bug for that.

Repeating the test with FF81 gives:

~$ MOZ_X11_EGL=1 MOZ_LOG="PlatformDecoderModule:5" firefox 2>&1 | grep 'VA-API'
[Child 7583: MediaPDecoder #3]: D/PlatformDecoderModule Initialising VA-API FFmpeg decoder
libva info: VA-API version 1.7.0
[Child 7583: MediaPDecoder #3]: D/PlatformDecoderModule VA-API FFmpeg init successful
[Child 7583: MediaPDecoder #1]: D/PlatformDecoderModule Choosing FFmpeg pixel format for VA-API video decoding.
[Child 7583: MediaPDecoder #1]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1601567281664000dts=1601567281464000 duration=120000 opaque=-9223372036854775808
[Child 7583: MediaPDecoder #1]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1601567281824000dts=1601567281625943 duration=40000 opaque=-9223372036854775808
[Child 7583: MediaPDecoder #3]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with
...

Does it mean the hardware does video decoding? The problem with this: The load doesn't decrease, have still the 50% while watching stream: www.welt.de

(In reply to star from comment #66)

Repeating the test with FF81 gives:

~$ MOZ_X11_EGL=1 MOZ_LOG="PlatformDecoderModule:5" firefox 2>&1 | grep 'VA-API'
[Child 7583: MediaPDecoder #3]: D/PlatformDecoderModule Initialising VA-API FFmpeg decoder
libva info: VA-API version 1.7.0
[Child 7583: MediaPDecoder #3]: D/PlatformDecoderModule VA-API FFmpeg init successful
[Child 7583: MediaPDecoder #1]: D/PlatformDecoderModule Choosing FFmpeg pixel format for VA-API video decoding.
[Child 7583: MediaPDecoder #1]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1601567281664000dts=1601567281464000 duration=120000 opaque=-9223372036854775808
[Child 7583: MediaPDecoder #1]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with pts=1601567281824000dts=1601567281625943 duration=40000 opaque=-9223372036854775808
[Child 7583: MediaPDecoder #3]: D/PlatformDecoderModule DMABUF/VA-API Got one frame output with
...

Does it mean the hardware does video decoding? The problem with this: The load doesn't decrease, have still the 50% while watching stream: www.welt.de

Please open a new bug for it and attach a full log here.
Thanks!

Flags: needinfo?(star)

Hasn't got assigned yet, so linking it here: VAAPI doesn't work on Polaris dGPU when Intel IGP is active at the same time:
https://bugzilla.mozilla.org/show_bug.cgi?id=1669331

Depends on: 1739499
Depends on: 1739546
No longer depends on: 1739499
No longer depends on: 1739546
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: