Open Bug 1683808 Opened 9 months ago Updated 3 days ago

Make VAAPI work in the GPU process

Categories

(Core :: Audio/Video: Playback, task, P3)

Unspecified
Linux
task

Tracking

()

People

(Reporter: jya, Unassigned)

References

(Depends on 2 open bugs, Blocks 3 open bugs)

Details

ffvpx VP8/VP9 decoder VAAPI acceleration doesn't work in the RDD process due to the sandbox.

The library is also not loaded and it should. This is a similar issue to bug 1634991.

The PDMFactory is initialised very early in the RDD process; it gfxVars::CanUseHardwareVideoDecoding may still return false at the time it's called which would cause VAAPI loading to not occur there https://searchfox.org/mozilla-central/rev/8698fade12984b9a6a43a85a287a5f17e8fd4ddf/dom/media/platforms/ffmpeg/FFmpegLibWrapper.cpp#262

as IsDMABufVAAPIEnabled() is dependent on the value returned by gfxVars::CanUseHardwareVideoDecoding(). VA-Lib should be loaded unconditionally, and only attempt to initialise the libs when we actually attempt to create a decoder.

Duplicate of this bug: 1683853
Blocks: RDD
Severity: -- → S3
Priority: -- → P3
Duplicate of this bug: 1688627
Duplicate of this bug: 1673184
Assignee: jya-moz → nobody
Duplicate of this bug: 1700439

Is there any workaround? Previously vaapi worked fine on my device (Intel Tigerlake), now RDD process is hogging my cpu, regardless of my settings, which are:

pref("media.ffmpeg.vaapi.enabled", true);
pref("media.ffmpeg.vaapi-drm-display", true);
pref("widget.wayland-dmabuf-vaapi.enabled", true);
pref("media.ffvpx.enabled", false);
pref("media.rdd-ffvpx.enabled", false)
pref("media.av1.enabled", false);
pref("media.rdd-ffmpeg.enabled", true);

I do not see any vaapi related output on terminal. Also disabling rdd via "rdd.process.enabled" makes the video tab instantly crash. Previously I set MOZ_DISABLE_RDD_SANDBOX=1 to make vaapi work, now (Firefox 87) this does not have any effect anymore.

So vaapi is essentially broken, have lost track which version was the last one that worked. Would appreciate if you could suggest me any workarounds until this works with RDD.

Flags: needinfo?(stransky)

(In reply to Esokrarkose from comment #5)
I have managed to get VAAPI work by disabling sandboxing and rdd process:
pref("security.sandbox.content.level", 0);
pref("media.rdd-process.enabled", false);
pref("media.ffmpeg.vaapi.enabled", true);
pref("media.ffvpx.enabled", false);

(In reply to serfreeman1337 from comment #6)

pref("security.sandbox.content.level", 0);
pref("media.rdd-process.enabled", false);

That works for me as well, thanks for the hint.
This is not a sensible workaround unfortunately, since disabling the whole sandbox is too much risk imho. Only disabling the rdd sandbox was ok for me but that no longer works.

So I hope this ticket gets done quickly to be able to really enjoy vaapi without any dangerous workarounds.

This depends on Bug 1698778 right now.

Flags: needinfo?(stransky)
OS: Unspecified → Linux
Summary: Make VAAPI work in the RDD process → Make VAAPI work in the GPU process

Hi. The only way to get hardware decoding in Firefox currently under Wayland is to do this: pref("media.rdd-process.enabled", false);

I have posted this on Reddit. The Arch wiki already advised this, although with a ENV variable which is now replaced with the config setting, it seems. This means the Arch wiki has been advising this workaround for a while.

I don't know what the security implications are, but some of these posts, such as mine, direct here, to this bug, for more information. It looks like this is hard and not fixable soon. Could someone here do a Dr Fauci and explain the risks to ordinary users?

(In reply to Tim Richardson from comment #9)

Hi. The only way to get hardware decoding in Firefox currently under Wayland is to do this: pref("media.rdd-process.enabled", false);

This is not quite correct and I'll clarify it here.

Firefox uses two VAAPI video decoders on Linux - ffvpx and ffmpeg.

  • ffvpx is Firefox build-in decoder (based on ffmpeg project) which can decode VP8 and VP9. It can use VAAPI when media.ffmpeg.vaapi.enabled is set. It's run by default in RDD process where VAAPI is blocked by sandbox. RDD can be disabled by 'media.rdd-ffvpx.enabled' pref value.

  • ffmpeg is external (system wide) decoder and it's supposed to decode H.264, VP8 and VP9 (AV1 is decoded by DAV1DDecoder). It can use VAAPI when media.ffmpeg.vaapi.enabled is set. It's run by default in content process where VAAPI is allowed by sandbox. RDD can be enabled by media.rdd-ffmpeg.enabled pref value which is off by default.

Decoder pickup sequence is ffvpx first and then ffmpeg, so when you open VP8 and VP9 (which is suported by ffvpx), ffvpx is selected. When media.ffmpeg.vaapi.enabled is set we try to open VAAPI but it fails so SW decoder is used.

Short term solution is to disable ffvpx and use ffmpeg only, set 'media.ffvpx.enabled' to false and restart browser.
Long term solution is to move both ffvpx and ffmpeg to GPU process where GPU access is enabled.

As for RDD, it's highly restricted process which is used for third party decoders and should not be generally disabled. When you set 'media.rdd-process.enabled' to false all SW video decoders are running in content process without restrictions.

Wow, thanks.
Does media.rdd-ffmpeg.enabled control whether the ffmpeg decoding happens in the RDD process? Mine is set to true, which sounds undesirable from what you've said. If I set this to false, then it sounds like it would skip the RDD process meaning I should be able to return the RDD process to enabled=true.

However, it does not work. I have not found any way of getting hardware decoding to work except by using pref("media.rdd-process.enabled", false);

I'm on Wayland and I set media.ffmpeg.vaapi.enabled to true, media.ffvpx.enabled to false, media.rdd-ffmpeg.enabled to true. But Firefox is not using VA-API. Version 91.0.2.

(In reply to Jay Chu from comment #12)

I'm on Wayland and I set media.ffmpeg.vaapi.enabled to true, media.ffvpx.enabled to false, media.rdd-ffmpeg.enabled to true. But Firefox is not using VA-API. Version 91.0.2.

Specifically, withenv MOZ_LOG="PlatformDecoderModule:5" firefox, I'm getting [RDD 42953: Main Thread]: D/PlatformDecoderModule VA-API FFmpeg is disabled by platform. I googled about it, and it points to two closed as fixed bugs, one on bugzilla, another on mesa/drm gitlab.

The only conbination worked for me is:

  • media.ffmpeg.vaapi.enabled to true
  • media.ffvpx.enabled to false
  • media.rdd-vpx.enabled to false

To get better log why VAAPI is disabled run Firefox with:

MOZ_LOG="Dmabuf:5, PlatformDecoderModule:5"

env variables. You will get exact info why VAAPI is disabled.

'VA-API FFmpeg is disabled by platform' message points to:
https://searchfox.org/mozilla-central/rev/03b44ad93d39a328215ffa71ea9a70fdee1d363b/dom/media/platforms/ffmpeg/FFmpegLibWrapper.cpp#295

which means nsDMABufDevice::IsDMABufVAAPIEnabled() returns false:
https://searchfox.org/mozilla-central/rev/03b44ad93d39a328215ffa71ea9a70fdee1d363b/widget/gtk/DMABufLibWrapper.cpp#275

Anyway, you're right - looks like something changed and media.ffvpx.enabled / media.rdd-vpx.enabled does not have any effect. VP8/9 is always played by ffvpx with RDD process enabled and only way how to enable VAAPI for VP8/9 is to disable RDD completely.

That's a bit unfortunate, we need to implement the VAAPI support correctly in GPU process.

(In reply to Martin Stránský [:stransky] (ni? me) from comment #10)

Short term solution is to disable ffvpx and use ffmpeg only, set 'media.ffvpx.enabled' to false and restart browser.

Firefox 91.0.2, Ubuntu 21.04 on a Dell XPS 13 9310 (Intel Tigerlake) here. Based on your answer I did the following on a fresh profile:

media.ffmpeg.vaapi.enabled to true
media.av1.enabled to false
media.ffvpx.enabled to false
media.rdd-ffvpx.enabled to false

Still, when playing 4K videos on youtube, tops shows me 500% for RDD process. Find the MOZ_LOG output here: https://pastebin.com/Yrg9xD1y

When I additionally set

media.rdd-vpx.enabled to false

I get the following:

libva info: VA-API version 1.10.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_10
Sandbox: seccomp sandbox violation: pid 21615, tid 21709, syscall 64, args 1140872792 1 1974 0 0 139880884603544.
Sandbox: seccomp sandbox violation: pid 21615, tid 21709, syscall 64, args 1140872792 1 438 0 0 139880884603544.
[Parent 21407, IPC I/O Parent] WARNING: FileDescriptorSet destroyed with unconsumed descriptors: file /build/firefox-DA9Zxg/firefox-91.0.2+build1/ipc/chromium/src/chrome/common/file_descriptor_set_posix.cc:19

Find the full log here: https://pastebin.com/KeyNy322

When setting

media.rdd-process.enabled to false

I'm getting the same seccomp error, see full output here: https://pastebin.com/t8tawh80

So I simply do not get VAAPI to work on my setup :-(.

Flags: needinfo?(stransky)

When additionally setting

security.sandbox.content.level to 0

then VAAPI works fine, see https://pastebin.com/QJFLuUHT

Is this seccomp call exception I am facing a known problem? Setting sandbox.level to 0 does not sound like a good solution, not even short time.

Whitelisting all offending seccomp calls, does not work

security.sandbox.content.syscall_whitelist 64,66,220

The seccomp errors disappear from the output, but the tab crashes still. Only way to make it work here is

security.sandbox.content.level to 0

(In reply to Martin Stránský [:stransky] (ni? me) from comment #15)

To get better log why VAAPI is disabled run Firefox with:

MOZ_LOG="Dmabuf:5, PlatformDecoderModule:5"

Then I get something like this when start up Firefox:

$ env MOZ_LOG="Dmabuf:5,PlatformDecoderModule:5" firefox
[Parent 58871: Main Thread]: D/Dmabuf wl_drm is available.
[Parent 58871: Main Thread]: D/Dmabuf zwp_linux_dmabuf_v1 is available.
[Parent 58871: Main Thread]: D/Dmabuf nsDMABufDevice::Configure()
[Parent 58871: Main Thread]: D/Dmabuf Loading DMABuf system library libgbm.so.1 ...
[Parent 58871: Main Thread]: D/Dmabuf DMABuf is enabled, using drm node /dev/dri/renderD128
[Child 59080: Main Thread]: D/PlatformDecoderModule PDMInitializer, Init PDMs in Content process
[Child 59080: Main Thread]: D/Dmabuf wl_drm is available.
[Child 59080: Main Thread]: D/Dmabuf zwp_linux_dmabuf_v1 is available.
[Child 59080: Main Thread]: D/Dmabuf nsDMABufDevice::Configure()
[Child 59080: Main Thread]: D/Dmabuf Loading DMABuf system library libgbm.so.1 ...
[Child 59080: Main Thread]: D/Dmabuf DMABuf is enabled, using drm node /dev/dri/renderD128
[Child 59080: Main Thread]: D/Dmabuf nsDMABufDevice::IsDMABufVAAPIEnabled: EGL 1 DMABufEnabled 1  media_ffmpeg_vaapi_enabled 1 CanUseHardwareVideoDecoding 1 !XRE_IsRDDProcess 1
[Child 59080: Main Thread]: D/Dmabuf nsDMABufDevice::IsDMABufVAAPIEnabled: EGL 1 DMABufEnabled 1  media_ffmpeg_vaapi_enabled 1 CanUseHardwareVideoDecoding 1 !XRE_IsRDDProcess 1
[Child 59080: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 59080: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[RDD 59198: Main Thread]: D/PlatformDecoderModule PDMInitializer, Init PDMs in RDD process
[RDD 59198: Main Thread]: D/Dmabuf nsDMABufDevice::Configure()
[RDD 59198: Main Thread]: D/Dmabuf Loading DMABuf system library libgbm.so.1 ...
[RDD 59198: Main Thread]: D/Dmabuf DMABuf is enabled, using drm node /dev/dri/renderD128
[RDD 59198: Main Thread]: D/Dmabuf nsDMABufDevice::IsDMABufVAAPIEnabled: EGL 1 DMABufEnabled 1  media_ffmpeg_vaapi_enabled 1 CanUseHardwareVideoDecoding 1 !XRE_IsRDDProcess 0
[RDD 59198: Main Thread]: D/PlatformDecoderModule VA-API FFmpeg is disabled by platform

(In reply to Martin Stránský [:stransky] (ni? me) from comment #15)

Anyway, you're right - looks like something changed and media.ffvpx.enabled / media.rdd-vpx.enabled does not have any effect. VP8/9 is always played by ffvpx with RDD process enabled and only way how to enable VAAPI for VP8/9 is to disable RDD completely.

My situation seems not the same with Esokrarkose's. Using the settings below is fine for me to "get a frame from VA-API ....". And Hardware acceleration is working. The "disabled by platform" message seems not related to this ability, although it's quite confusing to see it.

media.ffmpeg.vaapi.enabled to true
media.ffvpx.enabled to false
media.rdd-vpx.enabled to false

I'm using Arch / GNOME / Wayland / Firefox stable on Wayland / AMD CPU and integrated GPU / OpenSource AMD driver.

I forgot whether don't touch media.rdd-vpx.enabled and turn off media.rdd-ffvpx.enabled will work though. If I recall correctly, it won't.

I don't understand what this option media.rdd-vpx.enabled do, either. This vpx thing is different from ffvpx? I can't easily look up some document on this concept.

(In reply to Esokrarkose from comment #16)

(In reply to Martin Stránský [:stransky] (ni? me) from comment #10)

Short term solution is to disable ffvpx and use ffmpeg only, set 'media.ffvpx.enabled' to false and restart browser.

Firefox 91.0.2, Ubuntu 21.04 on a Dell XPS 13 9310 (Intel Tigerlake) here. Based on your answer I did the following on a fresh profile:

media.ffmpeg.vaapi.enabled to true
media.av1.enabled to false
media.ffvpx.enabled to false
media.rdd-ffvpx.enabled to false

Still, when playing 4K videos on youtube, tops shows me 500% for RDD process. Find the MOZ_LOG output here: https://pastebin.com/Yrg9xD1y

I don't see any ffmpeg/ffvpx logs here, looks like AV1 media playback which is not supported by VAAPI/Firefox yet.

When I additionally set

media.rdd-vpx.enabled to false

I get the following:

libva info: VA-API version 1.10.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_10
Sandbox: seccomp sandbox violation: pid 21615, tid 21709, syscall 64, args 1140872792 1 1974 0 0 139880884603544.
Sandbox: seccomp sandbox violation: pid 21615, tid 21709, syscall 64, args 1140872792 1 438 0 0 139880884603544.
[Parent 21407, IPC I/O Parent] WARNING: FileDescriptorSet destroyed with unconsumed descriptors: file /build/firefox-DA9Zxg/firefox-91.0.2+build1/ipc/chromium/src/chrome/common/file_descriptor_set_posix.cc:19

Find the full log here: https://pastebin.com/KeyNy322

Various Intel drivers can cause additional sandbox violations:
https://bugzilla.mozilla.org/show_bug.cgi?id=1619585

but I don't think this is the case here.

Flags: needinfo?(stransky)

(In reply to Martin Stránský [:stransky] (ni? me) from comment #21)

I don't see any ffmpeg/ffvpx logs here, looks like AV1 media playback which is not supported by VAAPI/Firefox yet.

Please note, that I explicitly disabled AV1 support (" media.av1.enabled to false"), furthermore VAAPI decoding worked for me when disabling rdd and setting sandbox level to 0.
Anyway this is coherent with what Jay Chu is reporting: Without "media.rdd-vpx.enabled to false
" or "media.rdd-process.enabled to false" vaapi decoding is not getting involved.

In contrast to Jay Chu I'm facing an additional issue, that disabling RDD is not enough because I'm getting sandbox violations.

Various Intel drivers can cause additional sandbox violations:
https://bugzilla.mozilla.org/show_bug.cgi?id=1619585

but I don't think this is the case here.

So for this second issue I'm facing, what should I do? File a new bug? What do you need for debugging? I'm facing this issue since around Firefox 85 or 86, I don't remember which release exactly introduced the issue.

(In reply to Martin Stránský [:stransky] (ni? me) from comment #21)

Various Intel drivers can cause additional sandbox violations:
https://bugzilla.mozilla.org/show_bug.cgi?id=1619585

but I don't think this is the case here.

To me it looks like https://bugzilla.mozilla.org/show_bug.cgi?id=1619585#c26 indeed describes to problem. Since I have the new Intel Iris Xe, I can't make use of the older i965 driver, thus I'm forced to the new iHD driver. I added my info to the other bug report, I would be glad if we could continue our discussion there.

Depends on: 1729355
No longer depends on: 1729355

In contrast to Jay Chu I'm facing an additional issue, that disabling RDD is not enough because I'm getting sandbox violations.

I also have Tigerlake. I didn't mention it above, but in my working configuration I also have sandbox level 0 as recommended by Arch wiki, (and RDD disabled).

(In reply to Tim Richardson from comment #24)

In contrast to Jay Chu I'm facing an additional issue, that disabling RDD is not enough because I'm getting sandbox violations.

I also have Tigerlake. I didn't mention it above, but in my working configuration I also have sandbox level 0 as recommended by Arch wiki, (and RDD disabled).

So you are facing https://bugzilla.mozilla.org/show_bug.cgi?id=1619585 as well. Could you edit the Arch wiki, to point out, that disabling the sandbox is only necessary, when using Intel's iHD drivers? Obviously people with older Intel chips supported by the i965 driver have a workaround, but not people with the newest Intel chips.
I think that it is also not a good idea, that the wiki "recommends" disabling the sandbox. I hope that either this issue or the sandbox issue gets resolved soon.

The Arch wiki should unambiguously state that disabling the content process sandbox means disabling the protection against attackers. Don't ever disable the content process sandbox. If iHD (intel-media-driver) causes a crash (sandbox violation), then uninstall it and install i965 (intel-vaapi-driver == i965-va-driver). It's already a risk to use VAAPI in the content process, please don't do the dirtiest workaround.
One needs the sandbox more than one needs VAAPI: If your hardware forces you to use the crashing iHD driver, then better wait until VAAPI has been moved into the GPU process - after a GPU process has been implemented for Wayland.

I'm going to look at the GPU process in FF94 time frame.

(In reply to Darkspirit from comment #26)

The Arch wiki should unambiguously state that disabling the content process sandbox means disabling the protection against attackers. Don't ever disable the content process sandbox.

Fully agreed.

If iHD (intel-media-driver) causes a crash (sandbox violation), then uninstall it and install i965 (intel-vaapi-driver == i965-va-driver).

Please keep in mind, that this workaround is not applicable to Tigerlake (Gen 12) and above graphics. So people with recent Intel hardware are unfortunately out of luck until this bug is fixed.

(In reply to Martin Stránský from comment #27)

I'm going to look at the GPU process in FF94 time frame.

Thanks Martin for your awesome work :-). Right now, Firefox is the only hope for hardware acceleration on Linux, since Chromium broke acceleration with their move to Ozone: https://www.phoronix.com/scan.php?page=news_item&px=Chrome-Ozone-X11-Future

So, what does media.rdd-vpx.enabled do and will disable it be somewhat safer than disable media.rdd-process.enabled?

On Fedora 34 x86_64 with firefox-92.0-2.fc34.x86_64 and Ryzen 4500U laptop I still get D/PlatformDecoderModule VA-API FFmpeg is disabled by platform even with the 3 preferences changed:

media.ffmpeg.vaapi.enabled to true
media.ffvpx.enabled to false
media.rdd-vpx.enabled to false
$ export MOZ_LOG="PlatformDecoderModule:5"
$ firefox-wayland 
[Child 200767: Main Thread]: D/PlatformDecoderModule PDMInitializer, Init PDMs in Content process
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[RDD 201130: Main Thread]: D/PlatformDecoderModule PDMInitializer, Init PDMs in RDD process
[RDD 201130: Main Thread]: D/PlatformDecoderModule VA-API FFmpeg is disabled by platform
[RDD 201130: Main Thread]: D/PlatformDecoderModule Agnostic decoder rejects requested type
[RDD 201130: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[RDD 201130: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[RDD 201130: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[RDD 201130: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[RDD 201130: Main Thread]: D/PlatformDecoderModule Agnostic decoder rejects requested type
[RDD 201130: Main Thread]: D/PlatformDecoderModule Agnostic decoder rejects requested type
[RDD 201130: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[RDD 201130: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[RDD 201130: Main Thread]: D/PlatformDecoderModule Agnostic decoder rejects requested type
[RDD 201130: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: MediaPDecoder #1]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: MediaPDecoder #1]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: MediaPDecoder #1]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: MediaPDecoder #1]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: MediaPDecoder #1]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: MediaPDecoder #2]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: MediaSupervisor #1]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: MediaSupervisor #1]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: MediaSupervisor #2]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[Child 200767: MediaSupervisor #2]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
[RDD 201130: MediaSupervisor #1]: D/PlatformDecoderModule Agnostic decoder supports requested type
[RDD 201130: MediaSupervisor #1]: D/PlatformDecoderModule Agnostic decoder supports requested type
[RDD 201130: MediaSupervisor #1]: D/PlatformDecoderModule Agnostic decoder supports requested type
[RDD 201130: MediaSupervisor #1]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 200767: MediaPDecoder #3]: D/PlatformDecoderModule AudioTrimmer[7f19a7034380] ::PrepareTrimmers: sample[439901000,439921000] no trimming information
[RDD 201130: MediaPDecoder #2]: D/PlatformDecoderModule OpusDataDecoder[7f18ca4471c0] ::Decode: Opus decoder skipping 312 of 960 frames
[Child 200767: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[7f19a7034380] ::HandleDecodedResult: sample[439901000,439921000] (decoded[439901000,439914500] no trimming needed

After switching media.rdd-process.enabled to false vaapi finally works. media.ffvpx.enabled and media.rdd-vpx.enabled can even stay on default true, they appear to have no effect whatsoever. I am not sure what the security implications of such set-up are:

$ export MOZ_LOG="PlatformDecoderModule:5"
$ firefox-wayland 
[Child 217137: Main Thread]: D/PlatformDecoderModule PDMInitializer, Init PDMs in Content process
[Child 217137: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 217137: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 217137: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 217137: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 217137: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 217137: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 217137: Main Thread]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 217137: MediaSupervisor #2]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 217137: MediaSupervisor #2]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 217137: MediaSupervisor #2]: D/PlatformDecoderModule Agnostic decoder supports requested type
[Child 217137: MediaPDecoder #2]: D/PlatformDecoderModule Initialising VA-API FFmpeg decoder
[Child 217137: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::PrepareTrimmers: sample[42901000,42921000] no trimming information
[Child 217137: MediaPDecoder #1]: D/PlatformDecoderModule OpusDataDecoder[7fe8adeee380] ::Decode: Opus decoder skipping 312 of 960 frames
[Child 217137: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::HandleDecodedResult: sample[42901000,42921000] (decoded[42901000,42914500] no trimming needed
libva info: VA-API version 1.11.0
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
[Child 217137: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::PrepareTrimmers: sample[42921000,42941000] no trimming information
[Child 217137: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::HandleDecodedResult: sample[42921000,42941000] (decoded[42914500,42934500] no trimming needed
libva info: Found init function __vaDriverInit_1_11
[Child 217137: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::PrepareTrimmers: sample[42941000,42961000] no trimming information
[Child 217137: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::HandleDecodedResult: sample[42941000,42961000] (decoded[42934500,42954500] no trimming needed
[Child 217137: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::PrepareTrimmers: sample[42961000,42981000] no trimming information
[Child 217137: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::HandleDecodedResult: sample[42961000,42981000] (decoded[42954500,42974500] no trimming needed
[Child 217137: MediaPDecoder #3]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::PrepareTrimmers: sample[42981000,43001000] no trimming information
[Child 217137: MediaPDecoder #3]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::HandleDecodedResult: sample[42981000,43001000] (decoded[42974500,42994500] no trimming needed
[Child 217137: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::PrepareTrimmers: sample[43001000,43021000] no trimming information
[Child 217137: MediaPDecoder #1]: D/PlatformDecoderModule AudioTrimmer[7fe8cb8d3240] ::HandleDecodedResult: sample[43001000,43021000] (decoded[42994500,43014500] no trimming needed
libva info: va_openDriver() returns 0
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x3231564e -> nv12.
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x30313050 -> p010le.
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x36313050 -> unknown.
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x30323449 -> yuv420p.
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x32315659 -> yuv420p.
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x56595559 -> unknown.
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x32595559 -> yuyv422.
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x59565955 -> uyvy422.
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x41524742 -> bgra.
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x41424752 -> rgba.
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x58524742 -> bgr0.
[AVHWDeviceContext @ 0x7fe8adc02fc0] Format 0x58424752 -> rgb0.
[AVHWDeviceContext @ 0x7fe8adc02fc0] VAAPI driver: Mesa Gallium driver 21.1.8 for AMD RENOIR (DRM 3.42.0, 5.14.5-300.s0ix01.fc34.x86_64, LLVM 12.0.1).
[AVHWDeviceContext @ 0x7fe8adc02fc0] Driver not found in known nonstandard list, using standard behaviour.
[Child 217137: MediaPDecoder #2]: D/PlatformDecoderModule VA-API FFmpeg init successful
[Child 217137: MediaPDecoder #3]: D/PlatformDecoderModule Choosing FFmpeg pixel format for VA-API video decoding.
[Child 217137: MediaPDecoder #3]: D/PlatformDecoderModule Requesting pixel format VAAPI_VLD

(In reply to Julian Sikorski from comment #30)
media.ffmpeg.vaapi.enabled=true
both rdd:
media.rdd-ffvpx.enabled=false
media.rdd-vpx.enabled=false

(In reply to Darkspirit from comment #32)

(In reply to Julian Sikorski from comment #30)
media.ffmpeg.vaapi.enabled=true
both rdd:
media.rdd-ffvpx.enabled=false
media.rdd-vpx.enabled=false

Thanks, this works as well. Summing up, four preferences need to be changed in order to get VA-API to work on AMD hardware:
media.ffmpeg.vaapi.enabled to true
media.ffvpx.enabled to false
media.rdd-ffvpx.enabled to false
media.rdd-vpx.enabled to false

You need to log in before you can comment on or make changes to this bug.