1828323 - NativeLayerWayland: widget.dmabuf.force-enabled=true (for vaapi on nvidia) will make firefox-nightly crash

Status: RESOLVED WORKSFORME

(Core :: Graphics, defect)

Description

[vitalyankh]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/114.0

Steps to reproduce:

To use VA-API with Firefox on Linux, Gnome Wayland on Nvidia RTX3060 Mobile, I followed the instructions here: https://github.com/elFarto/nvidia-vaapi-driver/#firefox. Everything is OK on Firefox developer edition (version 113). It makes use of nvidia-vaapi-driver and uses GPU to codec video. But Firefox nightly crashes with the same configurations. If I unset widget.dmabuf.force-enabled on Firefox nightly, it won't crash but it can't make use VA-API.

Actual results:

Firefox Nightly crashes when launching, the command line error is:

[GFX1-]: MozFramebuffer::CreateImpl(size:Size(1024,512), samples:0, depthAndStencil:true, colorTarget:0xde1, colorName:18): Incomplete: 0x0x8cd6
ExceptionHandler::GenerateDump cloned child 544634
ExceptionHandler::SendContinueSignalToChild sent continue signal to child
ExceptionHandler::WaitForContinueSignal waiting for continue signal...
Exiting due to channel error.
Failed to open curl lib from binary, use libcurl.so instead

The "Failed to open curl lib from binary, use libcurl.so instead" seems unrelated to this dmabuf problem because if I unset "widget.dmabuf.force-enabled", Firefox Nightly won't crash.
The crash report is here:
https://crash-stats.mozilla.org/report/index/dbc74f99-f8f4-48e1-93b6-ffa120230416

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Graphics' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Darkspirit]

Thanks for the report! There are two aspects:

https://crash-stats.mozilla.org/report/index/dbc74f99-f8f4-48e1-93b6-ffa120230416

Please set gfx.webrender.compositor and gfx.webrender.compositor.force-enabled back to false.
On Linux, webrender_compositor has been a prototype (to move Wayland surfaces around when scrolling instead of moving tiles inside an OpenGL window) that is unfinished, has bugs and is not recommended.
You should be using regular WebRender.

---

(In reply to vitalyankh from comment #0)

I followed the instructions here: https://github.com/elFarto/nvidia-vaapi-driver/#firefox.
If I unset widget.dmabuf.force-enabled on Firefox nightly, it won't crash but it can't make use VA-API.

bug 1788573 comment 26: Please make sure you also enable the Nvidia suspend service and NVreg_PreserveVideoMemoryAllocations=1 to avoid bug 1788573 which is a suspend&resume use-after-free bug.
Dmabuf will be re-enabled once a fixed Nvidia driver version has been released.

(Darkspirit from https://github.com/elFarto/nvidia-vaapi-driver/issues/182#issuecomment-1466171392)

So yes, for the moment widget.dmabuf.force-enabled=true is needed to override this. I'll update the README.

a) Please don't because users would run into a confirmed use-after-free Nvidia EGL driver bug: bug 1788573 comment 26
b) Please do it only if you explicitly state that the Nvidia suspend service must be enabled and NVreg_PreserveVideoMemoryAllocations=1 must be set:
https://askubuntu.com/a/1309807
https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Preserve_video_memory_after_suspend
https://download.nvidia.com/XFree86/Linux-x86_64/510.39.01/README/powermanagement.html

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The bug has a crash signature, thus the bug will be considered confirmed.

Comment 4

[vitalyankh]

(In reply to Darkspirit from comment #2)

Please set gfx.webrender.compositor and gfx.webrender.compositor.force-enabled back to false.
On Linux, webrender_compositor has been a prototype (to move Wayland surfaces around when scrolling instead of moving tiles inside an OpenGL window) that is unfinished, has bugs and is not recommended.
You should be using regular WebRender.

It is really great that you pointed out the cause of the crash just after I submit the crash report!

On firefox-nightly,

user_pref("gfx.webrender.all", true);
user_pref("gfx.webrender.compositor", true);
// no user_pref("gfx.webrender.compositor.force-enabled", true);
user_pref("widget.dmabuf.force-enabled", true);
user_pref("gfx.x11-egl.force-enabled", true);

won't crash and make use of the Nvidia VAAPI driver.

But on firefox-developer-edtion:

user_pref("gfx.webrender.all", true);
user_pref("gfx.webrender.compositor", true);
user_pref("gfx.webrender.compositor.force-enabled", true);
user_pref("widget.dmabuf.force-enabled", true);
user_pref("gfx.x11-egl.force-enabled", true);

can run normally and make use of the Nvidia VAAPI driver with user_pref("gfx.webrender.compositor.force-enabled", true);. Maybe the firefox-nightly is built with newer webrender compositor code and it's breaking something? Are there any difference between firefox-developer-edtion and firefox-nightly's webrender compositor?

Looking forward to the webrender compositor to be more complete on Linux in the future!

Comment 5

[vitalyankh]

I could confirm this problem is gone with the latest Firefox nightly (Built from https://hg.mozilla.org/mozilla-central/rev/07613976379f96a9416021b138ba21870fe2065d, the official Firefox nightly build). Should this bug be closed?