Closed Bug 1978911 Opened 1 year ago Closed 1 year ago

[NVIDIA 560.35.03] Instant crash of 141.0 on RHEL 9.4

Categories

(Core :: Widget: Gtk, defect, P3)

Product:
Core
Component:
Widget: Gtk
Version:
Firefox 141
Platform:
x86_64
Linux
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
143 Branch
Tracking Flags:
Tracking Status
firefox-esr115 --- unaffected
firefox-esr128 --- unaffected
firefox-esr140 --- unaffected
firefox141 + fixed
firefox142 + fixed
firefox143 + fixed

People

(Reporter: benjamin.francois, Assigned: stransky, NeedInfo)

References

(Regression)

Details

(5 keywords)

Crash Data

Attachments

(1 file)

Bug 1978911 [Linux] Block dmabuf on NVIDIA < 575.64.5 r?emilio
48 bytes, text/x-phabricator-request
pascalc
: approval-mozilla-beta+
pascalc
: approval-mozilla-release+
Details | Review

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0

Steps to reproduce:

Fresh install of Firefox 141.0 on a RHEL 9.4 workstation, start the browser : instantly crashes and throws the crashreporter window.

Actual results:

Firefox does not start.

Crash report was sent here : https://crash-stats.mozilla.org/report/index/c2a3a956-6db6-4ed4-a41a-a3d100250723

Expected results:

Firefox should have started.

The Bugbug bot thinks this bug should belong to the 'Toolkit::Crash Reporting' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Crash Reporting
Product: Firefox → Toolkit
Component: Crash Reporting → General
OS: Unspecified → Linux
Product: Toolkit → Firefox
Hardware: Unspecified → x86_64

Sorry BugBot I really don't think the problem here is the Crash Reporting. Reverting to Firefox.

Status: UNCONFIRMED → NEW
Crash Signature: [@ NvRmShimImportMemContextFromFd ]
status-firefox141: --- → affected
tracking-firefox141: --- → +
Ever confirmed: true
Keywords: crash, regression, regressionwindow-wanted

Martin, could you have a look please?, Thanks

Flags: needinfo?(stransky)

Benjamin, could you precise if you are using a distro-packaged Firefox (flatpak/rpm) or if you are using a binary downloaded and updated from Mozilla servers? Thanks

Flags: needinfo?(benjamin.francois)

Hello Pascal, it's the x86_64 binary for Linux downloaded straight from the Mozilla Firefox website at this page https://www.firefox.com/en-US/download/all/desktop-release/linux64/en-US/

Flags: needinfo?(benjamin.francois)

Looping in Emilio as well, just in case :)

Flags: needinfo?(emilio)

Can you post the output of env? I suspect there's some misconfiguration going on, it's crashing in libgbm).

Flags: needinfo?(emilio) → needinfo?(benjamin.francois)

We could add a gbm test to glxtest or so but it'd be useful to know what's going on first.

It's dmabuf misconfiguration - we're not supposed to use GBM on NVIDIA. Looks like we didn't detect NVIDIA correctly there and use gbm instead of export from ELGImage.

Component: General → Widget: Gtk
Priority: -- → P3
Product: Firefox → Core
Summary: Instant crash of 141.0 on RHEL 9.4 → [NVIDIA] Instant crash of 141.0 on RHEL 9.4

Benjamin, please attach about:support page.
Thanks.

Flags: needinfo?(stransky)
Flags: needinfo?(stransky)

The bug is marked as tracked for firefox141 (release), tracked for firefox142 (beta) and tracked for firefox143 (nightly). However, the bug still isn't assigned and has low priority.

:jstutte, could you please find an assignee and increase the priority for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Flags: needinfo?(jstutte)

As of comment 10 let's wait for more input first, thanks.

Flags: needinfo?(jstutte)
Severity: -- → S2

(In reply to Martin Stránský [:stransky] (ni? me) from comment #10)

Benjamin, please attach about:support page.

Hello Martin,

Just to make sure : do you want me to revert to 140.0.4 and provide the about:support page from there? Will this be relevant? Because 141.0 crashing on startup won't allow me to open the about:support page.

Thanks!

Just found there is a old bug 1814873 related.

(In reply to Thinker Li [:sinker] from comment #14)

Just found there is a old bug 1814873 related.

Hello! Prior to opening the current bug, I did some googling which did unearth this older one, but I am not sure how related it is: we do use Nvidia GPUs, and the Nvidia driver. If I understand properly what Martin says in 1814873 the problem there was the Nvidia driver being installed but the system using an Intel GPU.

Cheers,
Ben

(In reply to Emilio Cobos Álvarez (:emilio) from comment #7)

Can you post the output of env? I suspect there's some misconfiguration going on, it's crashing in libgbm).

Hi Emilio,
Here you go

$ env
MODULEPATH=/etc/scl/modulefiles:/etc/scl/modulefiles:/etc/scl/modulefiles:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles
MODULESHOME=/usr/share/Modules
SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/3261625,unix/unix:/tmp/.ICE-unix/3261625
SESSIONTYPE=gnome-xorg
MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD
NO_AT_BRIDGE=1
LANG=en_US.UTF-8
CCACHE_DIR=/disk1/tmp/ccache
XDG_CURRENT_DESKTOP=GNOME
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/0eea2094_5f94_4654_a297_042344b6702e
SYSTEMD_EXEC_PID=3261648
SSH_AUTH_SOCK=/run/user/5862/keyring/ssh
LESSOPEN=||/usr/bin/lesspipe.sh %s
QT_IM_MODULE=ibus
__MODULES_SHARE_MANPATH=:1
HOSTNAME=anna17b
OLDPWD=/apps/firefox/test/firefox-141.0
DESKTOP_SESSION=gnome-xorg
HISTSIZE=50000
KDEDIRS=/usr
USER=bfrancoi
XDG_MENU_PREFIX=gnome-
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/5862/bus
MANPATH=/usr/share/man::/opt/puppetlabs/puppet/share/man
SHLVL=1
MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl
XDG_DATA_DIRS=/disney/home/bfrancoi/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
VTE_VERSION=6402
__GL_SHADER_DISK_CACHE_PATH=/disk1/tmp/bfrancoi
GIT_CEILING_DIRECTORIES=/disney/home:/disney/shows:/jobs2:/share:/disney:/data:/show
WDAS_BUILD=el9
LOGNAME=bfrancoi
GNOME_TERMINAL_SERVICE=:1.77
MAIL=/var/spool/mail/bfrancoi
CCACHE_UMASK=002
which_declare=typeset -f
XDG_RUNTIME_DIR=/run/user/5862
XMODIFIERS=@im=ibus
SHELL=/bin/zsh
XDG_SESSION_TYPE=x11
PATH=/disney/home/bfrancoi/.local/bin:/usr/local/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/bin:/sbin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/disney/home/bfrancoi/bin
S_COLORS=auto
HOME=/disney/home/bfrancoi
HISTCONTROL=ignoredups
LOADEDMODULES=
P4CONFIG=.p4config
XAUTHORITY=/disney/home/bfrancoi/.Xauthority
COLORTERM=truecolor
__MODULES_LMINIT=module use --append /usr/share/Modules/modulefiles:module use --append /etc/modulefiles:module use --append /usr/share/modulefiles
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
TERM=xterm
DISPLAY=:100
XDG_SESSION_CLASS=user
PWD=/disney/home/bfrancoi
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:
ZSH=/disney/home/bfrancoi/.oh-my-zsh
PAGER=less
LESS=-R
LSCOLORS=Gxfxcxdxbxegedabagacad
_=/usr/bin/env

Thanks!

Flags: needinfo?(benjamin.francois)

I have a bit of an update!

Our production environment uses Xorg with version 560.35.03 of the Nvidia driver. Unfortunately we are stuck with it for a little while and won't be able to upgrade it any time soon, not until the end of the year at best. Firefox 141.0 instantly crashes with this driver.

I've however been able to update to 575.64.05 on a test machine, and 141.0 launches just fine.

So there is something introduced between Firefox 140.0.4 and Firefox 141.0 that makes it instant crash using the Nvidia driver 560.35.03, but somehow is handled properly by later versions like 575.64.05.

I hope this helps!

Would it be possible to run mozregression to see what exactly broke it?

$ pip install --user mozregression
$ mozregression --good 140 --bad 141
Flags: needinfo?(benjamin.francois)

(In reply to Emilio Cobos Álvarez (:emilio) from comment #18)

Would it be possible to run mozregression to see what exactly broke it?

Oh that is a nice little tool. Here is the result : https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=cd13ab11acd1d8720f9824ae5d2f6bfcfbdf8f5c&tochange=1c5835148de52e3821bc5ad92ef159342fcc1cc8

Cheers,

Flags: needinfo?(benjamin.francois)

Thanks... Martin, does that give you any clue? Might be that this is somehow still invalid but...

QA Whiteboard: [qa-investig-blocked-c143/b142]

The bug is linked to a topcrash signature, which matches the following criterion:

  • Top 5 desktop browser crashes on Linux on release (startup)

For more information, please visit BugBot documentation.

Keywords: topcrash, topcrash-startup

Does widget.dmabuf.enabled flip at about:config help here?

Flags: needinfo?(stransky) → needinfo?(benjamin.francois)

You may try to remove libnvidia-allocator.so.1 - it's the same issue as Bug 1814873.

Benjamin, I wonder why do you use stock Mozilla binaries and not RHEL ones? RHEL is going to ship 142.0 ESR and may carry downstream patch for it if it's needed.

So the solution here is:

  1. Set widget.dmabuf.enabled to false at about:config
  2. Remove libnvidia-allocator.so.1 (Bug 1814873)
  3. Use RHEL Firefox packages which should handle it (if not please report accordingly at RH)

Hello Martin, thank you for your response,

  1. Wouldn't this have an impact on performance, especially WebGL? Also I am not sure there is a way we'd be able to do that without reaching into our 1000+ users $HOME directories and Firefox profiles, which is not that simple to do at scale.
  2. This does indeed seem to work and allows Firefox, however I am wary of side effects of straight up removing files provided by the Nvidia driver. Could this break something else?
  3. I'll discuss this with our senior engineers, there probably was a reason why we dropped the Firefox RHEL RPM in the past in favor of upstream downloaded through autopkg. If that reason is no more, I have no qualms using the ESR release from Red Hat.

Thanks for the help,
Cheers,

Flags: needinfo?(benjamin.francois)

To add more context to 2.

Simply removing the symlink libnvidia-allocator.so.1 from /usr/lib and /usr/lib64 is not enough. I need to fully delete all libnvidia-allocator.so* from both locations for Firefox 141.0 to start. Which feels risky to me, since I am not sure what this involves.

Cheers,

(In reply to Benjamin FRANCOIS from comment #26)

Hello Martin, thank you for your response,

  1. Wouldn't this have an impact on performance, especially WebGL? Also I am not sure there is a way we'd be able to do that without reaching into our 1000+ users $HOME directories and Firefox profiles, which is not that simple to do at scale.

Yes, widget.dmabuf.enabled disables WebGL dmabuf backend and leads to ~20% performance penalty.
But I don't know if dmabuf is used on our setup with NVIDIA anyway - it's supposed to work with new drivers and was disabled for old ones.

I think what you see now it's just moved dmabuf check so you hit it early before it's disabled due to GBM failure.
You may try to compare WebGL performance with widget.dmabuf.enabled=false at 141.0 and 140.0 without it (I guess it will be the same).

  1. This does indeed seem to work and allows Firefox, however I am wary of side effects of straight up removing files provided by the Nvidia driver. Could this break something else?

Frankly I have no idea. There must be a reason the file is there but I don't use NVIDIA drivers and also I'm not in touch with NVIDIA gfx team.

  1. I'll discuss this with our senior engineers, there probably was a reason why we dropped the Firefox RHEL RPM in the past in favor of upstream downloaded through autopkg. If that reason is no more, I have no qualms using the ESR release from Red Hat.

Red Hat is going to ship ESR 140.0 as an update and also talks to NVIDIA folks so they should be able to solve this issue for you. I don't think Mozilla as upstream should add extra patches for old and known broken NVIDIA drivers.

Does this issue only happen on RHEL 9.4? According to the crash reports, we can see other distro as well.

Solution 1 doesn't work since firefox is not able to reach about:config. Users have to change prefs.js manually.

We probably want to detect this case and override the value of widget.dmabuf.enabled if possible since this issue is already on the top crash list.
When people run into this issue, they probably just turn away to use other browsers.

(In reply to Thinker Li [:sinker] from comment #29)

Does this issue only happen on RHEL 9.4? According to the crash reports, we can see other distro as well.

Solution 1 doesn't work since firefox is not able to reach about:config. Users have to change prefs.js manually.

We probably want to detect this case and override the value of widget.dmabuf.enabled if possible since this issue is already on the top crash list.
When people run into this issue, they probably just turn away to use other browsers.

In that case you'd need to add nsIGfxInfo::FEATURE_DMABUF block for appropriate NVIDIA driver version here:
https://searchfox.org/mozilla-central/rev/f2da682709d6dcdb1e5fd3573bc269e22db7e11e/widget/gtk/GfxInfo.cpp#1048
(and perhaps uplift that to beta then).

Flags: needinfo?(thinker.li)

Can we backout the regressor, bug 1959645, safely on the release branch to fix this issue in the upcoming 141 dot release?

Flags: needinfo?(stransky)

(In reply to Pascal Chevrel:pascalc from comment #31)

Can we backout the regressor, bug 1959645, safely on the release branch to fix this issue in the upcoming 141 dot release?

No, that will break Bug 1861759 and makes dmabuf tread unsafe. Better to add blocking to drivers. Also we don't know yet what's going on actually.

Flags: needinfo?(stransky)

Benjamin, please run Firefox 140.0 (the working version) on terminal with MOZ_LOG="Dmabuf:5" env variable, run any WebGL and attach the log here.
Thanks.

Flags: needinfo?(benjamin.francois)

This is a reminder regarding comment #11!

The bug is marked as tracked for firefox141 (release), tracked for firefox142 (beta) and tracked for firefox143 (nightly). We have limited time to fix this, the soft freeze is in 14 days. However, the bug still isn't assigned and has low priority.

(In reply to Martin Stránský [:stransky] (ni? me) from comment #33)

Benjamin, please run Firefox 140.0 (the working version) on terminal with MOZ_LOG="Dmabuf:5" env variable, run any WebGL and attach the log here.

Sure thing. Here's the output I get redirecting stderr into a log file when going to webglsamples.org with that env variable.

[Parent 632664: Main Thread]: D/Dmabuf DMABufDevice::Configure()
[Parent 632664: Main Thread]: D/Dmabuf Loading DMABuf system library libgbm.so.1 ...
[Parent 632664: Main Thread]: D/Dmabuf Using DRM device /dev/dri/renderD128
[Parent 632664: Main Thread]: D/Dmabuf DMABuf is enabled
[Parent 632664: Main Thread]: D/Dmabuf DMABufFormats::EnsureBasicFormats(): GBM_FORMAT_XRGB8888 is missing, adding.
[Parent 632664: Main Thread]: D/Dmabuf DMABufFormats::EnsureBasicFormats(): GBM_FORMAT_ARGB8888 is missing, adding.
[Parent 632664: Main Thread]: D/Dmabuf DMABufDevice::Configure()
[Parent 632664: Main Thread]: D/Dmabuf Loading DMABuf system library libgbm.so.1 ...
[Parent 632664: Main Thread]: D/Dmabuf Using DRM device /dev/dri/renderD128
[Parent 632664: Main Thread]: D/Dmabuf DMABuf is enabled
[Parent 632664: Main Thread]: D/Dmabuf DMABufFormats::EnsureBasicFormats(): GBM_FORMAT_XRGB8888 is missing, adding.
[Parent 632664: Main Thread]: D/Dmabuf DMABufFormats::EnsureBasicFormats(): GBM_FORMAT_ARGB8888 is missing, adding.
[ERROR error_support::handling] suggest-unexpected: Error executing SQL: interrupted (context: <none>)
[ERROR error_support::handling] suggest-unexpected: Error executing SQL: interrupted (context: <none>)
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
<3000 times this same line>
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 633158: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0

Cheers,

Flags: needinfo?(benjamin.francois)

Thanks. So according the log the dmabuf feature is disabled anyway and we just need to add blocklist to avoid test it for broken NVIDIA driver version. widget.dmabuf.enabled will also do the trick.

Assignee: nobody → stransky
Status: NEW → ASSIGNED

Benjamin,
please attach about:support page from Firefox running on 575.64.05 drivers and also MOZ_LOG="Dmabuf:5" log from it.
There should be dmabuf_export feature used and you should see ~20% performance gain compared to 560.35.03.
Thanks.

Flags: needinfo?(benjamin.francois)
Summary: [NVIDIA] Instant crash of 141.0 on RHEL 9.4 → [NVIDIA 560.35.03] Instant crash of 141.0 on RHEL 9.4

We're going to want Beta & Release uplift requests on this pretty quickly too if this is going to make it into the upcoming 141.0.2 dot release.

Flags: needinfo?(stransky)

(In reply to Martin Stránský [:stransky] (ni? me) from comment #38)

Benjamin,
please attach about:support page from Firefox running on 575.64.05 drivers and also MOZ_LOG="Dmabuf:5" log from it.
There should be dmabuf_export feature used and you should see ~20% performance gain compared to 560.35.03.
Thanks.

Hello Martin,

I wasn't sure whether you wanted this from Firefox 140 or 141 so I did both.

Here's the output with Nvidia drivers 575.64.05 and Firefox 140.0.4

[Parent 8555: Main Thread]: D/Dmabuf DMABufDevice::Configure()
[Parent 8555: Main Thread]: D/Dmabuf Loading DMABuf system library libgbm.so.1 ...
[Parent 8555: Main Thread]: D/Dmabuf Using DRM device /dev/dri/renderD128
[Parent 8555: Main Thread]: D/Dmabuf DMABuf is enabled
[Parent 8555: Main Thread]: D/Dmabuf DMABufFormats::EnsureBasicFormats(): GBM_FORMAT_XRGB8888 is missing, adding.
[Parent 8555: Main Thread]: D/Dmabuf DMABufFormats::EnsureBasicFormats(): GBM_FORMAT_ARGB8888 is missing, adding.
[ERROR error_support::handling] suggest-unexpected: Error executing SQL: interrupted (context: <none>)
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 8795: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0

And here's the output with Nvidia drivers 575.64.05 and Firefox 140.0.4

[Parent 11079: Main Thread]: D/Dmabuf DMABufDeviceLock::DMABufDeviceLock() [7fff71b2d440]
[Parent 11079: Main Thread]: D/Dmabuf DMABufDevice::Init()
[Parent 11079: Main Thread]: D/Dmabuf Loading DMABuf system library libgbm.so.1 ...
[Parent 11079: Main Thread]: D/Dmabuf Using DRM device /dev/dri/renderD128
[Parent 11079: Main Thread]: D/Dmabuf DMABuf is enabled
[Parent 11079: Main Thread]: D/Dmabuf EnsureDMABufDevice(): created DMABufDevice
[Parent 11079: Main Thread]: D/Dmabuf DMABufDevice::GetDevice() [7fcc1e9f65e0]
[Parent 11079: Main Thread]: D/Dmabuf DMABufFormats::EnsureBasicFormats(): GBM_FORMAT_XRGB8888 is missing, adding.
[Parent 11079: Main Thread]: D/Dmabuf DMABufFormats::EnsureBasicFormats(): GBM_FORMAT_ARGB8888 is missing, adding.
[Parent 11079: Main Thread]: D/Dmabuf DMABufDeviceLock::~DMABufDeviceLock() [7fff71b2d440]
[Child 11334: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 11334: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 11334: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 11334: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 11334: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 11334: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 11334: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 11334: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 11334: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 11334: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0
[Child 11334: Main Thread]: D/Dmabuf DMABufDevice::IsDMABufWebGLEnabled: UseDMABuf 1 sUseWebGLDmabufBackend 1 UseDMABufWebGL 0

Thank you,
Cheers

Flags: needinfo?(benjamin.francois)

Correction: the second log is with Firefox 141.0

Looks like dmabuf is not used even with 141.0 & new drivers, not sure why. Will check it.
Meanwhile we may disable dmabuf on old NVIDIA as it doesn't seem to be enabled anyway.

https://hg.mozilla.org/mozilla-central/rev/5bf4e63df623

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
status-firefox143: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → 143 Branch

Please add uplift requests so as that we can include it on tomorrow's beta and Tuesday's dot release, thanks.

Comment on attachment 9504449 [details]
Bug 1978911 [Linux] Block dmabuf on NVIDIA < 575.64.5 r?emilio

Beta/Release Uplift Approval Request

  • User impact if declined/Reason for urgency: Crash on startup with outdated NVIDIA drivers.
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Driver blocklist, dmabuf is disabled anyway on NVIDIA.
  • String changes made/needed:
  • Is Android affected?: Yes
Flags: needinfo?(stransky)
Attachment #9504449 - Flags: approval-mozilla-release?
Attachment #9504449 - Flags: approval-mozilla-beta?

Benjamin,
can you try latest nightly with the old NVIDIA drivers? It should not crash any more:
https://fedoraproject.org/wiki/How_to_debug_Firefox_problems#Testing_Mozilla_Nightly_binaries
Thanks.

Flags: needinfo?(benjamin.francois)
Attachment #9504449 - Flags: approval-mozilla-release?
Attachment #9504449 - Flags: approval-mozilla-release+
Attachment #9504449 - Flags: approval-mozilla-beta?
Attachment #9504449 - Flags: approval-mozilla-beta+

(In reply to Martin Stránský [:stransky] (ni? me) from comment #47)

Benjamin,
can you try latest nightly with the old NVIDIA drivers? It should not crash any more:
https://fedoraproject.org/wiki/How_to_debug_Firefox_problems#Testing_Mozilla_Nightly_binaries
Thanks.

Hello Martin!

I confirm the latest nightly does not crash with the old drivers. Actually writing this comment using it.

Thank y'all for this prompt fix! Much appreciated.

Cheers,

Flags: needinfo?(benjamin.francois)

Happy to report that 141.0.2 also does not crash.
As far as we're concerned this issue is no more.

Thanks again to everybody who chimed in.

See Also: → 1981326
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: