1826583 - [wayland-1.22] Crash in [@ libwayland-client.so.0@0x6c9d]: "warning: queue 0x7f0821b867c0 destroyed while proxies still attached:"

Status: RESOLVED FIXED

(Core :: Widget: Gtk, defect, P2)

Description

[Matt Fagnani]

I updated a Fedora 38 KDE Plasma installation with sudo dnf offline-upgrade download and sudo dnf offline-upgrade reboot The update included wayland-1.22.0-1.fc38 https://bodhi.fedoraproject.org/updates/FEDORA-2023-3e957cd827

After the update, I started Firefox 113.0a1 20230405093623 on Wayland in Plasma 5.27.3 and closed it. Firefox crashed in the Renderer thread 3/3 times after I closed it. The crash involved libwayland-client.so.0 and had the reason warning: queue 0x7f0821b867c0 destroyed while proxies still attached. This type of crash happened when closing Firefox 111.0.1 on Wayland. Thunderbird 113.0a1 20230405103412 on Wayland also crashed each time I closed it with the same sort of trace. https://crash-stats.mozilla.org/report/index/186b31c5-5b58-4ee6-9f96-9b8850230405

Crash report: https://crash-stats.mozilla.org/report/index/317024a0-da78-46a1-b510-d7c880230405

MOZ_CRASH Reason: ```warning: queue 0x7f0821b867c0 destroyed while proxies still attached:

Top 10 frames of crashing thread:

0 libxul.so MOZ_Crash mfbt/Assertions.h:261
0 libxul.so mozilla::widget::WlCrashHandler widget/gtk/nsWaylandDisplay.cpp:294
1 libwayland-client.so.0 libwayland-client.so.0@0x6c9d
2 libwayland-client.so.0 libwayland-client.so.0@0x8853
3 libwayland-client.so.0 libwayland-client.so.0@0x8928
4 libEGL_mesa.so.0 libEGL_mesa.so.0@0x2ad83
5 libEGL_mesa.so.0 libEGL_mesa.so.0@0x1fe57
6 libEGL_mesa.so.0 libEGL_mesa.so.0@0x2037f
7 libEGL_mesa.so.0 libEGL_mesa.so.0@0xdabb
8 libxul.so mozilla::gl::GLLibraryEGL::fTerminate const gfx/gl/GLLibraryEGL.h:254

Comment 1

[Darkspirit]

Other signatures with "destroyed while proxies still attached".
Seems to occur since 110.0.1.
Not reproducible for me on Debian Testing.

Comment 2

[Darkspirit]

Some reports have an Uptime of 1s.

Comment 4

[Matt Fagnani]

Since the libwayland-client and mesa functions didn't seem to show up in the trace, I ran Firefox 113.0a1 20230405093623 ASan build with -P and selected a profile with the libwayland-client and mesa-libEGL debuginfo packages installed. The ASan trace of the crash showed the libwayland-client and mesa functions and lines. The crash looks like it involves freeing a Wayland queue.

MOZ_ENABLE_WAYLAND=1 /programs/firefox-asan/firefox -P &
[1] 7424
Crash Annotation GraphicsCriticalError: |[0][GFX1-]: glxtest: VA-API test failed: no supported VAAPI profile found. (t=0.544492) [GFX1-]: glxtest: VA-API test failed: no supported VAAPI profile found.
AddressSanitizer:DEADLYSIGNAL

==7480==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7ff50aa014c1 bp 0x7ffcc0d85150 sp 0x7ffcc0d84d40 T0)
==7480==The signal is caused by a WRITE memory access.
==7480==Hint: address points to the zero page.
#0 0x7ff50aa014c1 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:287:7
#1 0x55abce008e68 in content_process_main /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:57:28
#2 0x55abce008e68 in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:353:18
#3 0x7ff514963b49 in __libc_start_call_main /usr/src/debug/glibc-2.37-1.fc38.x86_64/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#4 0x7ff514963c0a in __libc_start_main@GLIBC_2.2.5 /usr/src/debug/glibc-2.37-1.fc38.x86_64/csu/../csu/libc-start.c:360:3
#5 0x55abcdf46dd8 in _start (/programs/firefox-asan/firefox-bin+0xacdd8) (BuildId: 3b8a3ab238f2b97d6daef662b0b135d20bfca7b5)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:287:7 in XRE_InitChildProcess(int, char**, XREChildData const*)
==7480==ABORTING
[Parent 7424, IPC I/O Parent] WARNING: process 7480 exited with status 1: file /builds/worker/checkouts/gecko/ipc/chromium/src/base/process_util_posix.cc:270
AddressSanitizer:DEADLYSIGNAL

==7424==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f59bd68ed1f bp 0x7f59864aaf60 sp 0x7f59864aaf60 T21)
==7424==The signal is caused by a WRITE memory access.
==7424==Hint: address points to the zero page.
#0 0x7f59bd68ed1f in MOZ_Crash /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:261:3
#1 0x7f59bd68ed1f in mozilla::widget::WlCrashHandler(char const*, __va_list_tag*) /builds/worker/checkouts/gecko/widget/gtk/nsWaylandDisplay.cpp:294:3
#2 0x7f59c9bd2c9d /usr/src/debug/wayland-1.22.0-1.fc38.x86_64/redhat-linux-build/../src/wayland-util.c:449:2
#3 0x7f59c9bd4853 in wl_event_queue_release /usr/src/debug/wayland-1.22.0-1.fc38.x86_64/redhat-linux-build/../src/wayland-client.c:308:4
#4 0x7f59c9bd4928 in wl_event_queue_destroy /usr/src/debug/wayland-1.22.0-1.fc38.x86_64/redhat-linux-build/../src/wayland-client.c:352:2
#5 0x7f5984236d83 in dri2_teardown_wayland /usr/src/debug/mesa-23.0.1-2.fc38.x86_64/redhat-linux-build/../src/egl/drivers/dri2/platform_wayland.c:2802:7
#6 0x7f598422be57 in dri2_display_destroy /usr/src/debug/mesa-23.0.1-2.fc38.x86_64/redhat-linux-build/../src/egl/drivers/dri2/egl_dri2.c:1153:7
#7 0x7f598422c37f in dri2_display_release /usr/src/debug/mesa-23.0.1-2.fc38.x86_64/redhat-linux-build/../src/egl/drivers/dri2/egl_dri2.c:1115:4
#8 0x7f598422c37f in dri2_display_release /usr/src/debug/mesa-23.0.1-2.fc38.x86_64/redhat-linux-build/../src/egl/drivers/dri2/egl_dri2.c:1100:1
#9 0x7f598422c37f in dri2_terminate /usr/src/debug/mesa-23.0.1-2.fc38.x86_64/redhat-linux-build/../src/egl/drivers/dri2/egl_dri2.c:1219:4
#10 0x7f5984219abb in eglTerminate /usr/src/debug/mesa-23.0.1-2.fc38.x86_64/redhat-linux-build/../src/egl/main/eglapi.c:769:7
#11 0x7f59b53d25a6 in fTerminate /builds/worker/checkouts/gecko/gfx/gl/GLLibraryEGL.h:254:53
#12 0x7f59b53d25a6 in fTerminate /builds/worker/checkouts/gecko/gfx/gl/GLLibraryEGL.h:742:42
#13 0x7f59b53d25a6 in mozilla::gl::EglDisplay::~EglDisplay() /builds/worker/checkouts/gecko/gfx/gl/GLLibraryEGL.cpp:825:3
#14 0x7f59b54574e4 in destroy<mozilla::gl::EglDisplay> /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/ext/new_allocator.h:140:28
#15 0x7f59b54574e4 in destroy<mozilla::gl::EglDisplay> /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/alloc_traits.h:487:8
#16 0x7f59b54574e4 in operator() /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/shared_ptr_base.h:1310:6
#17 0x7f59b54574e4 in std::_Sp_counted_deleter<mozilla::gl::EglDisplay*, std::__shared_ptr<mozilla::gl::EglDisplay, (__gnu_cxx::_Lock_policy)2>::_Deleter<std::allocator<mozilla::gl::EglDisplay>>, std::allocator<mozilla::gl::EglDisplay>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/shared_ptr_base.h:470:9
#18 0x7f59b53c47a0 in _M_release /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/shared_ptr_base.h:154:6
#19 0x7f59b53c47a0 in ~__shared_count /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/shared_ptr_base.h:684:11
#20 0x7f59b53c47a0 in ~__shared_ptr /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/shared_ptr_base.h:1123:31
#21 0x7f59b53c47a0 in mozilla::gl::GLContextEGL::~GLContextEGL() /builds/worker/checkouts/gecko/gfx/gl/GLContextProviderEGL.cpp:409:1
#22 0x7f59b53c4a7d in mozilla::gl::GLContextEGL::~GLContextEGL() /builds/worker/checkouts/gecko/gfx/gl/GLContextProviderEGL.cpp:392:31
#23 0x7f59b5d8e6ab in mozilla::wr::RenderThread::ShutDownTask() /builds/worker/checkouts/gecko/gfx/webrender_bindings/RenderThread.cpp:226:3
#24 0x7f59b460768c in mozilla::detail::runnable_args_base<(mozilla::detail::RunnableResult)0>::Run() /builds/worker/checkouts/gecko/dom/media/webrtc/transport/runnable_utils.h:41:5
#25 0x7f59b2e0d8a0 in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1233:16
#26 0x7f59b2e1a462 in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:479:10
#27 0x7f59b47f2ec3 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:330:5
#28 0x7f59b4640390 in RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:369:10
#29 0x7f59b4640390 in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:362:3
#30 0x7f59b4640390 in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:344:3
#31 0x7f59b2e03baf in nsThread::ThreadFunc(void*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:391:10
#32 0x7f59cc4ee990 in _pt_root /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:201:5
#33 0x7f59ccca8c56 in start_thread /usr/src/debug/glibc-2.37-1.fc38.x86_64/nptl/pthread_create.c:444:8
#34 0x7f59ccd2ea6f in __GI___clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:261:3 in MOZ_Crash
Thread T21 created by T0 here:
#0 0x5591bd2a069c in pthread_create /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:208:3
#1 0x7f59cc4dc83e in _PR_CreateThread /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:458:14
#2 0x7f59cc4c6eae in PR_CreateThread /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:533:12
#3 0x7f59b2e071db in nsThread::Init(nsTSubstring<char> const&) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:633:18
#4 0x7f59b2e1808d in nsThreadManager::NewNamedThread(nsTSubstring<char> const&, nsIThreadManager::ThreadCreationOptions, nsIThread**) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadManager.cpp:548:12
#5 0x7f59b2e257b4 in NS_NewNamedThread(nsTSubstring<char> const&, nsIThread**, already_AddRefed<nsIRunnable>, nsIThreadManager::ThreadCreationOptions) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:175:57
#6 0x7f59b5d8cc1d in NS_NewNamedThread<9UL> /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:76:10
#7 0x7f59b5d8cc1d in mozilla::wr::RenderThread::Start(unsigned int) /builds/worker/checkouts/gecko/gfx/webrender_bindings/RenderThread.cpp:125:17
#8 0x7f59b5a3cddf in gfxPlatform::InitLayersIPC() /builds/worker/checkouts/gecko/gfx/thebes/gfxPlatform.cpp:1317:7
#9 0x7f59b5a37c34 in gfxPlatform::Init() /builds/worker/checkouts/gecko/gfx/thebes/gfxPlatform.cpp:974:3
#10 0x7f59b5a3573b in gfxPlatform::GetPlatform() /builds/worker/checkouts/gecko/gfx/thebes/gfxPlatform.cpp:464:5
#11 0x7f59bd53814d in nsWindow::Create(nsIWidget*, void*, mozilla::gfx::IntRectTyped<mozilla::LayoutDevicePixel> const&, mozilla::widget::InitData*) /builds/worker/checkouts/gecko/widget/gtk/nsWindow.cpp:5923:13
#12 0x7f59bd34766b in nsIWidget::Create(nsIWidget*, void*, mozilla::gfx::IntRectTyped<mozilla::DesktopPixel> const&, mozilla::widget::InitData*) /builds/worker/checkouts/gecko/widget/nsIWidget.h:463:12
#13 0x7f59c1ad5eb5 in mozilla::AppWindow::Initialize(nsIAppWindow*, nsIAppWindow*, int, int, bool, mozilla::widget::InitData&) /builds/worker/checkouts/gecko/xpfe/appshell/AppWindow.cpp:213:17
#14 0x7f59c1afd1a9 in nsAppShellService::JustCreateTopWindow(nsIAppWindow*, nsIURI*, unsigned int, int, int, bool, mozilla::AppWindow**) /builds/worker/checkouts/gecko/xpfe/appshell/nsAppShellService.cpp:672:15
#15 0x7f59c1afe19f in nsAppShellService::CreateTopLevelWindow(nsIAppWindow*, nsIURI*, unsigned int, int, int, nsIAppWindow**) /builds/worker/checkouts/gecko/xpfe/appshell/nsAppShellService.cpp:178:8
#16 0x7f59c290cb22 in nsAppStartup::CreateChromeWindow(nsIWebBrowserChrome*, unsigned int, nsIOpenWindowInfo*, bool*, nsIWebBrowserChrome**) /builds/worker/checkouts/gecko/toolkit/components/startup/nsAppStartup.cpp:750:15
#17 0x7f59c2ad3ede in nsWindowWatcher::CreateChromeWindow(nsIWebBrowserChrome*, unsigned int, nsIOpenWindowInfo*, nsIWebBrowserChrome**) /builds/worker/checkouts/gecko/toolkit/components/windowwatcher/nsWindowWatcher.cpp:441:33
#18 0x7f59c2acff80 in nsWindowWatcher::OpenWindowInternal(mozIDOMWindowProxy*, nsTSubstring<char> const&, nsTSubstring<char> const&, nsTSubstring<char> const&, bool, bool, bool, nsIArray*, bool, bool, bool, nsPIWindowWatcher::PrintKind, nsDocShellLoadState*, mozilla::dom::BrowsingContext**) /builds/worker/checkouts/gecko/toolkit/components/windowwatcher/nsWindowWatcher.cpp:1048:12
#19 0x7f59c2acbbc4 in nsWindowWatcher::OpenWindow(mozIDOMWindowProxy*, nsTSubstring<char> const&, nsTSubstring<char> const&, nsTSubstring<char> const&, nsISupports*, mozIDOMWindowProxy**) /builds/worker/checkouts/gecko/toolkit/components/windowwatcher/nsWindowWatcher.cpp:297:3
#20 0x7f59c2bede26 in ShowProfileManager(nsIToolkitProfileService*, nsINativeAppSupport*) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:2798:27
#21 0x7f59c2bed18a in SelectProfile(nsToolkitProfileService*, nsINativeAppSupport*, nsIFile**, nsIFile**, nsIToolkitProfile**, bool*) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:2947:12
#22 0x7f59c2be900a in XREMain::XRE_mainStartup(bool*) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:4812:8
#23 0x7f59c2bf9cd4 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5831:12
#24 0x7f59c2bfae79 in XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5899:21
#25 0x5591bd2f4b88 in do_main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:226:22
#26 0x5591bd2f4b88 in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:423:16
#27 0x7f59ccc43b49 in __libc_start_call_main /usr/src/debug/glibc-2.37-1.fc38.x86_64/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#28 0x7f59ccc43c0a in __libc_start_main@GLIBC_2.2.5 /usr/src/debug/glibc-2.37-1.fc38.x86_64/csu/../csu/libc-start.c:360:3
#29 0x5591bd232dd8 in _start (/programs/firefox-asan/firefox-bin+0xacdd8) (BuildId: 3b8a3ab238f2b97d6daef662b0b135d20bfca7b5)

==7424==ABORTING

When I started and closed the Firefox 113.0a1 20230405093623 ASan build normally, AddressSanitizer:DEADLYSIGNAL was shown, but there wasn't a trace.

Comment 5

[Matt Fagnani]

Attachment: Full trace of all threads of Firefox 111.0.1 on Wayland crash when closing

Here's the trace of Firefox 111.0.1 on Wayland crashing when closing with libwayland-client-1.22.0-1.fc38 from coredumpctl gdb.

Core was generated by `/usr/lib64/firefox/firefox --name firefox-wayland'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=11,
no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
[Current thread is 1 (Thread 0x7f6d755ff6c0 (LWP 10443))]

(gdb) bt
#0 __pthread_kill_implementation
(threadid=<optimized out>, signo=signo@entry=11, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x00007f6d9caaac03 in __pthread_kill_internal (signo=11, threadid=<optimized out>)
at pthread_kill.c:78
#2 0x00007f6d9ca59aee in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
#3 0x00007f6d956f0c90 in nsProfileLock::FatalSignalHandler(int, siginfo_t*, void*)
(signo=11, info=<optimized out>, context=0x7f6d755fe1c0)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/toolkit/profile/nsProfileLock.cpp:174
#4 0x00007f6d9ca59ba0 in <signal handler called> () at /lib64/libc.so.6
c#5 MOZ_Crash
(aReason=0x7f6d5df30600 "warning: queue 0x7f6d777ce4f0 destroyed while proxies still attached:\n", aLine=275, aFilename=<synthetic pointer>)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/objdir/dist/include/mozilla/Assertions.h:261
#6 mozilla::widget::WlCrashHandler(const char , typedef __va_list_tag __va_list_tag )
(format=<optimized out>, args=<optimized out>)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/widget/gtk/nsWaylandDisplay.cpp:275
#7 0x00007f6d9aaeac9e in wl_log
(fmt=fmt@entry=0x7f6d9aaf0b30 "warning: queue %p destroyed while proxies still attached:\n")
at ../src/wayland-util.c:449
#8 0x00007f6d9aaec854 in wl_event_queue_release (queue=queue@entry=0x7f6d777ce4f0)
at ../src/wayland-client.c:308
#9 0x00007f6d9aaec929 in wl_event_queue_destroy (queue=0x7f6d777ce4f0)
at ../src/wayland-client.c:352
--Type <RET> for more, q to quit, c to continue without paging--c
#10 0x00007f6d6b756d84 in dri2_teardown_wayland (dri2_dpy=0x7f6d799eaa00)
at ../src/egl/drivers/dri2/platform_wayland.c:2802
#11 0x00007f6d6b74be58 in dri2_display_destroy (disp=disp@entry=0x7f6d777c2b00)
at ../src/egl/drivers/dri2/egl_dri2.c:1153
#12 0x00007f6d6b74c380 in dri2_display_release (disp=0x7f6d777c2b00)
at ../src/egl/drivers/dri2/egl_dri2.c:1115
#13 dri2_display_release (disp=0x7f6d777c2b00) at ../src/egl/drivers/dri2/egl_dri2.c:1100
#14 dri2_terminate (disp=0x7f6d777c2b00) at ../src/egl/drivers/dri2/egl_dri2.c:1219
#15 0x00007f6d6b739abc in eglTerminate (dpy=<optimized out>) at ../src/egl/main/eglapi.c:769
#16 0x00007f6d91f2470e in mozilla::gl::GLLibraryEGL::fTerminate(void) const
(display=<optimized out>, this=<optimized out>)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/gfx/gl/GLLibraryEGL.h:254
#17 mozilla::gl::EglDisplay::fTerminate() (this=0x7f6d777e9fb0)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/gfx/gl/GLLibraryEGL.h:742
#18 mozilla::gl::EglDisplay::EglDisplay() (this=0x7f6d777e9fb0, __in_chrg=<optimized out>)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/gfx/gl/GLLibraryEGL.cpp:825
#19 0x00007f6d91bdc44e in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use()
(this=0x7f6d777e9fa0) at /usr/include/c++/13/bits/shared_ptr_base.h:175
#20 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()
(this=0x7f6d777e9fa0) at /usr/include/c++/13/bits/shared_ptr_base.h:199
#21 0x00007f6d91f3e82e in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count()
(this=0x7f6d9c89e490, __in_chrg=<optimized out>)
at /usr/include/c++/13/bits/shared_ptr_base.h:1071
#22 std::__shared_ptr<mozilla::gl::EglDisplay, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr()
(this=0x7f6d9c89e488, __in_chrg=<optimized out>)
at /usr/include/c++/13/bits/shared_ptr_base.h:1524
#23 std::shared_ptr<mozilla::gl::EglDisplay>::~shared_ptr()
(this=0x7f6d9c89e488, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/shared_ptr.h:175
#24 mozilla::gl::GLContextEGL::~GLContextEGL() (this=0x7f6d9c89da00, __in_chrg=<optimized out>)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/gfx/gl/GLContextProviderEGL.cpp:409
#25 0x00007f6d91f3ea2d in mozilla::gl::GLContextEGL::~GLContextEGL()
(this=0x7f6d9c89da00, __in_chrg=<optimized out>)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/gfx/gl/GLContextProviderEGL.cpp:409
#26 0x00007f6d92213135 in mozilla::wr::RenderThread::ShutDownTask() (this=0x7f6d9c841900)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/gfx/webrender_bindings/RenderThread.cpp:201
#27 0x00007f6d91b9233e in mozilla::detail::runnable_args_base<(mozilla::detail::RunnableResult)0>::Run() (this=<optimized out>)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/dom/media/webrtc/transport/runnable_utils.h:41
#28 0x00007f6d915308d4 in nsThread::ProcessNextEvent(bool, bool)
(this=0x7f6d77618800, aMayWait=<optimized out>, aResult=0x7f6d755feb77)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/objdir/dist/include/nsCOMPtr.h:851
#29 0x00007f6d9151d7ef in NS_ProcessNextEvent(nsIThread*, bool) (aThread=<optimized out>,
aThread@entry=0x7f6d77618800, aMayWait=aMayWait@entry=true)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/xpcom/threads/nsThreadUtils.cpp:477
#30 0x00007f6d91bf1dca in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) (this=0x7f6d776bcac0, aDelegate=0x7f6d755fec40)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/ipc/glue/MessagePump.cpp:330
#31 0x00007f6d91bacbe2 in MessageLoop::RunInternal() (this=0x7f6d9c89da00)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/objdir/dist/include/mozilla/RefPtr.h:280
#32 MessageLoop::RunHandler() (this=0x7f6d9c89da00)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/ipc/chromium/src/base/message_loop.cc:374
#33 MessageLoop::Run() (this=this@entry=0x7f6d755fec40)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/ipc/chromium/src/base/message_loop.cc:356
#34 0x00007f6d915300ea in nsThread::ThreadFunc(void*) (aArg=<optimized out>)
at /usr/src/debug/firefox-111.0.1-1.fc38.x86_64/xpcom/threads/nsThread.cpp:384
#35 0x00007f6d9bfe1759 in _pt_root () at /lib64/libnspr4.so
#36 0x00007f6d9caa8c57 in start_thread (arg=<optimized out>) at pthread_create.c:444
#37 0x00007f6d9cb2ea70 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

I'm attaching the full trace of all threads.

Comment 6

[Matt Fagnani]

Using mozregression, 101.0a1 20220501190841 didn't show a crash when closing, while closing 105.0a1 20220801215129 showed WARNING: Process exited with code 11. The Mozilla Crash Reporter is disabled when using mozregression, so interpreted that crash message as indicating the build had this problem with wayland 1.22.0. I ran MOZ_ENABLE_WAYLAND=1 mozregression --good 2022-5-1 --bad 2022-8-1 --persist /tmp/mozregression

The first bad revision had 12 Wayland/Linux related changes for Bug 1777664 including at least two involving destroying the layer manager.
[Linux] Destory layer manager at nsWindow::ReleaseGdkWindow() https://hg.mozilla.org/integration/autoland/rev/af89aa9ca4fbbaccbcd895eb33c89a8e141d7c85
[Linux] Replace CleanLayerManagerRecursive() with DestroyLayerManager() as they're the same https://hg.mozilla.org/integration/autoland/rev/0fa3652858e9181ace74c9c72e62e46c58eed85a

12:18.71 INFO: Narrowed integration regression window from [2cbab5d7, cd97145a] (3 builds) to [2cbab5d7, e081a912] (2 builds) (~1 steps left)
12:18.71 INFO: No more integration revisions, bisection finished.
12:18.71 INFO: Last good revision: 2cbab5d7e2a99011a19f9c9cc3c3724ccd755f5d
12:18.71 INFO: First bad revision: e081a91279ccae716d4717ed818fe297a68836aa
12:18.71 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=2cbab5d7e2a99011a19f9c9cc3c3724ccd755f5d&tochange=e081a91279ccae716d4717ed818fe297a68836aa

Could those changes have led to the Wayland queue in the trace being freed while still in use with Wayland 1.22.0?

Comment 7

[Ryan VanderMeulen [:RyanVM]]

This is quickly becoming a Nightly topcrash.

Comment 8

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

It may be a bug in wayland-1.22.0 library: https://bugzilla.redhat.com/show_bug.cgi?id=2184842

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 10 desktop browser crashes on nightly (startup)

:stransky, could you consider increasing the severity of this top-crash bug?

For more information, please visit auto_nag documentation.

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox112 (beta) and tracked for firefox113 (nightly). We have limited time to fix this, the soft freeze is today. However, the bug still isn't assigned.

:gcp, could you please find an assignee for this tracked bug? Given that it is a regression and we know the cause, we could also simply backout the regressor. If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit auto_nag documentation.

Comment 11

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Attachment: Bug 1826583 [Wayland] Don't crash on Wayland log handler r?emilio

Comment 12

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Origin looks like Mesa/EGL bug: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21646
We crash because we do that in Wayland log handler.

Comment 14

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

I think we need to store the log message somewhere and use it in crash handler if we're terminated by wl_abort().

Comment 15

[gene bug]

I see 2 different crashes depending on which computer - one on start and other on exit.

Info logged here:

https://gitlab.freedesktop.org/wayland/wayland/-/issues/374

Comment 16

[michelesr]

With nvidia libaries installed, I get a crash at startup -> https://crash-stats.mozilla.org/report/index/0cc5771e-fc21-4ec0-87f9-915790230407
Without nvidia EGL libraries installed, I get this crash when I close Firefox -> https://crash-stats.mozilla.org/report/index/ebf901e5-e0e5-4563-a1a1-666050230407

I suppose they are different manifestations of the same problem.

Comment 17

[gene bug]

This is was resolved for me on Arch with the release of mesa 23.0.2
I am NOT using nvidia anything

Comment 18

[michelesr]

On Arch Linux, downgrading the wayland package to 1.21 from 1.22 I cannot reproduce the issue anymore.

Comment 19

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

(In reply to gene bug from comment #17)

This is was resolved for me on Arch with the release of mesa 23.0.2
I am NOT using nvidia anything

Please report that at https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21646
It's MESA bug.

Comment 20

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Firefox has a bug that we crash/submit report at any wayland log event.

Correct Firefox approach will be to save/log wayland log event and if any abort() signal comes from wl_abort() we should use that log message and submit it along with the crash report.

Comment 21

[Darkspirit]

Can you printf if wl_log contains "destroyed while proxies still attached" and otherwise keep the MOZ_CRASH, so that we don't miss other warnings?

Comment 22

[CapSel]

upgrading mesa to 23.0.2 did not help. I use nvidia.
my current workaround is to run firefox wrapped in prime-run script:

#!/bin/bash
__NV_PRIME_RENDER_OFFLOAD=1 __VK_LAYER_NV_optimus=NVIDIA_only __GLX_VENDOR_LIBRARY_NAME=nvidia "$@"

Comment 23

[michelesr]

On Arch Linux updating through pacman fixed the issue.

Wayland: 1.22.0
Mesa: 23.0.2

I cannot reproduce the bug, even when nvidia drivers and libraries are loaded.

Comment 24

[BELL]

(In reply to michelesr from comment #23)

On Arch Linux updating through pacman fixed the issue.

I cannot reproduce the bug, even when nvidia drivers and libraries are loaded.
I believe Archlinux just backport that mesa bug fix and repack it. because the version turns 23.0.2-1 to 23.0.2-2

that makes sense, since this bug fix only one line code change and does not affect any existing function.

Comment 25

[BELL]

(In reply to BELL from comment #24)

I believe Archlinux just backport that mesa bug fix and repack it. because the version turns 23.0.2-1 to 23.0.2-2

Comment 26

[lilydjwg]

They backported an egl-wayland patch to fix the nvidia one too.

Comment 27

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1777664

Comment 28

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

This is not a regression but a bug in Mesa GFX library.

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ddba22b4de24
[Wayland] Don't crash on Wayland log handler r=emilio

Comment 30

[Wayne Mery (:wsmwk)]

#3 crash for linux + Thunderbird

Comment 31

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/ddba22b4de24

Comment 32

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Comment on attachment 9327427 [details]
Bug 1826583 [Wayland] Don't crash on Wayland log handler r?emilio

Beta/Release Uplift Approval Request

• User impact if declined: Crashes on broken Mesa (gfx drivers) on Wayland.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): We don't quit Firefox on Wayland log handler but rather log wayland log as gfx critical error which is printed in crash report if Wayland default log handler aborts Firefox later in wl_abort() in case the error is fatal.
• String changes made/needed:
• Is Android affected?: Yes

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: We'd like to reduce noise from crashes on broken setups (old MESA) on Wayland.
• User impact if declined: Crashes on broken Mesa (gfx drivers) on Wayland.
• Fix Landed on Version:
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): We don't quit Firefox on Wayland log handler but rather log wayland log as gfx critical error which is printed in crash report if Wayland default log handler aborts Firefox later in wl_abort() in case the error is fatal.

Comment 33

[Pascal Chevrel:pascalc]

Comment on attachment 9327427 [details]
Bug 1826583 [Wayland] Don't crash on Wayland log handler r?emilio

Approved for 113 beta 3, thanks.

Comment 34

[Pascal Chevrel:pascalc]

https://hg.mozilla.org/releases/mozilla-beta/rev/1ce91ea81758

Comment 35

[Dianna Smith [:diannaS]]

do you want to nominate this for release uplift this for the 112 dot release as well?

Comment 36

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9327427 [details]
Bug 1826583 [Wayland] Don't crash on Wayland log handler r?emilio

Approved for 102.11esr.

Comment 37

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr102/rev/a6c2a5af588f

Comment 38

[Martin Stránský [:stransky] (ni? me) (PTO, back on Feb 17)]

Comment on attachment 9327427 [details]
Bug 1826583 [Wayland] Don't crash on Wayland log handler r?emilio

Beta/Release Uplift Approval Request

• User impact if declined: Crashes on broken Mesa (gfx drivers) on Wayland.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): We don't quit Firefox on Wayland log handler but rather log wayland message as a gfx critical error which is printed in crash report if Wayland default log handler aborts Firefox later in wl_abort() in case the error is fatal.
• String changes made/needed:
• Is Android affected?: No

Comment 39

[Dianna Smith [:diannaS]]

Comment on attachment 9327427 [details]
Bug 1826583 [Wayland] Don't crash on Wayland log handler r?emilio

Approved for 112.0.2 dot release

Comment 40

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-release/rev/94ef9c555302