Open Bug 1965713 Opened 1 year ago Updated 9 days ago

Crash in [@ __syscall_cancel_arch]

Categories

(Core :: XPCOM, defect)

Other
Linux
defect

Tracking

()

Tracking Status
firefox140 --- affected

People

(Reporter: release-mgmt-account-bot, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: crash, Whiteboard: [tbird crash])

Crash Data

Attachments

(1 file)

Crash report: https://crash-stats.mozilla.org/report/index/84928c8e-96ca-4a83-9c36-a41660250511

Reason: SIGABRT

Top 10 frames of crashing thread:

0  libc.so.6  __syscall_cancel_arch  sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
1  libc.so.6  __internal_syscall_cancel  nptl/nptl/cancellation.c:49
2  libc.so.6  __futex_abstimed_wait_common64  nptl/nptl/futex-internal.c:57
2  libc.so.6  __futex_abstimed_wait_common  nptl/nptl/futex-internal.c:87
3  libc.so.6  __pthread_cond_wait_common  nptl/nptl/pthread_cond_wait.c:426
3  libc.so.6  pthread_cond_wait@@GLIBC_2.3.2  nptl/nptl/pthread_cond_wait.c:458
4  firefox-bin  mozilla::detail::ConditionVariableImpl::wait  mozglue/misc/ConditionVariable_posix.cpp:106
4  firefox-bin  mozilla::detail::ConditionVariableImpl::wait_for  mozglue/misc/ConditionVariable_posix.cpp:113
5  libxul.so  mozilla::OffTheBooksCondVar::Wait  xpcom/threads/CondVar.h:65
5  libxul.so  mozilla::Monitor::Wait  xpcom/threads/Monitor.h:39

By querying Nightly crashes reported within the last 2 months, here are some insights about the signature:

  • First crash report: 2025-03-06
  • Process type: Multiple distinct types
  • Is startup crash: No
  • Has user comments: No
  • Is null crash: Yes - 1 out of 18 crashes happened on null or near null memory address

The Bugbug bot thinks this bug should belong to the 'Core::XPCOM' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: General → XPCOM

I think this is more of a new crash signature variation for ConditionVariableImpl::wait than a new crash.

Severity: -- → S3

Indeed, it's probably worth ignoring that frame.

Whiteboard: [tbird crash]

bugzilla.redhat.com/show_bug.cgi?id=2375067#c14 is an example of when I have encountered this, per #bugzilla.

I dug a bit through this crashes and there's something I don't understand: it seems that we're waiting on a condition variable w/ a timeout, the timeout expired and we're now trying to cancel the wait, and this operation triggers a SIGABRT at the point where we invoke the syscall, which is bizarre. Anyway the code around the crashing point looks like this.

A thing about crashes sent with kill or similar on Linux is that the si_addr field is not used, but si_pid and si_uid occupy the same space, so the crash report includes the pid of the process that sent the signal as part of the β€œaddress”. It also includes, in the raw minidump, the thread IDs for every thread in the process. In this case, the SIGABRT was sent by a different process owned by the same user. I think it might be some kind of external hang monitor that's part of the desktop environment, force-killing Firefox because it got stuck.

Looking at the stack for bp-84928c8e-96ca-4a83-9c36-a41660250511, we got an input event (the end of a touchpad drag maybe?), we're trying to delete a popup window, which sends a sync IPC message to the compositor, which is also in the parent process, so, considering the Compositor thread… I think this could be a deadlock in APZC? Other crashes under this signature might have different causes.

Thanks Jed, this is very useful! So the 3e8 address that keeps coming up is really uid 1000 which makes a lot of sense. The comments are all over the place (there's probably 3 or 4 underlying causes) but these must all be crashes triggered by Firefox hanging and being killed by whatever hang-detection service the desktop environment has.

Not any desktop environment either. It seems that >90% of these crashes are from users with KDE Plasma on Wayland.

Depends on: 1985968

I'll chime in here to corroborate that KDE Plasma does indeed have its own hang monitor and crash reporter.

The crash reporter (drkonqi) is solely utilised for applications distributed by KDE. Though, the OS may hook into it with a myriad of tools (like systemd's coredumpctl, alongside abrt, which consumes its cores, but also monitors crashes for data which it collects, all of which may be invoked alongside drkonqi).

Considering that I most recently encountered this in discuss.kde.org/t/39560/5, which appeared to be an organic example of [Q]DBus being overwhelmed, despite it also occurring in the very different ticket which #c4 cites, I agree that this may generally be a red-herring. However, it definitely occurs in more than hung states.

This bug has had no comment from a person since 2025-09-13. Clouseau analysed a recent crash report on its signature; the analysis is below.

Crash report: https://crash-stats.mozilla.org/report/index/cb2d9db5-9c27-4f16-91f3-59abf0261010

Crash Reason:

SIGABRT at 0x000003e800032445

Top 10 frames:

0  libc.so.6  __syscall_cancel_arch
1  libc.so.6  __GI_ppoll
2  libglib-2.0.so.0  g_main_context_iterate_unlocked
3  libglib-2.0.so.0  g_main_loop_run
4  libgio-2.0.so.0  initable_init
5  libgio-2.0.so.0  g_initable_new_valist
6  libgio-2.0.so.0  g_initable_new
7  libgio-2.0.so.0  g_dbus_proxy_new_for_bus_sync
8  libxul.so  mozilla::dom::GCLocProviderPriv::Startup()  dom/system/linux/GeoclueLocationProvider.cpp:1037
9  libxul.so  mozilla::GeolocationService::Init()  toolkit/components/geolocation/GeolocationService.cpp:92

There is 1 crash in nightly 159 with buildid 20261007162542.

In the last 30 days, Firefox has 354 crashes (from 183 installations) on esr, 178 crashes (from 48 installations) on release, 8 crashes (from 8 installations) on beta/DevEdition and 6 crashes (from 4 installations) on nightly with this signature.

This signature has been reported since build 20240331093654 (2024-03-31), 920 days before the build above.

This bug looks actionable because:

  • The parent-process main thread blocks inside g_dbus_proxy_new_for_bus_sync in GCLocProviderPriv::Startup (GeoclueLocationProvider.cpp:1037) while waiting on the system bus for Geoclue, until another process kills it with SIGABRT.

    1. A content process sends AddGeolocationListener, and ContentParent::RecvAddGeolocationListener runs it on the parent main thread through Geolocation::NonWindowSingleton and GeolocationService::GetGeolocationService.
    2. GeolocationService::Init probes Geoclue by calling gcProvider->Startup() synchronously, when geo.provider.use_geoclue is set and the location portal is not used.
    3. GCLocProviderPriv::Startup calls g_dbus_proxy_new_for_bus_sync(G_BUS_TYPE_SYSTEM, G_DBUS_PROXY_FLAGS_NONE, ...) for the Geoclue Manager, and GLib spins a nested g_main_loop_run in initable_init waiting in ppoll for the bus reply.
    4. While the reply is pending the main thread runs nothing else, so the browser appears hung, and a same-user process (uid 1000, pid 205893) sends SIGABRT.

    The stack shows the nested wait directly (frames #0-#8), and the fault address 0x000003e800032445 holds the sender's uid 1000 and pid 205893, so the signal came from another process. This fits bug 1965713 comments 6-8, which describe this signature as hangs killed by a desktop hang monitor. Why the system bus or Geoclue was slow on this machine is not recorded in the report. Look first at making the Geoclue availability probe in GeolocationService::Init / GCLocProviderPriv::Startup asynchronous (g_dbus_proxy_new_for_bus with a callback) or moving it off the main thread during IPC message handling.

  • The failing code comes from 7357f3be9f82 (bug 1063572) by :mail.

Code references:

:nika, as triage owner, can you have a look please?

Posted automatically by Clouseau, which analyses nightly crashes with an LLM. Nothing above was written or checked by a human.

Flags: needinfo?(nika)

Note: there's a wild range of stacks leading to this, comment 11 is probably one of many possible reasons why we get stuck and might not even be very common. We should first adjust the signature for this so that we can separate the root causes and clearly label each crash as an external kill instead of relying on us manually sifting through this individual signature.

Once we get bug 587729 through the finish line, we should have a clearer picture as external kills on Linux should have a SignalOrigin annotation. However, I suppose I should finish that work and surface that annotation in the public crash reports!

See Also: → 2080739

Yes, the ultimate goal here would be to change the signature from __syscall_cancel_arch to something like [external kill by <program>] <signature_with_the_syscall_cancel_frames_ignored>.

:gsvelto is more familiar and has already taken a look.

Flags: needinfo?(nika)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: