1778564 - UBSan crash in moz_container_wayland_frame_callback_handler

Status: RESOLVED DUPLICATE of bug 1778767

(Core :: Widget: Gtk, defect)

Description

[Jim Blandy :jimb]

When I build Firefox with the attached mozconfig on Fedora Core 36 and run it in a Wayland session, UBSan complains (and crashes the process) as soon as Firefox opens a window:

MozContainerWayland.cpp:247:5: runtime error: call to function wl_callback_destroy(wl_callback*) through pointer to incorrect function type void (*)(void *)

/home/jimb/moz/central/widget/gtk/MozContainerWayland.cpp:247:5: runtime error: call to function wl_callback_destroy(wl_callback*) through pointer to incorrect function type 'void (*)(void *)'
/home/jimb/.mozbuild/sysroot-x86_64-linux-gnu/usr/include/wayland-client-protocol.h:335: note: wl_callback_destroy(wl_callback*) defined here
    #0 0x7ff775f88b88 in moz_container_wayland_frame_callback_handler(void*, wl_callback*, unsigned int) /home/jimb/moz/central/widget/gtk/MozContainerWayland.cpp:247:5
    #1 0x7ff78f173745  (/lib64/libffi.so.8+0x7745) (BuildId: bfe3659c787e4ca7b1c367d1353ce552da06f75d)
    #2 0x7ff78f1704d1  (/lib64/libffi.so.8+0x44d1) (BuildId: bfe3659c787e4ca7b1c367d1353ce552da06f75d)
    #3 0x7ff78f285e02  (/lib64/libwayland-client.so.0+0x7e02) (BuildId: 8310a7a579a6e25d0ef3e9d3fa70b753db183519)
    #4 0x7ff78f286572  (/lib64/libwayland-client.so.0+0x8572) (BuildId: 8310a7a579a6e25d0ef3e9d3fa70b753db183519)
    #5 0x7ff78f28673b in wl_display_dispatch_queue_pending (/lib64/libwayland-client.so.0+0x873b) (BuildId: 8310a7a579a6e25d0ef3e9d3fa70b753db183519)
    #6 0x7ff78fc182aa  (/lib64/libgdk-3.so.0+0x712aa) (BuildId: 3b29fbe36b80a21fa1cee8ac8f403376327ad437)
    #7 0x7ff78fbde3a0 in gdk_display_get_event (/lib64/libgdk-3.so.0+0x373a0) (BuildId: 3b29fbe36b80a21fa1cee8ac8f403376327ad437)
    #8 0x7ff78fc1c045  (/lib64/libgdk-3.so.0+0x75045) (BuildId: 3b29fbe36b80a21fa1cee8ac8f403376327ad437)
    #9 0x7ff78fabffae in g_main_context_dispatch (/lib64/libglib-2.0.so.0+0x54fae) (BuildId: 28db949e96e0a76c593d4a30e8104576a0c3da62)

Comment 1

[Jim Blandy :jimb]

My build is seeing the following declaration of g_clear_pointer from ~/.mozbuild/sysroot-x86_64-linux-gnu/usr/include/glib-2.0/glib/gmem.h:

extern
void g_clear_pointer (gpointer *pp,
                           GDestroyNotify destroy);

with:

typedef void* gpointer;

and

typedef void (*GDestroyNotify) (gpointer data);

Per the C++ standard [expr.call] para 6,

Calling a function through an expression whose function type E is different from the function type F of the called function’s definition results in undefined behavior unless the type “pointer to F” can be converted to the type “pointer to E” via a function pointer conversion (7.3.14).

§7.3.14 only talks about converting between noexcept and ordinary function pointers.

Upshot: I think UBSan is right.

Comment 2

[Jim Blandy :jimb]

Indeed, writing out the call to g_clear_pointer straightens out that warning (but we hit another g_clear_pointer call immediately):

@@ -244,7 +244,8 @@ static void moz_container_wayland_frame_
   {
     // Protect mozcontainer internals changes by container_lock.
     MutexAutoLock lock(*wl_container->container_lock);
-    g_clear_pointer(&wl_container->frame_callback_handler, wl_callback_destroy);
+    wl_callback_destroy(wl_container->frame_callback_handler);
+    wl_container->frame_callback_handler = nullptr;
     // It's possible that container is already unmapped so quit in such case.
     if (!wl_container->surface) {
       LOGWAYLAND("  container is unmapped, quit.");