1751238 - UBSan build of firefox crashing when launched under rr (eglQueryDeviceStringEXT returning nullptr)

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[Matthew Gaudet (he/him) [:mgaudet]]

While trying to investigate a bug, I tried to get an rr recording, but I was unable to run the firefox under grizzly when using rr, as it would fail to launch and immediately report an asan failure:

$ cat /tmp/grizzly/launch_failures/c3aafac2_2022-01-20_10-47-57_logs/log_ffp_asan_1496956.log.1497312.txt 
==1497312==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x000000000000 (pc 0x1ad34e09e5b1 bp 0x7ffe18c807b0 sp 0x7ffe18c806f8 T1497312)
==1497312==The signal is caused by a READ memory access.
==1497312==Hint: address points to the zero page.
    #0 0x1ad34e09e5b1  /build/glibc-eX1tMB/glibc-2.31/string/../sysdeps/x86_64/multiarch/strstr-sse2-unaligned.S:40
    #1 0x57086f41d587 in get_gles_status /builds/worker/checkouts/gecko/toolkit/xre/glxtest.cpp:544:9
    #2 0x57086f41d587 in get_egl_status(void*, bool, bool) /builds/worker/checkouts/gecko/toolkit/xre/glxtest.cpp:637:21
    #3 0x57086f4167c5 in x11_egltest /builds/worker/checkouts/gecko/toolkit/xre/glxtest.cpp:901:8
    #4 0x57086f4167c5 in childgltest /builds/worker/checkouts/gecko/toolkit/xre/glxtest.cpp:1281:10
    #5 0x57086f416e54 in fire_glxtest_process() /builds/worker/checkouts/gecko/toolkit/xre/glxtest.cpp:1320:14
    #6 0x57086f404280 in XREMain::XRE_mainInit(bool*) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:3639:3
    #7 0x57086f40f505 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5513:16
    #8 0x57086f40fe09 in XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5598:21
    #9 0x55dd84ec1f45 in do_main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:225:22
    #10 0x55dd84ec1f45 in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:395:16
    #11 0x1ad34e0070b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16
    #12 0x55dd84e9d7bc in _start (/home/matthew/bugs/1750870/firefox/firefox-bin+0x157bc)

UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV /build/glibc-eX1tMB/glibc-2.31/string/../sysdeps/x86_64/multiarch/strstr-sse2-unaligned.S:40 
==1497312==ABORTING

STR:

1. Fetch the firefox build.

$ python3 -m fuzzfetch -d --fuzzing -n firefox
[2022-01-20 07:41:32] Identified task: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-fuzzing-debug
[2022-01-20 07:41:32] > Task ID: HxSMz2ykSPGirQ1PuqYD9A
[2022-01-20 07:41:32] > Rank: 1642673041
[2022-01-20 07:41:32] > Changeset: 3ddab45ce6bdbc0501b54ae64cbef202e8f92fb5
[2022-01-20 07:41:32] > Build ID: 20220120100401
[2022-01-20 07:41:33] > Downloading: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/HxSMz2ykSPGirQ1PuqYD9A/artifacts/public/build/target.tar.bz2 (279.72MiB total)
[2022-01-20 07:41:45] .. downloaded (22.54MB/s)
[2022-01-20 07:41:45] .. extracting
[2022-01-20 07:42:44] Extracted into /home/matthew/bugs/1750870/firefox

2. Create an empty html file to use as a test case: touch a.html
3. Launch with rr or pernosco recording:

python3 -m grizzly.replay ./firefox/firefox a.html --xvfb  --pernosco

Expected:

[2022-01-20 10:52:01] Starting Grizzly Replay
[2022-01-20 10:52:01] Ignoring: timeout, log-limit
[2022-01-20 10:52:01] Running with Xvfb
[2022-01-20 10:52:01] Using time limit: 30s, timeout: 45s
[2022-01-20 10:52:01] Repeat: 1, Minimum crashes: 1, Relaunch 1
[2022-01-20 10:52:05] Running test (1/1)...
...

Actual:

[2022-01-20 10:51:27] Starting Grizzly Replay
[2022-01-20 10:51:27] Ignoring: timeout, log-limit
[2022-01-20 10:51:27] Running with Xvfb
[2022-01-20 10:51:27] Running with RR (Pernosco mode)
[2022-01-20 10:51:27] Using time limit: 30s, timeout: 45s
[2022-01-20 10:51:27] Repeat: 1, Minimum crashes: 1, Relaunch 1
[2022-01-20 10:51:31] FFPuppet LaunchError: Failure waiting for browser connection

Comment 1

[Matthew Gaudet (he/him) [:mgaudet]]

Should mention: rr is revision fa3efe8c5c1c36ff52a9ae34161a3f0d5a835a66; I did however also try a revision about 6 months old which also failed.

Comment 2

[Matthew Gaudet (he/him) [:mgaudet]]

Here's a pernosco recording of me doing pernosco-record python3 -m grizzly.replay ./firefox.wont.rr.record/firefox a.html --xvfb -- https://pernos.co/debug/ij6VGqSjYj2rFpyuFKjFYQ/index.html#f{m[Qz4,AVw_,t[DQ,GHHm_,f{e[KPQ,AQ___/

Unfortunately, it's not symbolicated, so may not be particularly useful.

Comment 3

[Matthew Gaudet (he/him) [:mgaudet]]

Ok, got a symbolicated recording: https://pernos.co/debug/vR682q1dfwxUTj5Z7Zft4A/index.html#f{m[S+M,AVY_,t[BA,GH4t_,f{e[H7o,VIq5_,s{aAj82M6AA,bAbQ,uFjz25A,oFj7ILQ___/

This says that in glxtest.cpp the following is going wrong (only under RR!)

  EGLDeviceEXT device;
  if (eglQueryDisplayAttribEXT(dpy, EGL_DEVICE_EXT, (EGLAttrib*)&device) ==
      EGL_TRUE) {
    const char* deviceExtensions =
        eglQueryDeviceStringEXT(device, EGL_EXTENSIONS); // This is returning null, and so the following strstr crashes. 
  if (strstr(deviceExtensions, "EGL_MESA_device_software")) {
      record_value("MESA_ACCELERATED\nFALSE\n");
    } else {

Moving this to Core :: Graphics for some help, as I'm baffled.

Comment 4

[Robert Mader [:rmader]]

Hm weird, AFAIK that string should never be empty when the eglQueryDisplayAttribEXT query above succeeds. So I think strictly this is a driver bug. Anyway, adding a simple nullptr check should work, also for the else case below.

Comment 5

[Robert Mader [:rmader]]

Attachment: Bug 1751238 - Handle empty device extension strings in glxtest, r=#gfx-reviewers

While the string should never be empty when the extension is actually
supported (which is checked above), there seem to be edge-cases where
its the case.
Handle that gracefully.

Comment 6

[Matthew Gaudet (he/him) [:mgaudet]]

Hrm. I wonder if I should still open an rr bug for this.

Comment 7

[Matthew Gaudet (he/him) [:mgaudet]]

For completeness, opened https://github.com/rr-debugger/rr/issues/3058

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/23f61716bce4
Handle empty device extension strings in glxtest, r=gfx-reviewers,jgilbert

Comment 9

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/23f61716bce4