1824768 - TSan startup failures due to external (gfx related) libs

Status: RESOLVED FIXED

(Core :: Sanitizers, task)

Description

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

Attachment: tsan.txt

This is a start up crash on Ubuntu 22.04.

I believe this will require a suppression.

Comment 1

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

Jonathan, is this the TSan report you were seeing when working on bug 1824200? If not could you please add the log and I will add a suppression if needed.

Comment 2

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

Hmm, I'm going to assume that is a bugzilla issues since I've seen a few others do the exact same thing...

Comment 3

[Jonathan Kew [:jfkthame]]

(In reply to Tyson Smith [:tsmith] from comment #1)

Jonathan, is this the TSan report you were seeing when working on bug 1824200? If not could you please add the log and I will add a suppression if needed.

No, mine's "unlock of an unlocked mutex (or by a wrong thread)" inside of libappmenu-gtk3-parser.so.0. I'll attach a log.

(Note that I'm on an older Ubuntu, so I expect all the system libs are different versions than yours.)

Comment 4

[Jonathan Kew [:jfkthame]]

Attachment: tsan log from firefox startup on ubuntu 20.04.5 LTS

Comment 5

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

Attachment: Bug 1824768 - Add TSan suppresions for extenal libs. r=decoder

Comment 6

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

Jonathan: Would you mind testing the patch locally?

Or the try build (https://treeherder.mozilla.org/#/jobs?repo=try&revision=87a88b05de781c414298c2db6981af4548d68260)

Comment 7

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

Attachment: Bug 1824768 - Add TSan suppressions for external libs. r=decoder

Comment 8

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

Attachment: tsan-log-22.04-with-patch.txt

With the current patch applied I am still getting this error. I though the libwayland-client.so.0 entry would have prevented this, not sure what I am missing.

Comment 9

[Jonathan Kew [:jfkthame]]

(In reply to Tyson Smith [:tsmith] from comment #6)

Jonathan: Would you mind testing the patch locally?

Or the try build (https://treeherder.mozilla.org/#/jobs?repo=try&revision=87a88b05de781c414298c2db6981af4548d68260)

I tried the current patch in my local build, and it does avoid the error from comment 4; startup seems to get somewhat further, but then it dies with a new issue. This time it looks like a data race inside a Radeon driver, afaics. I'll attach a new log.

Comment 10

[Jonathan Kew [:jfkthame]]

Attachment: tsan log with patch, Ubuntu 20.04.5 LTS

Comment 11

[Jonathan Kew [:jfkthame]]

I was able to start the browser successfully after adding

+    "called_from_lib:libEGL_mesa\n"                 \
+    "called_from_lib:radeonsi_dri\n"                \

to the list, but I don't really know the significance of what I'm suppressing here... :)

Comment 12

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

Is the called_from_lib:libEGL_mesa\n line required? It is already in https://searchfox.org/mozilla-central/source/mozglue/build/TsanOptions.cpp#56

I see this when I add that line:
==12521==ThreadSanitizer: library '/lib/x86_64-linux-gnu/libEGL_mesa.so.0' that was matched against called_from_lib suppression 'libEGL_mesa' is unloaded

Comment 13

[Jonathan Kew [:jfkthame]]

Attachment: failure without the libEGL_mesa line

Without that line, it dies with the attached errors.

(Initially, I tried adding libLLVM-12 to the list, but still hit a problem, so then tried libEGL_mesa and that seems to have resolved it for me -- and then I didn't need libLLVM-12.)

Comment 14

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

I'll add Wayland related suppressions in Bug 1825171

Comment 15

[Jesse Schwartzentruber (:truber)]

Attachment: log_tsan_nvidia_x11.txt

Attached launch failure log on Debian with NVidia driver 525.116.03 (X11).

Comment 16

[Jesse Schwartzentruber (:truber)]

Attachment: log_tsan_nvidia_x11_#2.txt

Log from https://treeherder.mozilla.org/jobs?repo=try&revision=02e7f39e2a55758ea8ded80dc71ce697aa156392

Comment 17

[Jesse Schwartzentruber (:truber)]

Attachment: log_tsan_nvidia_x11_#3.txt

Crash with https://treeherder.mozilla.org/jobs?repo=try&revision=750be3aad97a93b3179a16712fed06838b827ef1 on NVidia.

Comment 18

[Jesse Schwartzentruber (:truber)]

Attachment: log_tsan_nvidia_x11_#4.txt

Crash with https://treeherder.mozilla.org/jobs?repo=try&revision=31935bd64d0fbfa423901f955a488505d9ab8969 on NVidia.
This trace doesn't seem to have any NVidia libs in it.

Comment 19

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

The build is now able to launch in VMs with Wayland, X11 and hardware using Intel and AMD cards. nvidia crashes for other reasons and I will log a follow up bug.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c26206463772
Add TSan suppressions for external libs. r=decoder

Comment 21

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/c26206463772