2038640 - TSAN reports: How do we white list libdbus lock inversion issues when we test TSAN build (for C-C TB in my case) on treeherder

Status: NEW

(Testing :: Mozbase, defect)

Description

[ISHIKAWA, Chiaki]

I am testing the bugfixes for IMAP module of Comm-Central Thunderbird. It had thread-race issues.

Starting a few days ago, TSAN tests running on my local PC running Debian GNUJ/Linux began occasionally create TSAN runtime reports regarding lock inversion issue in system libdbus library.
It was rare on my local PC somehow.

However, submitting the TSAN test of C-C TB on treeherder now produces so many libdbus TSAN reports and is basically useless.
https://treeherder.mozilla.org/jobs?repo=try-comm-central&selectedTaskRun=Oh84TX5-QrGJ0QZnNYAFjg.0&revision=3d77cfc015f14f10385f09218286618a8d76601f

I think I have to ADD whitelist entries to TSAN whitelist when I run TSAN test on treeherder.

Immediate questions come to my mind:
Where should I create the whitelist ?
-> I think I can create one that has all the TSAN whitelist entries of my local PC at, say, ./comm central subdirectory.
This is reasonable since I am testing C-C TB.

How do I specify the whitelist? This is a blocker.
I can pass the environment variable
TSAN_OPTIONS=path-to-the-whitelist-file ...
using not so well documented --setenv paramter to the test process and TSAN binary picks it up.

HOWEVER, I am not sure what the absolute pathname of the whitelist file created under ./comm subdirectory is.

.../comm/tsan-whitelist.txt

TSAN test needs to understand EITHER the absollute pathname of the above file, or relative pathname of the above file to reference it successfully when it runs.

Locally it is easy. I specify the absolute path name:
export TSAN_OPTIONS="suppressions=/home/ishikawa/Dropbox/TB-DIR/tsan-whitelist.txt verbosity=1"
before running local TSAN test.

I am not sure if the absolute pathname of the test environemtn is consistent across different submissions even.
So relative pathname seems to be the way to go.

Any thought?

BTW, this superceding whatever system may provide to TSAN_OPTIONS, esp. whitelisting, is faulty for mochitest.
xpcshell test honors the setting by the user, but mochitest ignores it and overrides the setting with its own.
This is Bug 2031796

So installing per user whitelist to work around this libdbus TSAN issue won't be functional until that bug is fixed.

The libdbus lock inversion issue seems to be widely known.
E.g. https://www.google.com/search?client=firefox-b-d&q=libdbus+lock+inversion+TSAN+report

I don't know why it started to appear, but obviously the problem started after my local linux upgrade to take care of the serious
security issue, i.e., Copy Fail, etc.
https://www.microsoft.com/en-us/security/blog/2026/05/01/cve-2026-31431-copy-fail-vulnerability-enables-linux-root-privilege-escalation/
https://www.elastic.co/security-labs/copy-fail-dirtyfrag-linux-page-bugs-in-the-wild

Someone mentioned libdbus was missing in his local PC installation about one month ago (and probably my PC also before the recent update), and thus this TSAN issue was not noticed on such linux installation.
E.g.: Bug 2017096 (filed two months ago)

The recent upgrade of the linux kernel due to security issue must have mandated the use of libdbus in the new desktop infrastructure, thus the sudden appearance of libdbus TSAN reports (I understand treeherder uses ubuntu and it uses many of Debian distribution packages).

Any ideas how to cope with issue, and the fix of Bug 2031796 is appreciated.

Comment 1

[Worcester12345]

Would knowing the SPECIFIC VERSION of Linux help here?

See: https://bugzilla.mozilla.org/show_bug.cgi?id=2052728

Comment 2

[Wayne Mery (:wsmwk)]

jmaher, is comment 0 something you can answer?

Comment 3

[ISHIKAWA, Chiaki]

(In reply to Worcester12345 from comment #1)

Would knowing the SPECIFIC VERSION of Linux help here?

See: https://bugzilla.mozilla.org/show_bug.cgi?id=2052728

Sorry, I was tied up with firefighting work for day workfor the last couple of months.

Mine is Debian GNU/linux and I suppose treeherder uses Ubuntu (which is a derivative of Debian.)

Searching by google lists many known libdbus lock inversion issues, and people have to suppress them using whitelist.
I suspect this is not distribution specific.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The task linked in comment 0

• had a status exit code which reported a failure
• only lists the lock-order inversion issues

Adding the lock-order inversion issue to an ignore list in Treeherder would still report the task as failed but no error lines in the Failure Summary tab.

Comment 5

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:ahal, could you have a look please?

For more information, please visit BugBot documentation.