1616462 - Intermittent SUMMARY: ThreadSanitizer: data race /builds/worker/workspace/build/src/ipc/chromium/src/third_party/libevent/signal.c:125:16 in evsig_set_base_

Status: RESOLVED DUPLICATE of bug 1658072

(Core :: IPC, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: nbeleuzu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=289453443&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TFmdzKfcRPCMCQ1GziyX-g/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-02-19T02:47:11.701Z] 02:47:11 INFO - TEST-INFO | Ran 1 Loops
[task 2020-02-19T02:47:11.708Z] 02:47:11 INFO - SimpleTest FINISHED
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | ==================
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | WARNING: ThreadSanitizer: data race (pid=1990)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | Write of size 4 at 0x7f23e939ced8 by thread T2:
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #0 evsig_set_base_ /builds/worker/workspace/build/src/ipc/chromium/src/third_party/libevent/signal.c:125:16 (libxul.so+0x126b7cf)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #1 event_base_loop /builds/worker/workspace/build/src/ipc/chromium/src/third_party/libevent/event.c:1901:3 (libxul.so+0x1264514)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #2 base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) /builds/worker/workspace/build/src/ipc/chromium/src/base/message_pump_libevent.cc (libxul.so+0x1245ca9)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #3 RunInternal /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:315:10 (libxul.so+0x1242fdc)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #4 RunHandler /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:308:3 (libxul.so+0x1242fdc)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #5 MessageLoop::Run() /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:290:3 (libxul.so+0x1242fdc)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #6 base::Thread::ThreadMain() /builds/worker/workspace/build/src/ipc/chromium/src/base/thread.cc:192:16 (libxul.so+0x12535e3)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #7 ThreadFunc(void*) /builds/worker/workspace/build/src/ipc/chromium/src/base/platform_thread_posix.cc:40:13 (libxul.so+0x124d1be)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | Previous read of size 4 at 0x7f23e939ced8 by main thread:
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | [failed to restore the stack]
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | Location is global 'evsig_base_fd' of size 4 at 0x7f23e939ced8 (libxul.so+0x00000a793ed8)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | Thread T2 'Gecko_IOThread' (tid=1998, running) created by main thread at:
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #0 pthread_create /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:967:3 (firefox+0x571ab)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #1 CreateThread /builds/worker/workspace/build/src/ipc/chromium/src/base/platform_thread_posix.cc:123:14 (libxul.so+0x124a9b7)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #2 PlatformThread::Create(unsigned long, PlatformThread::Delegate*, unsigned long*) /builds/worker/workspace/build/src/ipc/chromium/src/base/platform_thread_posix.cc:134:10 (libxul.so+0x124a9b7)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #3 base::Thread::StartWithOptions(base::Thread::Options const&) /builds/worker/workspace/build/src/ipc/chromium/src/base/thread.cc:97:8 (libxul.so+0x12531c6)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #4 NS_InitXPCOM /builds/worker/workspace/build/src/xpcom/build/XPCOMInit.cpp:316:9 (libxul.so+0xaa2728)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #5 Initialize /builds/worker/workspace/build/src/toolkit/xre/nsAppRunner.cpp:1264:8 (libxul.so+0x632e2dd)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #6 XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/workspace/build/src/toolkit/xre/nsAppRunner.cpp:4697:22 (libxul.so+0x632e2dd)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #7 XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/workspace/build/src/toolkit/xre/nsAppRunner.cpp:4752:21 (libxul.so+0x632e6b4)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #8 mozilla::BootstrapImpl::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/workspace/build/src/toolkit/xre/Bootstrap.cpp:45:12 (libxul.so+0x6337432)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #9 do_main /builds/worker/workspace/build/src/browser/app/nsBrowserApp.cpp:217:22 (firefox+0xc8b33)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | #10 main /builds/worker/workspace/build/src/browser/app/nsBrowserApp.cpp:331:16 (firefox+0xc8b33)
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | SUMMARY: ThreadSanitizer: data race /builds/worker/workspace/build/src/ipc/chromium/src/third_party/libevent/signal.c:125:16 in evsig_set_base_
[task 2020-02-19T02:47:20.882Z] 02:47:20 INFO - GECKO(1990) | ==================
[task 2020-02-19T02:47:21.064Z] 02:47:21 INFO - TEST-INFO | Main app process: killed by SIGIOT
[task 2020-02-19T02:47:21.065Z] 02:47:21 INFO - Buffered messages finished
[task 2020-02-19T02:47:21.066Z] 02:47:21 ERROR - TEST-UNEXPECTED-FAIL | Last test finished | application terminated with exit code -6

Comment 1

[Simon Giesecke [:sg] [he/him]]

This is reported upstream as https://github.com/libevent/libevent/issues/779. There is also a PR with a fix at https://github.com/libevent/libevent/pull/741, but it has not been accepted (yet).

Comment 2

[Intermittent Failures Robot]

1 failures in 4045 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-02-17&endday=2020-02-23&tree=all

Comment 3

[Intermittent Failures Robot]

5 failures in 5336 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-beta: 1
• mozilla-central: 1

Platform breakdown:

• linux64-tsan: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-03-09&endday=2020-03-15&tree=all

Comment 4

[Intermittent Failures Robot]

17 failures in 4955 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 13
• mozilla-central: 3
• try: 1

Platform breakdown:

• linux64-tsan: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-03-16&endday=2020-03-22&tree=all

Comment 5

[Bogdan Tara[:bogdan_tara | bogdant]]

In the last 7 days there have been 20 occurrences on linux 64 tsan opt.

Recent failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=295049632&repo=autoland&lineNumber=3224

Comment 6

[Intermittent Failures Robot]

22 failures in 4773 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 17
• mozilla-central: 5

Platform breakdown:

• linux64-tsan: 12
• linux1804-64-tsan: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-03-23&endday=2020-03-29&tree=all

Comment 7

[Intermittent Failures Robot]

15 failures in 5015 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• try: 2
• mozilla-central: 1

Platform breakdown:

• linux1804-64-tsan: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-03-30&endday=2020-04-05&tree=all

Comment 8

[Intermittent Failures Robot]

16 failures in 4823 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-central: 2
• autoland: 13

Platform breakdown:

• linux1804-64-tsan: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-04-06&endday=2020-04-12&tree=all

Comment 10

[Intermittent Failures Robot]

20 failures in 4528 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 13
• mozilla-central: 4
• try: 3

Platform breakdown:

• linux1804-64-tsan: 19
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-04-13&endday=2020-04-19&tree=all

Comment 11

[Intermittent Failures Robot]

28 failures in 5582 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 21
• mozilla-central: 6
• mozilla-beta: 1

Platform breakdown:

• linux1804-64-tsan: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-04-20&endday=2020-04-26&tree=all

Comment 12

[Andrei Ciure[:aciure]]

There are 26 failures associated to this bug in the last 7 days. These are occurring on linux1804-64-tsan opt builds.

log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=299884943&repo=autoland&lineNumber=2529
[task 2020-04-28T23:18:07.550Z] 23:18:07 INFO - GECKO(2698) | WARNING: ThreadSanitizer: data race (pid=2698)
[task 2020-04-28T23:18:07.551Z] 23:18:07 INFO - GECKO(2698) | Write of size 4 at 0x7f99ff683038 by thread T4:
[task 2020-04-28T23:18:07.551Z] 23:18:07 INFO - GECKO(2698) | #0 evsig_set_base_ /builds/worker/checkouts/gecko/ipc/chromium/src/third_party/libevent/signal.c:125:16 (libxul.so+0x1333c1f)
[task 2020-04-28T23:18:07.551Z] 23:18:07 INFO - GECKO(2698) | #1 event_base_loop /builds/worker/checkouts/gecko/ipc/chromium/src/third_party/libevent/event.c:1901:3 (libxul.so+0x132c994)
[task 2020-04-28T23:18:07.551Z] 23:18:07 INFO - GECKO(2698) | #2 base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_pump_libevent.cc (libxul.so+0x130d609)
[task 2020-04-28T23:18:07.552Z] 23:18:07 INFO - GECKO(2698) | #3 RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:315:10 (libxul.so+0x130a94c)
[task 2020-04-28T23:18:07.553Z] 23:18:07 INFO - GECKO(2698) | #4 RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:308:3 (libxul.so+0x130a94c)
[task 2020-04-28T23:18:07.553Z] 23:18:07 INFO - GECKO(2698) | #5 MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:290:3 (libxul.so+0x130a94c)
[task 2020-04-28T23:18:07.553Z] 23:18:07 INFO - GECKO(2698) | #6 base::Thread::ThreadMain() /builds/worker/checkouts/gecko/ipc/chromium/src/base/thread.cc:192:16 (libxul.so+0x131ace5)
[task 2020-04-28T23:18:07.553Z] 23:18:07 INFO - GECKO(2698) | #7 ThreadFunc(void*) /builds/worker/checkouts/gecko/ipc/chromium/src/base/platform_thread_posix.cc:40:13 (libxul.so+0x1314afe)
[task 2020-04-28T23:18:07.553Z] 23:18:07 INFO - GECKO(2698) | Previous read of size 4 at 0x7f99ff683038 by main thread:
[task 2020-04-28T23:18:07.553Z] 23:18:07 INFO - GECKO(2698) | [failed to restore the stack]
[task 2020-04-28T23:18:07.553Z] 23:18:07 INFO - GECKO(2698) | Location is global 'evsig_base_fd' of size 4 at 0x7f99ff683038 (libxul.so+0x00000ab57038)
[task 2020-04-28T23:18:07.554Z] 23:18:07 INFO - GECKO(2698) | Thread T4 'Gecko_IOThread' (tid=2708, running) created by main thread at:
[task 2020-04-28T23:18:07.554Z] 23:18:07 INFO - GECKO(2698) | #0 pthread_create /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:967:3 (firefox+0x5749b)
[task 2020-04-28T23:18:07.554Z] 23:18:07 INFO - GECKO(2698) | #1 CreateThread /builds/worker/checkouts/gecko/ipc/chromium/src/base/platform_thread_posix.cc:123:14 (libxul.so+0x1312187)
[task 2020-04-28T23:18:07.554Z] 23:18:07 INFO - GECKO(2698) | #2 PlatformThread::Create(unsigned long, PlatformThread::Delegate*, unsigned long*) /builds/worker/checkouts/gecko/ipc/chromium/src/base/platform_thread_posix.cc:134:10 (libxul.so+0x1312187)
[task 2020-04-28T23:18:07.554Z] 23:18:07 INFO - GECKO(2698) | #3 base::Thread::StartWithOptions(base::Thread::Options const&) /builds/worker/checkouts/gecko/ipc/chromium/src/base/thread.cc:97:8 (libxul.so+0x131a8e1)
[task 2020-04-28T23:18:07.554Z] 23:18:07 INFO - GECKO(2698) | #4 NS_InitXPCOM /builds/worker/checkouts/gecko/xpcom/build/XPCOMInit.cpp:317:9 (libxul.so+0xb1114d)
[task 2020-04-28T23:18:07.555Z] 23:18:07 INFO - GECKO(2698) | #5 ScopedXPCOMStartup::Initialize(bool) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:1311:8 (libxul.so+0x638dcfd)
[task 2020-04-28T23:18:07.556Z] 23:18:07 INFO - GECKO(2698) | #6 XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:4739:22 (libxul.so+0x63968b3)
[task 2020-04-28T23:18:07.557Z] 23:18:07 INFO - GECKO(2698) | #7 XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:4797:21 (libxul.so+0x6396c24)
[task 2020-04-28T23:18:07.557Z] 23:18:07 INFO - GECKO(2698) | #8 mozilla::BootstrapImpl::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/Bootstrap.cpp:45:12 (libxul.so+0x63a1762)
[task 2020-04-28T23:18:07.558Z] 23:18:07 INFO - GECKO(2698) | #9 do_main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:217:22 (firefox+0xc93e3)
[task 2020-04-28T23:18:07.558Z] 23:18:07 INFO - GECKO(2698) | #10 main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:331:16 (firefox+0xc93e3)
[task 2020-04-28T23:18:07.558Z] 23:18:07 INFO - GECKO(2698) | SUMMARY: ThreadSanitizer: data race /builds/worker/checkouts/gecko/ipc/chromium/src/third_party/libevent/signal.c:125:16 in evsig_set_base_
[task 2020-04-28T23:18:07.559Z] 23:18:07 INFO - GECKO(2698) | ==================
[task 2020-04-28T23:18:07.735Z] 23:18:07 INFO - TEST-INFO | Main app process: killed by SIGIOT
[task 2020-04-28T23:18:07.735Z] 23:18:07 INFO - Buffered messages finished
[task 2020-04-28T23:18:07.736Z] 23:18:07 ERROR - TEST-UNEXPECTED-FAIL | Last test finished | application terminated with exit code -6
[task 2020-04-28T23:18:07.736Z] 23:18:07 INFO - runtests.py | Application ran for: 0:00:46.715597
[task 2020-04-28T23:18:07.736Z] 23:18:07 INFO - zombiecheck | Reading PID log: /tmp/tmpENcMNrpidlog
[task 2020-04-28T23:18:07.736Z] 23:18:07 INFO - ==> process 2698 launched child process 2713
[task 2020-04-28T23:18:07.736Z] 23:18:07 INFO - ==> process 2698 launched child process 2823
[task 2020-04-28T23:18:07.737Z] 23:18:07 INFO - ==> process 2698 launched child process 2843
[task 2020-04-28T23:18:07.737Z] 23:18:07 INFO - ==> process 2698 launched child process 2919
[task 2020-04-28T23:18:07.737Z] 23:18:07 INFO - ==> process 2698 launched child process 2948
[task 2020-04-28T23:18:07.737Z] 23:18:07 INFO - zombiecheck | Checking for orphan process with PID: 2713
[task 2020-04-28T23:18:07.737Z] 23:18:07 INFO - zombiecheck | Checking for orphan process with PID: 2919
[task 2020-04-28T23:18:07.737Z] 23:18:07 INFO - zombiecheck | Checking for orphan process with PID: 2843
[task 2020-04-28T23:18:07.738Z] 23:18:07 INFO - zombiecheck | Checking for orphan process with PID: 2948
[task 2020-04-28T23:18:07.738Z] 23:18:07 INFO - zombiecheck | Checking for orphan process with PID: 2823
[task 2020-04-28T23:18:07.738Z] 23:18:07 INFO - Stopping web server

:jld can you take a look?

Comment 13

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

There's a fundamentally hard problem here, which seems to be why upstream doesn't have a fix: if an async signal handler is using some kind of resource, like a fd or heap memory, it's not enough to unset the signal handler before freeing the resource; it's necessary to ensure that the handler is no longer running on any other thread, and locks can't be used for this.

We may not need to care about that: the fd is owned by the event_base (the abstraction of the native I/O multiplexer) and doesn't appear to be closed until it's destroyed, which for IPC is during shutdown. (Specifically, after ShutdownXPCOM deletes the BrowserProcessSubThread and sends a ThreadQuitTask to the I/O thread.)

But then there's what we're being yelled at about here: the fd is stored into the global variable, then the signal handler is installed, then — according to TSan — it's possible for another thread on another CPU to run the signal handler before it sees the change to the non-atomic global. For that, a change proposed in one of the GitHub issues should suffice: adding the C11 _Atomic qualifier.

Comment 14

[Intermittent Failures Robot]

26 failures in 5423 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 22
• mozilla-central: 3
• mozilla-release: 1

Platform breakdown:

• linux1804-64-tsan: 26

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-04-27&endday=2020-05-03&tree=all

Comment 15

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I have a patch, and it doesn't appear to break the build.

Comment 16

[Intermittent Failures Robot]

20 failures in 5276 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 15
• mozilla-beta: 2
• try: 1

Platform breakdown:

• linux1804-64-tsan: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-05-04&endday=2020-05-10&tree=all

Comment 17

[Intermittent Failures Robot]

11 failures in 5488 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 2
• mozilla-beta: 1

Platform breakdown:

• linux1804-64-tsan: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 18

[Intermittent Failures Robot]

10 failures in 4727 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 7
• mozilla-beta: 1

Platform breakdown:

• linux1804-64-tsan: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 19

[Intermittent Failures Robot]

1 failures in 5458 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-05-25&endday=2020-05-31&tree=all

Comment 20

[Intermittent Failures Robot]

3 failures in 4949 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• mozilla-beta: 1
• autoland: 1

Platform breakdown:

• linux1804-64-tsan: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-06-01&endday=2020-06-07&tree=all

Comment 21

[Intermittent Failures Robot]

2 failures in 5211 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux1804-64-tsan: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-06-08&endday=2020-06-14&tree=all

Comment 22

[Intermittent Failures Robot]

2 failures in 4035 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux1804-64-tsan: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-06-15&endday=2020-06-21&tree=all

Comment 23

[Intermittent Failures Robot]

4 failures in 5252 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 3

Platform breakdown:

• linux1804-64-tsan: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-06-22&endday=2020-06-28&tree=all

Comment 24

[Intermittent Failures Robot]

5 failures in 4516 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 2
• autoland: 2
• try: 1

Platform breakdown:

• linux1804-64-tsan: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-06-29&endday=2020-07-05&tree=all

Comment 25

[Intermittent Failures Robot]

5 failures in 5265 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-beta: 1

Platform breakdown:

• linux1804-64-tsan: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-07-06&endday=2020-07-12&tree=all

Comment 26

[Intermittent Failures Robot]

6 failures in 4014 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-esr78: 1
• try: 1

Platform breakdown:

• linux1804-64-tsan: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-07-13&endday=2020-07-19&tree=all

Comment 27

[Intermittent Failures Robot]

2 failures in 4310 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux1804-64-tsan: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-07-20&endday=2020-07-26&tree=all

Comment 28

[Intermittent Failures Robot]

1 failures in 4289 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-07-27&endday=2020-08-02&tree=all

Comment 29

[Simon Giesecke [:sg] [he/him]]

Jed, in comment #15 you mentioned you had a patch to fix this, but it's not attached to this bug. What's the state of that?

Comment 30

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Attachment: Bug 1616462 - Make event_base_fd atomic to avoid a TSan error.

See comment #13 on the bug for an explanation of what this fixes, what it
doesn't fix, and why the upstream bug report about the larger issue here
has stalled.

Comment 31

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Simon Giesecke [:sg] [he/him] from comment #29)

Jed, in comment #15 you mentioned you had a patch to fix this, but it's not attached to this bug. What's the state of that?

Thanks for the reminder.

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c574dd7d46ef
Make event_base_fd atomic to avoid a TSan error. r=froydnj

Comment 33

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/c574dd7d46ef

Comment 34

[Andreea Pavel [:apavel]]

Still shows up after the fix landed on central: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=312141776&repo=mozilla-central&lineNumber=2259

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&revision=297a47c209fae11e550bd0c52513fed00d9b0b2d&selectedTaskRun=ApOeK2e3QQmj3rUfh28a4g.0

Comment 35

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Slightly different: it's complaining about evsig_base, not evsig_base_fd.

There's… a lot going on here. If evsig_base_fd were loaded first in the signal handler, because it's stored last in the code that updates the globals, it would act as a barrier for the accesses to evsig_base. But those globals are set every time a signal handlers registered and every time we enter the event loop (which ought to be just once, but that's the source of the write that's called out in the TSan report).

So, I could keep trying to outwit TSan, or I could rewrite the code in IPC that handles child process termination on Unix, which is overdue for some cleanup, to not use this feature of libevent.

Comment 36

[Andreea Pavel [:apavel]]

(In reply to Jed Davis [:jld] ⟨⏰|UTC-6⟩ ⟦he/him⟧ from comment #35)

Slightly different: it's complaining about evsig_base, not evsig_base_fd.

There's… a lot going on here. If evsig_base_fd were loaded first in the signal handler, because it's stored last in the code that updates the globals, it would act as a barrier for the accesses to evsig_base. But those globals are set every time a signal handlers registered and every time we enter the event loop (which ought to be just once, but that's the source of the write that's called out in the TSan report).

So, I could keep trying to outwit TSan, or I could rewrite the code in IPC that handles child process termination on Unix, which is overdue for some cleanup, to not use this feature of libevent.

Should we file a different bug or can we use this one?

Comment 37

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Andreea Pavel [:apavel] from comment #36)

Should we file a different bug or can we use this one?

I'll file another bug and have it block this one.

Comment 38

[Atila Butkovits]

Backed out as requested.

Backout link: https://hg.mozilla.org/integration/autoland/rev/a606d54431cffc3d91cc628b3da8f2ec3d975134

Comment 39

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout merged: https://hg.mozilla.org/mozilla-central/rev/a606d54431cf

Comment 40

[Intermittent Failures Robot]

3 failures in 4618 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2
• mozilla-central: 1

Platform breakdown:

• linux1804-64-tsan: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-08-03&endday=2020-08-09&tree=all

Comment 41

[Intermittent Failures Robot]

2 failures in 3601 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux1804-64-tsan: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-08-10&endday=2020-08-16&tree=all

Comment 42

[Intermittent Failures Robot]

1 failures in 4100 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-08-17&endday=2020-08-23&tree=all

Comment 43

[Intermittent Failures Robot]

1 failures in 4046 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-08-24&endday=2020-08-30&tree=all

Comment 44

[Intermittent Failures Robot]

1 failures in 2984 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-08-31&endday=2020-09-06&tree=all

Comment 45

[Intermittent Failures Robot]

1 failures in 4128 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1616462&startday=2020-11-02&endday=2020-11-08&tree=all

Comment 46

[Intermittent Failures Robot]

1 failures in 3712 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1616462&startday=2020-11-30&endday=2020-12-06&tree=all

Comment 47

[Intermittent Failures Robot]

1 failures in 3421 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 1

Platform breakdown:

• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1616462&startday=2020-12-14&endday=2020-12-20&tree=all

Comment 48

[Intermittent Failures Robot]

1 failures in 525 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1616462&startday=2020-12-28&endday=2021-01-03&tree=all

Comment 49

[Intermittent Failures Robot]

1 failures in 3966 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-tsan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1616462&startday=2021-02-01&endday=2021-02-07&tree=all

Comment 50

[Intermittent Failures Robot]

2 failures in 4322 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• linux1804-64-tsan: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1616462&startday=2021-02-08&endday=2021-02-14&tree=all

Comment 51

[Intermittent Failures Robot]

1 failures in 4622 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-tsan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1616462&startday=2021-03-08&endday=2021-03-14&tree=all

Comment 52

[Intermittent Failures Robot]

1 failures in 4257 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-tsan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1616462&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 53

[Intermittent Failures Robot]

1 failures in 3854 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-tsan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1616462&startday=2021-04-05&endday=2021-04-11&tree=all

Comment 54

[Intermittent Failures Robot]

1 failures in 4255 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-tsan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1616462&startday=2021-04-12&endday=2021-04-18&tree=all

Comment 55

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

As far as we know the bug is still present, even if TSan runs aren't finding it anymore for some reason, but bug 1658072 will take care of it.