1445441 - Intermittent LeakSanitizer | leak at Malloc, nsTArray_base, AppendElements, AppendElements

Status: RESOLVED FIXED

(Core :: XPCOM, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=167732055&repo=mozilla-central

https://queue.taskcluster.net/v1/task/NvwIx6dES1KXL5w3oQxo1g/runs/0/artifacts/public/logs/live_backing.log

[task 2018-03-13T18:26:30.343Z] 18:26:30     INFO - GECKO(1054) | ###!!! [Child][MessageChannel] Error: (msgtype=0x2C0104,name=PContent::Msg_BHRThreadHang) Closed channel: cannot send/recv
[task 2018-03-13T18:26:30.380Z] 18:26:30     INFO - GECKO(1054) | -----------------------------------------------------
[task 2018-03-13T18:26:30.380Z] 18:26:30     INFO - GECKO(1054) | Suppressions used:
[task 2018-03-13T18:26:30.380Z] 18:26:30     INFO - GECKO(1054) |   count      bytes template
[task 2018-03-13T18:26:30.381Z] 18:26:30     INFO - GECKO(1054) |     670      21360 nsComponentManagerImpl
[task 2018-03-13T18:26:30.381Z] 18:26:30     INFO - GECKO(1054) |       8       1664 mozJSComponentLoader::LoadModule
[task 2018-03-13T18:26:30.382Z] 18:26:30     INFO - GECKO(1054) |     611      17713 libfontconfig.so
[task 2018-03-13T18:26:30.383Z] 18:26:30     INFO - GECKO(1054) |       1         29 libglib-2.0.so
[task 2018-03-13T18:26:30.387Z] 18:26:30     INFO - GECKO(1054) | -----------------------------------------------------
[task 2018-03-13T18:26:30.491Z] 18:26:30     INFO - GECKO(1054) | ###!!! [Child][MessageChannel] Error: (msgtype=0x2C0104,name=PContent::Msg_BHRThreadHang) Closed channel: cannot send/recv
[task 2018-03-13T18:26:32.882Z] 18:26:32     INFO - GECKO(1054) | -----------------------------------------------------
[task 2018-03-13T18:26:32.882Z] 18:26:32     INFO - GECKO(1054) | Suppressions used:
[task 2018-03-13T18:26:32.886Z] 18:26:32     INFO - GECKO(1054) |   count      bytes template
[task 2018-03-13T18:26:32.887Z] 18:26:32     INFO - GECKO(1054) |     682      21712 nsComponentManagerImpl
[task 2018-03-13T18:26:32.888Z] 18:26:32     INFO - GECKO(1054) |      11       2288 mozJSComponentLoader::LoadModule
[task 2018-03-13T18:26:32.889Z] 18:26:32     INFO - GECKO(1054) |     611      17713 libfontconfig.so
[task 2018-03-13T18:26:32.889Z] 18:26:32     INFO - GECKO(1054) |       2         88 _PR_Getfd
[task 2018-03-13T18:26:32.890Z] 18:26:32     INFO - GECKO(1054) |       1         29 libglib-2.0.so
[task 2018-03-13T18:26:32.892Z] 18:26:32     INFO - GECKO(1054) | -----------------------------------------------------
[task 2018-03-13T18:26:33.603Z] 18:26:33     INFO - GECKO(1054) | -----------------------------------------------------
[task 2018-03-13T18:26:33.604Z] 18:26:33     INFO - GECKO(1054) | Suppressions used:
[task 2018-03-13T18:26:33.605Z] 18:26:33     INFO - GECKO(1054) |   count      bytes template
[task 2018-03-13T18:26:33.606Z] 18:26:33     INFO - GECKO(1054) |     667      21272 nsComponentManagerImpl
[task 2018-03-13T18:26:33.606Z] 18:26:33     INFO - GECKO(1054) |      10       2080 mozJSComponentLoader::LoadModule
[task 2018-03-13T18:26:33.606Z] 18:26:33     INFO - GECKO(1054) |     611      17713 libfontconfig.so
[task 2018-03-13T18:26:33.606Z] 18:26:33     INFO - GECKO(1054) |       1         29 libglib-2.0.so
[task 2018-03-13T18:26:33.610Z] 18:26:33     INFO - GECKO(1054) | -----------------------------------------------------
[task 2018-03-13T18:26:36.972Z] 18:26:36     INFO - GECKO(1054) | =================================================================
[task 2018-03-13T18:26:36.973Z] 18:26:36    ERROR - GECKO(1054) | ==1054==ERROR: LeakSanitizer: detected memory leaks
[task 2018-03-13T18:26:36.980Z] 18:26:36     INFO - GECKO(1054) | Direct leak of 144 byte(s) in 1 object(s) allocated from:
[task 2018-03-13T18:26:36.982Z] 18:26:36     INFO - GECKO(1054) |     #0 0x4c1c93 in malloc /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:88:3
[task 2018-03-13T18:26:36.983Z] 18:26:36     INFO - GECKO(1054) |     #1 0x4f26fd in moz_xmalloc /builds/worker/workspace/build/src/memory/mozalloc/mozalloc.cpp:70:17
[task 2018-03-13T18:26:36.985Z] 18:26:36     INFO - GECKO(1054) |     #2 0x7fa2b9255028 in operator new /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/mozalloc.h:156:12
[task 2018-03-13T18:26:36.986Z] 18:26:36     INFO - GECKO(1054) |     #3 0x7fa2b9255028 in mozilla::BackgroundHangThread::ReportHang(unsigned int) /builds/worker/workspace/build/src/toolkit/components/backgroundhangmonitor/BackgroundHangMonitor.cpp:507
[task 2018-03-13T18:26:36.988Z] 18:26:36     INFO - GECKO(1054) |     #4 0x7fa2b9253cb7 in mozilla::BackgroundHangManager::RunMonitorThread() /builds/worker/workspace/build/src/toolkit/components/backgroundhangmonitor/BackgroundHangMonitor.cpp:403:26
[task 2018-03-13T18:26:36.989Z] 18:26:36     INFO - GECKO(1054) |     #5 0x7fa2b92533e3 in mozilla::BackgroundHangManager::MonitorThread(void*) /builds/worker/workspace/build/src/toolkit/components/backgroundhangmonitor/BackgroundHangMonitor.cpp:79:49
[task 2018-03-13T18:26:36.991Z] 18:26:36     INFO - GECKO(1054) |     #6 0x7fa2ca3d647e in _pt_root /builds/worker/workspace/build/src/nsprpub/pr/src/pthreads/ptthread.c:201:5
[task 2018-03-13T18:26:36.994Z] 18:26:36     INFO - GECKO(1054) |     #7 0x7fa2ce7516b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)

Comment 1

[Intermittent Failures Robot]

15 failures in 814 pushes (0.018 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 7
* mozilla-central: 6
* autoland: 2

Platform breakdown:
* linux64: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1445441&startday=2018-03-12&endday=2018-03-18&tree=all

Comment 2

[Intermittent Failures Robot]

15 failures in 822 pushes (0.018 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 6
* mozilla-central: 4
* autoland: 4
* try: 1

Platform breakdown:
* linux64: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-03-20&endday=2018-03-26&tree=trunk

Comment 3

[Intermittent Failures Robot]

12 failures in 718 pushes (0.017 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 5
* mozilla-central: 4
* mozilla-inbound: 2
* try: 1

Platform breakdown:
* linux64: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-03-26&endday=2018-04-01&tree=trunk

Comment 4

[Intermittent Failures Robot]

13 failures in 765 pushes (0.017 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 5
* mozilla-inbound: 4
* autoland: 3
* try: 1

Platform breakdown:
* linux64: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-04-02&endday=2018-04-08&tree=trunk

Comment 5

[Intermittent Failures Robot]

4 failures in 792 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 2
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* linux64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-04-09&endday=2018-04-15&tree=trunk

Comment 6

[Intermittent Failures Robot]

20 failures in 782 pushes (0.026 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 12
* autoland: 5
* mozilla-central: 2
* try: 1

Platform breakdown:
* linux64: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-04-16&endday=2018-04-22&tree=trunk

Comment 7

[Intermittent Failures Robot]

18 failures in 780 pushes (0.023 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 7
* autoland: 7
* mozilla-central: 4

Platform breakdown:
* linux64: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-04-23&endday=2018-04-29&tree=trunk

Comment 8

[Intermittent Failures Robot]

23 failures in 551 pushes (0.042 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 14
* autoland: 7
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-04-30&endday=2018-05-06&tree=trunk

Comment 9

[Intermittent Failures Robot]

17 failures in 660 pushes (0.026 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 7
* autoland: 7
* mozilla-inbound: 3

Platform breakdown:
* linux64: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-05-07&endday=2018-05-13&tree=trunk

Comment 10

[Intermittent Failures Robot]

30 failures in 639 pushes (0.047 failures/push) were associated with this bug in the last 7 days. 

This is the #47 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 13
* mozilla-central: 10
* autoland: 7

Platform breakdown:
* linux64: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-05-14&endday=2018-05-20&tree=trunk

Comment 11

[Noemi Erli[:noemi_erli]]

There are 38 failures in the past 7 days, and all occurrences are on linux64 asan.
Recent log failure: https://treeherder.mozilla.org/logviewer.html#?job_id=179526568&repo=autoland&lineNumber=3570

Relevant part of the log: 
[task 2018-05-21T17:04:24.737Z] 17:04:24     INFO - GECKO(1049) | -----------------------------------------------------
[task 2018-05-21T17:04:24.739Z] 17:04:24     INFO - GECKO(1049) | Suppressions used:
[task 2018-05-21T17:04:24.740Z] 17:04:24     INFO - GECKO(1049) |   count      bytes template
[task 2018-05-21T17:04:24.741Z] 17:04:24     INFO - GECKO(1049) |     681      21608 nsComponentManagerImpl
[task 2018-05-21T17:04:24.742Z] 17:04:24     INFO - GECKO(1049) |       9       1872 mozJSComponentLoader::LoadModule
[task 2018-05-21T17:04:24.743Z] 17:04:24     INFO - GECKO(1049) |     611      17713 libfontconfig.so
[task 2018-05-21T17:04:24.745Z] 17:04:24     INFO - GECKO(1049) |       2         88 _PR_Getfd
[task 2018-05-21T17:04:24.746Z] 17:04:24     INFO - GECKO(1049) |       1         29 libglib-2.0.so
[task 2018-05-21T17:04:24.747Z] 17:04:24     INFO - GECKO(1049) | -----------------------------------------------------
[task 2018-05-21T17:04:28.866Z] 17:04:28     INFO - GECKO(1049) | =================================================================
[task 2018-05-21T17:04:28.868Z] 17:04:28    ERROR - GECKO(1049) | ==1049==ERROR: LeakSanitizer: detected memory leaks
[task 2018-05-21T17:04:28.874Z] 17:04:28     INFO - GECKO(1049) | Direct leak of 144 byte(s) in 1 object(s) allocated from:
[task 2018-05-21T17:04:28.875Z] 17:04:28     INFO - GECKO(1049) |     #0 0x4c1c93 in malloc /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:88:3
[task 2018-05-21T17:04:28.875Z] 17:04:28     INFO - GECKO(1049) |     #1 0x4f26fd in moz_xmalloc /builds/worker/workspace/build/src/memory/mozalloc/mozalloc.cpp:70:17
[task 2018-05-21T17:04:28.876Z] 17:04:28     INFO - GECKO(1049) |     #2 0x7f9e7b432948 in operator new /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/mozalloc.h:156:12
[task 2018-05-21T17:04:28.877Z] 17:04:28     INFO - GECKO(1049) |     #3 0x7f9e7b432948 in mozilla::BackgroundHangThread::ReportHang(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>) /builds/worker/workspace/build/src/toolkit/components/backgroundhangmonitor/BackgroundHangMonitor.cpp:512
[task 2018-05-21T17:04:28.877Z] 17:04:28     INFO - GECKO(1049) |     #4 0x7f9e7b431508 in ReportPermaHang /builds/worker/workspace/build/src/toolkit/components/backgroundhangmonitor/BackgroundHangMonitor.cpp:546:3
[task 2018-05-21T17:04:28.878Z] 17:04:28     INFO - GECKO(1049) |     #5 0x7f9e7b431508 in mozilla::BackgroundHangManager::RunMonitorThread() /builds/worker/workspace/build/src/toolkit/components/backgroundhangmonitor/BackgroundHangMonitor.cpp:377
[task 2018-05-21T17:04:28.879Z] 17:04:28     INFO - GECKO(1049) |     #6 0x7f9e7b430c23 in mozilla::BackgroundHangManager::MonitorThread(void*) /builds/worker/workspace/build/src/toolkit/components/backgroundhangmonitor/BackgroundHangMonitor.cpp:79:49
[task 2018-05-21T17:04:28.887Z] 17:04:28     INFO - GECKO(1049) |     #7 0x7f9e8cfd647e in _pt_root /builds/worker/workspace/build/src/nsprpub/pr/src/pthreads/ptthread.c:201:5
[task 2018-05-21T17:04:28.888Z] 17:04:28     INFO - GECKO(1049) |     #8 0x7f9e913196b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)

:jimm can you please take a look here?

Comment 12

[Jim Mathies [:jimm]]

noted, thanks.

Comment 13

[Intermittent Failures Robot]

37 failures in 632 pushes (0.059 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 13
* try: 9
* mozilla-inbound: 9
* mozilla-central: 6

Platform breakdown:
* linux64: 37

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-05-21&endday=2018-05-27&tree=trunk

Comment 14

[Intermittent Failures Robot]

24 failures in 655 pushes (0.037 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 13
* mozilla-inbound: 9
* mozilla-central: 2

Platform breakdown:
* linux64: 24

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-05-28&endday=2018-06-03&tree=trunk

Comment 15

[Bogdan Tara[:bogdan_tara | bogdant]]

30 failures in the last 7 days, all on Linu 64, asan.

Comment 16

[Kris Maglione [:kmag]]

(In reply to Bogdan Tara[:bogdan_tara] from comment #15)
> 30 failures in the last 7 days, all on Linu 64, asan.

LeakSanitizer failures only happen on ASan, and we currenlty only run ASan on Linux64, so this is not surprising.

Comment 17

[Andrew McCreight [:mccr8]]

The allocation of the leaked object happens here:
    nsCOMPtr<nsIRunnable> processHangStackRunnable =
      new ProcessHangStackRunnable(std::move(hangDetails));
The runnable is then passed to nsThread::Dispatch(), which intentionally leaks its argument when it fails, so presumably that is happening here (there's NS_WARNING logging for that, but it won't show up in LSan). This is happening because in general, destroying a runnable may release mainthread-only objects. Of course, in this particular case it does not.

I don't see any intermittent leaks being reported for ProcessHangStackRunnable, which is odd.

Comment 18

[Andrew McCreight [:mccr8]]

Nathan, is there any good way we could make this leak go away? I can't find any variant of dispatch that doesn't leak the object. Maybe we could disable BHR on ASan, or increase the limit so that it doesn't get hit as much.

Comment 19

[Andrew McCreight [:mccr8]]

"BHR disabled for debug builds because of bug 979069."

Well, that explains why we don't see this in debug builds. (from toolkit/components/backgroundhangmonitor/moz.build)

I guess we should just disable it in ASan builds, too....

Comment 20

[Intermittent Failures Robot]

36 failures in 669 pushes (0.054 failures/push) were associated with this bug in the last 7 days. 

This is the #35 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 20
* mozilla-inbound: 8
* mozilla-central: 7
* try: 1

Platform breakdown:
* linux64: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-06-04&endday=2018-06-10&tree=trunk

Comment 21

[Intermittent Failures Robot]

18 failures in 408 pushes (0.044 failures/push) were associated with this bug in the last 7 days. 

This is the #37 most frequent failure this week.   

Repository breakdown:
* mozilla-inbound: 11
* autoland: 4
* mozilla-central: 2
* try: 1

Platform breakdown:
* linux64: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-06-11&endday=2018-06-17&tree=trunk

Comment 22

[Andrew McCreight [:mccr8]]

I'll close this if there are no more failures in a few days.

Comment 23

[Andrew McCreight [:mccr8]]

Looks fixed.

Comment 24

[Intermittent Failures Robot]

11 failures in 521 pushes (0.021 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 5
* autoland: 4
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445441&startday=2018-06-18&endday=2018-06-24&tree=trunk