1718960 - Intermittent [tier 2] LeakSanitizer | leak at alloc::alloc::alloc::h2f060a4097bf3f0c, alloc::alloc::Global::alloc_impl::h8194b5bf11a2401b, _$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$::allocate::hddd3472725329b92, alloc::raw_vec::Raw...

Status: RESOLVED INCOMPLETE

(Core :: Internationalization, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: imoraru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=344227017&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PPMWFEWvSLy9RM1_Lk4DTA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-07-02T10:31:38.134Z] 10:31:38     INFO - TEST-START | browser/base/content/test/about/browser_bug633691.js
[task 2021-07-02T10:31:39.339Z] 10:31:39     INFO - GECKO(1928) | MEMORY STAT | vsize 20976021MB | residentFast 1452MB
[task 2021-07-02T10:31:39.346Z] 10:31:39     INFO - TEST-OK | browser/base/content/test/about/browser_bug633691.js | took 1207ms
[task 2021-07-02T10:31:39.410Z] 10:31:39     INFO - checking window state
[task 2021-07-02T10:31:39.525Z] 10:31:39     INFO - GECKO(1928) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-07-02T10:31:39.547Z] 10:31:39     INFO - GECKO(1928) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-07-02T10:31:40.092Z] 10:31:40     INFO - GECKO(1928) | -----------------------------------------------------
[task 2021-07-02T10:31:40.093Z] 10:31:40     INFO - GECKO(1928) | Suppressions used:
[task 2021-07-02T10:31:40.094Z] 10:31:40     INFO - GECKO(1928) |   count      bytes template
[task 2021-07-02T10:31:40.095Z] 10:31:40     INFO - GECKO(1928) |      14        448 nsComponentManagerImpl
[task 2021-07-02T10:31:40.096Z] 10:31:40     INFO - GECKO(1928) |       2        288 libfontconfig.so
[task 2021-07-02T10:31:40.097Z] 10:31:40     INFO - GECKO(1928) | -----------------------------------------------------
[task 2021-07-02T10:31:43.422Z] 10:31:43     INFO - GECKO(1928) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-07-02T10:31:43.443Z] 10:31:43     INFO - GECKO(1928) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-07-02T10:31:44.504Z] 10:31:44     INFO - GECKO(1928) | Completed ShutdownLeaks collections in process 2089
<...>
[task 2021-07-02T10:32:04.572Z] 10:32:04     INFO - GECKO(1928) | -----------------------------------------------------
[task 2021-07-02T10:32:04.573Z] 10:32:04     INFO - GECKO(1928) | Suppressions used:
[task 2021-07-02T10:32:04.573Z] 10:32:04     INFO - GECKO(1928) |   count      bytes template
[task 2021-07-02T10:32:04.573Z] 10:32:04     INFO - GECKO(1928) |      14        432 nsComponentManagerImpl
[task 2021-07-02T10:32:04.577Z] 10:32:04     INFO - GECKO(1928) |     633      18083 libfontconfig.so
[task 2021-07-02T10:32:04.577Z] 10:32:04     INFO - GECKO(1928) |       4      31872 style::sharing::SHARING_CACHE_KEY
[task 2021-07-02T10:32:04.578Z] 10:32:04     INFO - GECKO(1928) |       4      16448 style::bloom::BLOOM_KEY
[task 2021-07-02T10:32:04.578Z] 10:32:04     INFO - GECKO(1928) |       2        768 libcairo.so
[task 2021-07-02T10:32:04.579Z] 10:32:04     INFO - GECKO(1928) |       2      16393 libglib-2.0.so
[task 2021-07-02T10:32:04.579Z] 10:32:04     INFO - GECKO(1928) |       4       2080 libpixman-1.so
[task 2021-07-02T10:32:04.580Z] 10:32:04     INFO - GECKO(1928) |       3        624 mozJSComponentLoader
[task 2021-07-02T10:32:04.580Z] 10:32:04     INFO - GECKO(1928) | -----------------------------------------------------
[task 2021-07-02T10:32:04.581Z] 10:32:04     INFO - TEST-INFO | Main app process: exit 0
[task 2021-07-02T10:32:04.581Z] 10:32:04     INFO - TEST-INFO | LeakSanitizer | To show the addresses of leaked objects add report_objects=1 to LSAN_OPTIONS
[task 2021-07-02T10:32:04.582Z] 10:32:04     INFO - TEST-INFO | LeakSanitizer | This can be done in testing/mozbase/mozrunner/mozrunner/utils.py
[task 2021-07-02T10:32:04.582Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at alloc::alloc::alloc::h2f060a4097bf3f0c, alloc::alloc::Global::alloc_impl::h8194b5bf11a2401b, _$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$::allocate::hddd3472725329b92, alloc::raw_vec::RawVec$LT$T$C$A$GT$::allocate_in::he075aeedbef9ca6f
[task 2021-07-02T10:32:04.583Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, nsTSubstring, NonVoidStringToJsval
[task 2021-07-02T10:32:04.583Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at alloc::alloc::realloc::hecd6ab118fbbab66, _$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$::shrink::hefae2870413abc16, alloc::raw_vec::RawVec$LT$T$C$A$GT$::shrink::h6d6f9c551641856c, alloc::raw_vec::RawVec$LT$T$C$A$GT$::shrink_to_fit::hdd6b88aabd9ac198
[task 2021-07-02T10:32:04.584Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_arena_malloc, pod_arena_malloc, pod_malloc
[task 2021-07-02T10:32:04.584Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::detail::HashTable, mozilla::detail::HashTable, mozilla::detail::HashTable, reserve
[task 2021-07-02T10:32:04.585Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at alloc::alloc::alloc::h2f060a4097bf3f0c, alloc::alloc::Global::alloc_impl::h8194b5bf11a2401b, _$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$::allocate::hddd3472725329b92, alloc::alloc::exchange_malloc::h9f41935042b7bcd7
[task 2021-07-02T10:32:04.585Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at alloc::alloc::alloc::h2f060a4097bf3f0c, alloc::alloc::Global::alloc_impl::h8194b5bf11a2401b, _$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$::allocate::hddd3472725329b92, hashbrown::raw::alloc::inner::do_alloc::he84808bf576fc0ca
[task 2021-07-02T10:32:04.587Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::detail::HashTable, mozilla::detail::HashTable, reserve, reserve
[task 2021-07-02T10:32:04.588Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at alloc::raw_vec::finish_grow::h025e34b30a11e2dd, alloc::raw_vec::RawVec$LT$T$C$A$GT$::grow_amortized::hb6666a3bb4964990, alloc::raw_vec::RawVec$LT$T$C$A$GT$::reserve::do_reserve_and_handle::hd93854fe8319fe7c, alloc::raw_vec::RawVec$LT$T$C$A$GT$::reserve::h92d5791e2cf717f8
[task 2021-07-02T10:32:04.588Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at AllocateProtoAndIfaceCache, mozilla::dom::CreateGlobal, mozilla::dom::Window_Binding::Wrap, CreateNativeGlobalForInner
[task 2021-07-02T10:32:04.589Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::intl::FluentBundle::Constructor, mozilla::dom::FluentBundle_Binding::_constructor, CallJSNative, CallJSNativeConstructor
[task 2021-07-02T10:32:04.589Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_arena_malloc, pod_arena_malloc, AllocChars
[task 2021-07-02T10:32:04.590Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_realloc, maybe_pod_arena_realloc, pod_realloc, js::Nursery::reallocateBuffer
[task 2021-07-02T10:32:04.590Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_new, MakeUnique, js::SharedPropMap::addChild, js::SharedPropMap::addPropertyInternal
[task 2021-07-02T10:32:04.591Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at alloc::alloc::alloc::h2f060a4097bf3f0c, alloc::alloc::Global::alloc_impl::h8194b5bf11a2401b, _$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$::allocate::hddd3472725329b92, alloc::raw_vec::RawVec$LT$T$C$A$GT$::allocate_in::h25e93d58f20b80bd
[task 2021-07-02T10:32:04.591Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_arena_malloc, js::detail::OrderedHashTable, pod_malloc
[task 2021-07-02T10:32:04.591Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::detail::HashTable, mozilla::detail::HashTable, putNew, putNew
[task 2021-07-02T10:32:04.592Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_arena_malloc, pod_arena_malloc, make_pod_arena_array
[task 2021-07-02T10:32:04.592Z] 10:32:04    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_arena_malloc, maybe_pod_malloc, js::gc::GCRuntime::tryNewTenuredObject
[task 2021-07-02T10:32:04.593Z] 10:32:04     INFO - runtests.py | Application ran for: 0:04:24.747070

Comment 1

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Core::JavaScript: GC' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Intermittent Failures Robot]

5 failures in 723 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 3

Platform and build breakdown:

• linux1804-64-asan: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1718960&startday=2021-06-28&endday=2021-07-04&tree=all

Comment 3

[Intermittent Failures Robot]

16 failures in 410 pushes (0.039 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16

Platform and build breakdown:

• linux1804-64-asan: 12
  • opt: 12
• linux1804-64-asan-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1718960&startday=2021-07-05&endday=2021-07-05&tree=all

Comment 4

[Intermittent Failures Robot]

15 failures in 510 pushes (0.029 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 14
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-asan: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1718960&startday=2021-07-07&endday=2021-07-07&tree=all

Comment 5

[Alexandru Michis [:malexandru]]

There have been 65 total failures in the last 7 days (recent failure log).
Affected platforms are linux1804-64-asan and linux1804-64-asan-qr.

Steven, could you please take a look?

Comment 6

[Intermittent Failures Robot]

67 failures in 2387 pushes (0.028 failures/push) were associated with this bug in the last 7 days.

This is the #11 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 51
• mozilla-central: 16

Platform and build breakdown:

• linux1804-64-asan: 59
  • opt: 59
• linux1804-64-asan-qr: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1718960&startday=2021-07-05&endday=2021-07-11&tree=all

Comment 7

[Steven DeTar [:sdetar]]

Nicolas, do you have an input on this bug or how to attempt to find the root cause.

Comment 8

[Nicolas B. Pierron [:nbp] {away}]

The orange factor link of the bug show that this bug first appeared on autoland at c471c8c , and on mozilla-central at c72cd8a on the 2021-07-02. Based on the frequency after these first cases, the bug is unlikely to have been there for more than 1 or 2 days (in normal days) but our CI has been under used on the lasy week of June so we should we should look at commit since 2021-06-25.

Looking at the frequencies of orange factors, and the number of pushes in the last week of June / first week of July.
We have a likelyhood of seeing the bug appear per push of 3.37% (96.63% to not have any report). Then, we can compute the likelyhood that the repository contains a bug at a given push in the push-log by raising 96.63% with the number of pushes since c471c8c [1]

Seeing mozilla::intl::FluentBundle::Constructor in the log, I wonder if this issue might not be related to Bug 1715979. (likelyhood 35.7%) or Bug 1284868 (likelyhood 26.26%)
Otherwise the only GC change in the guessed range seems to be Bug 1703443 (likelyhood 0.9%)

Greg, André, would these leaked data structure sounds familiar to you?

[1] Adding the following CSS rules is better than counting by hand:

table {
    counter-reset: row-num;
}
tr.parity0 + tr.parity1 {
    counter-increment: row-num;
}
tr.parity1 + tr.parity0 {
    counter-increment: row-num;
}
td:first-child::before {
    content: counter(row-num) ". ";
}

Comment 9

[André Bargull [:anba]]

Bug 1284868 only changes which if-statement ICU is taking in this code. So it doesn't seem likely that this change is causing any memory leaks, especially because that ICU code doesn't allocate any memory. :-)

Comment 10

[Intermittent Failures Robot]

10 failures in 2224 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-asan: 5
  • opt: 5
• linux1804-64-asan-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1718960&startday=2021-07-12&endday=2021-07-18&tree=all

Comment 11

[Greg Tatum [:gregtatum]]

Bug 1715979 shouldn't be related to the FluentBundle constructor. It doesn't invoke the DateTimePatternGenerator.

I touched some functions that the constructor may call recently when unifying the DateTimeFormat API in Bug 1705156, but that was awhile back, which is suspicious if this hadn't increased then.

I am suspicious that it's the DateTimeFormat operation on the Fluent side of things, as the cert error page that is being run here does display a date.

Comment 12

[Greg Tatum [:gregtatum]]

The Fluent constructor can create a mozilla::intl::DateTimeFormat here:
https://searchfox.org/mozilla-central/rev/70a94bffbb73d2b0ba751fb3905428fdbcd4d402/intl/l10n/rust/fluent-ffi/src/bundle.rs#125-135

The date operation here is being invoked probably from here using Intl.DateTimeFormat, these look unrelated though. The values are being passed in as arguments.

https://searchfox.org/mozilla-central/rev/70a94bffbb73d2b0ba751fb3905428fdbcd4d402/browser/base/content/certerror/aboutNetError.js#24
https://searchfox.org/mozilla-central/rev/70a94bffbb73d2b0ba751fb3905428fdbcd4d402/browser/base/content/certerror/aboutNetError.js#1227

In summary, I'm not really feeling confident on if this caused by my recent changes.

Comment 13

[Nicolas B. Pierron [:nbp] {away}]

I guess the only remaining way to find out would be to bisect on autoland which of the pushes introduce this failure.

Comment 14

[Intermittent Failures Robot]

4 failures in 2940 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 3

Platform and build breakdown:

• linux1804-64-asan-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1718960&startday=2021-07-19&endday=2021-07-25&tree=all

Comment 15

[Intermittent Failures Robot]

5 failures in 3131 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1718960&startday=2021-07-26&endday=2021-08-01&tree=all

Comment 16

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 17

[Intermittent Failures Robot]

1 failures in 2845 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr91: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1718960&startday=2021-10-25&endday=2021-10-31&tree=all

Comment 18

[Intermittent Failures Robot]

1 failures in 3263 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr91: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1718960&startday=2021-12-13&endday=2021-12-19&tree=all