1551825 - Intermittent TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, js::Nursery::allocateBuffer, js::Nursery::reallocateBuffer in accessible/tests/browser/events/

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=246550523&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/Z0TXtEU6SKSjOOYj5tuGbQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2019-05-15T09:39:48.044Z] 09:39:48 INFO - TEST-INFO | LeakSanitizer | To show the addresses of leaked objects add report_objects=1 to LSAN_OPTIONS
[task 2019-05-15T09:39:48.044Z] 09:39:48 INFO - TEST-INFO | LeakSanitizer | This can be done in testing/mozbase/mozrunner/mozrunner/utils.py
[task 2019-05-15T09:39:48.044Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, AllocChars, JSRope::flattenInternal
[task 2019-05-15T09:39:48.044Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at createTable, mozilla::detail::HashTable, reserve, reserve
[task 2019-05-15T09:39:48.044Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at createTable, mozilla::detail::HashTable, rehashIfOverloaded, infallibleRehashIfOverloaded
[task 2019-05-15T09:39:48.045Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, js::Nursery::allocateBuffer, js::Nursery::reallocateBuffer
[task 2019-05-15T09:39:48.046Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, js::TenuringTracer::moveSlotsToTenured, movePlainObjectToTenured
[task 2019-05-15T09:39:48.047Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, AllocateObjectBuffer, js::NativeObject::growElements
[task 2019-05-15T09:39:48.048Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, js::TypeNewScript::make, js::ObjectGroup::defaultNewGroup
[task 2019-05-15T09:39:48.048Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, pod_malloc, js::ObjectGroup::newPlainObject
[task 2019-05-15T09:39:48.053Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at EntrySlotOrCreate, EntrySlotOrCreate, mozilla::dom::TelemetryStopwatch_Binding::CreateInterfaceObjects, mozilla::dom::GetPerInterfaceObjectHandle
[task 2019-05-15T09:39:48.053Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at EntrySlotOrCreate, EntrySlotOrCreate, mozilla::dom::EventTarget_Binding::CreateInterfaceObjects, mozilla::dom::GetPerInterfaceObjectHandle
[task 2019-05-15T09:39:48.055Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at EntrySlotOrCreate, EntrySlotOrCreate, mozilla::dom::MatchPatternSet_Binding::CreateInterfaceObjects, mozilla::dom::GetPerInterfaceObjectHandle
[task 2019-05-15T09:39:48.055Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, mozilla::UniquePtr, js::FunctionScope::clone
[task 2019-05-15T09:39:48.056Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, pod_malloc, js::frontend::CreateScriptSourceObject
[task 2019-05-15T09:39:48.057Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::dom::Performance::CreateForMainThread, nsPIDOMWindowInner::CreatePerformanceObjectIfNeeded, nsPIDOMWindowInner::GetPerformance, mozilla::dom::Window_Binding::get_performance
[task 2019-05-15T09:39:48.058Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at createTable, mozilla::detail::HashTable, js::Shape::removeChild, SweepThing
[task 2019-05-15T09:39:48.059Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, tryNewTenuredObject, js::AllocateObject
[task 2019-05-15T09:39:48.059Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, pod_malloc, js::ForOfPIC::createForOfPICObject
[task 2019-05-15T09:39:48.060Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_realloc, maybe_pod_realloc, js::Nursery::reallocateBuffer, ReallocateObjectBuffer
[task 2019-05-15T09:39:48.060Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_calloc, maybe_pod_calloc, pod_calloc, js::ShapeIC::init
[task 2019-05-15T09:39:48.061Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at js_pod_arena_malloc, maybe_pod_malloc, pod_malloc, js::ObjectGroup::allocationSiteGroup
[task 2019-05-15T09:39:48.062Z] 09:39:48 ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, nsTSubstring, nsTSubstring

last test run: accessible/tests/browser/events/browser_test_textcaret.js

Comment 1

[Bogdan Tara[:bogdan_tara | bogdant]]

Started r/b on range: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux%2Cx64%2Casan%2Cmochitests%2Ctest-linux64-asan%2Fopt-mochitest-browser-chrome-e10s-3%2Cm%28bc3%29&fromchange=556e4f98beeb3442f8d588be05ae6afdaff44d13&tochange=b8c66ae8e149ff7dd0be76a815ffb43060ba482a

The test is switching chunks quite often.

Comment 2

[Intermittent Failures Robot]

53 failures in 907 pushes (0.058 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 48
• mozilla-inbound: 4
• mozilla-beta: 1

Platform breakdown:

• linux64: 52
• android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1551825&startday=2019-05-15&endday=2019-05-15&tree=all

Comment 3

[Intermittent Failures Robot]

37 failures in 887 pushes (0.042 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 26
• mozilla-inbound: 6
• mozilla-central: 4
• mozilla-beta: 1

Platform breakdown:

• linux64: 37

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1551825&startday=2019-05-16&endday=2019-05-16&tree=all

Comment 4

[Bogdan Tara[:bogdan_tara | bogdant]]

Continued r/b up to: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux%2Cx64%2Casan%2Cmochitests%2Ctest-linux64-asan%2Fopt-mochitest-browser-chrome-e10s&fromchange=90ff166702b5dd7e9848eef62746457b11d8fe92&tochange=556e4f98beeb3442f8d588be05ae6afdaff44d13&selectedJob=246513308

Comment 5

[Dorel Luca [:dluca]]

Rettrigers shows that it likely started here :https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux%2Cx64%2Casan%2Cmochitests%2Ctest-linux64-asan%2Fopt-mochitest-browser-chrome-e10s-3%2Cm%28bc3%29&tochange=b8c66ae8e149ff7dd0be76a815ffb43060ba482a&fromchange=61f3f19ee0de3b05060506337f8fb71a08c2e07b&selectedJob=247202701 with the push of Bug 1550783 : https://hg.mozilla.org/integration/autoland/rev/3b3fc85fd247f3beb4a446a3b80da22e3cf0e703

Comment 6

[Intermittent Failures Robot]

105 failures in 4658 pushes (0.023 failures/push) were associated with this bug in the last 7 days.

This is the #15 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 82
• mozilla-inbound: 13
• mozilla-beta: 2
• mozilla-central: 6
• try: 2

Platform breakdown:

• linux64: 104
• android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1551825&startday=2019-05-13&endday=2019-05-19&tree=all

Comment 9

[Andrew McCreight [:mccr8]]

[Tracking Requested - why for this release]: extremely frequent large leak regression

Comment 10

[Andreea Pavel [:apavel]]

Updated the summary so it's shown in treeherder as a suggestion

Comment 11

[Andreea Pavel [:apavel]]

(In reply to Andreea Pavel [:apavel] from comment #10)

Updated the summary so it's shown in treeherder as a suggestion

Sorry Andrew, seems like bugzilla messed up the tracking flags.

Comment 12

[Andrew McCreight [:mccr8]]

[Tracking Requested - why for this release]:

Comment 13

[Andrew McCreight [:mccr8]]

Thanks for finding the regression range, Bodgan.

John, can you take a look? I can help with some of the basic investigation.

Comment 14

[Andrew McCreight [:mccr8]]

Well, that regression range doesn't really make any sense. Those are some test only changes, and not involving the test that leaked, so I think it only affected chunking.

Comment 15

[Andrew McCreight [:mccr8]]

It looks like the leak was in bc11 before bug 1550783: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux%2Cx64%2Casan%2Cmochitests%2Ctest-linux64-asan%2Fopt-mochitest-browser-chrome&revision=6eea4e18512a82b11c1f8d49a2c084ffecf325ff&selectedJob=246543390

Comment 16

[Andrew McCreight [:mccr8]]

Here's a tighter regression range: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux%2Cx64%2Casan%2Cmochitests%2Ctest-linux64-asan%2Fopt-mochitest-browser-chrome-e10s-&tochange=9c251c872a3370203295a8196c970195abc3bcca&fromchange=1db216f33d399e886e6d2e425a5e5e1019c5c704

Though note that somewhere in there accessible/tests/browser/events/ switched from bc11 (and the end of the range) to bc6.

edit: I updated the link to include all BC, due to the chunk switch in there somewhere.

Comment 17

[Andrew McCreight [:mccr8]]

I added some retriggers to try to figure out what is to blame, but I did notice that bug 1190882 is an a11y change in that range.

Comment 18

[Andrew McCreight [:mccr8]]

Well, coalescing or something is throwing out my attempts to do retriggers, but I think the range is narrow enough to suggest that bug 1190882 is to blame. In addition to changing a11y code, it added a new test to the directory that is leaking.

James, can you take a look? If this can't be fixed quickly, it should be backed out, as it seems to be leaking almost all of the time (or maybe all of the time). Thanks.

Comment 19

[James Teh [:Jamie]]

Eitan graciously offered to look into this for me. Eitan, please bounce NI back to me if you can't get to this.

Comment 20

[Intermittent Failures Robot]

25 failures in 996 pushes (0.025 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 23
• mozilla-central: 1
• mozilla-beta: 1

Platform breakdown:

• linux64: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1551825&startday=2019-05-21&endday=2019-05-21&tree=all

Comment 21

[Eitan Isaacson [:eeejay]]

Didn't do a complete deep dive on this, but managed to reproduce and fix locally. Like Jamie and I discussed, this is probably happening because of a strong reference the accessibility service has of an accessible. In this case it is probably a DocAccessible because it looks like its ShutDown method does not release all of its held references. Specifically in this case it is not releasing mAnchorJumpElm from a previous scroll event test.

Explicitly releasing it, (and mInvalidationList for good measure) fixes the issue locally, so I think it is worth pushing that change and seeing if it remedies this.

Comment 22

[Eitan Isaacson [:eeejay]]

Here is a green try build (i know its intermittent, but maybe this is helpful?)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c162fde8eed2a7d15b2637a24bd7ef9a89e3fa74

Comment 23

[Eitan Isaacson [:eeejay]]

Attachment: Bug 1551825 - Release mAnchorJumpElm and mAnchorJumpElm in DocAccessible::Shutdown. r?Jamie!

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cd3ba329d43a
Release mAnchorJumpElm and mInvalidationList in DocAccessible::Shutdown. r=Jamie

Comment 25

[Intermittent Failures Robot]

17 failures in 801 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 14
• mozilla-beta: 2
• mozilla-inbound: 1

Platform breakdown:

• linux64: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1551825&startday=2019-05-22&endday=2019-05-22&tree=all

Comment 26

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/cd3ba329d43a

Comment 27

[Intermittent Failures Robot]

23 failures in 808 pushes (0.028 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-release: 1
• mozilla-inbound: 6
• mozilla-central: 1
• try: 14
• mozilla-beta: 1

Platform breakdown:

• linux64: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1551825&startday=2019-05-23&endday=2019-05-23&tree=all

Comment 28

[Razvan Maries]

New occurrence: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=248403983&repo=mozilla-central&lineNumber=15121

Comment 29

[Andrew McCreight [:mccr8]]

(In reply to Razvan Maries from comment #28)

New occurrence: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=248403983&repo=mozilla-central&lineNumber=15121

That failure is in web platform tests, in the /mixed-content/worklet-audio-top-level/http-csp/same-host-https/top-level/keep-scheme-redirect/ directory, so it looks like a different leak. Please file a new bug for it.

The original leak looks fixed. Eitan, can you nominate for uplift to beta 68? Thanks.

Comment 30

[Intermittent Failures Robot]

87 failures in 4730 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #21 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• try: 33
• autoland: 37
• mozilla-central: 2
• mozilla-beta: 7
• mozilla-inbound: 7
• mozilla-release: 1

Platform breakdown:

• linux64: 77
• android-em-4-3-armv7-api16: 3
• linux64-qr: 1
• windows10-64: 4
• linux64-shippable-qr: 1
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1551825&startday=2019-05-20&endday=2019-05-26&tree=all

Comment 31

[Eitan Isaacson [:eeejay]]

Comment on attachment 9066827 [details]
Bug 1551825 - Release mAnchorJumpElm and mAnchorJumpElm in DocAccessible::Shutdown. r?Jamie!

Beta/Release Uplift Approval Request

• User impact if declined: Potential memory leaks
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): We just added an extra cleanup step to DocAccessible::Shutdown.
• String changes made/needed:

Comment 32

[Julien Cristau [:jcristau]]

Comment on attachment 9066827 [details]
Bug 1551825 - Release mAnchorJumpElm and mAnchorJumpElm in DocAccessible::Shutdown. r?Jamie!

a11y memleak fix, approved for 68.0b6

Comment 33

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/releases/mozilla-beta/rev/2aeaee73c3a4

Comment 34

[Intermittent Failures Robot]

4 failures in 4337 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• mozilla-beta: 3

Platform breakdown:

• windows10-64-nightly: 1
• linux64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1551825&startday=2019-05-27&endday=2019-06-02&tree=all