1636624 - Intermittent leakcheck | tab 464 bytes leaked (BackstagePass, XPCNativeInterface, XPCNativeMember, XPCNativeSet, XPCWrappedNative, ...)

Status: RESOLVED FIXED

(Core :: XPConnect, defect)

Description

[Treeherder Bug Filer]

Filed by: nbeleuzu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=301486955&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZJgzotFwSTKax0odO6oH7g/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-05-08T23:27:02.156Z] 23:27:02 INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 7393
[task 2020-05-08T23:27:02.156Z] 23:27:02 INFO -
[task 2020-05-08T23:27:02.157Z] 23:27:02 INFO - |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2020-05-08T23:27:02.158Z] 23:27:02 INFO - | | Per-Inst Leaked| Total Rem|
[task 2020-05-08T23:27:02.158Z] 23:27:02 INFO - 0 |TOTAL | 41 464| 66335 11|
[task 2020-05-08T23:27:02.159Z] 23:27:02 INFO - 24 |BackstagePass
[task 2020-05-08T23:27:02.159Z] 23:27:02 INFO - 534 |XPCNativeInterface | 56 56| 45 1|
[task 2020-05-08T23:27:02.160Z] 23:27:02 INFO - 535 |XPCNativeMember | 16 16| 2027 1|
[task 2020-05-08T23:27:02.161Z] 23:27:02 INFO - 536 |XPCNativeSet | 32 32| 59 1|
[task 2020-05-08T23:27:02.161Z] 23:27:02 INFO - 539 |XPCWrappedNative | 96 96| 143 1|
[task 2020-05-08T23:27:02.162Z] 23:27:02 INFO - 540 |XPCWrappedNativeProto | 40 40| 42 1|
[task 2020-05-08T23:27:02.163Z] 23:27:02 INFO - 542 |XPCWrappedNativeTearOff | 32 32| 190 1|
[task 2020-05-08T23:27:02.163Z] 23:27:02 INFO - 721 |nsJSPrincipals | 24 24| 119 1|
[task 2020-05-08T23:27:02.167Z] 23:27:02 INFO - 819 |nsTArray_base | 8 24| 24317 3|
[task 2020-05-08T23:27:02.167Z] 23:27:02 INFO -
[task 2020-05-08T23:27:02.168Z] 23:27:02 INFO - nsTraceRefcnt::DumpStatistics: 859 entries
[task 2020-05-08T23:27:02.169Z] 23:27:02 INFO - TEST-INFO | leakcheck | tab leaked 1 BackstagePass
[task 2020-05-08T23:27:02.169Z] 23:27:02 INFO - TEST-INFO | leakcheck | tab leaked 1 XPCNativeInterface
[task 2020-05-08T23:27:02.170Z] 23:27:02 INFO - TEST-INFO | leakcheck | tab leaked 1 XPCNativeMember
[task 2020-05-08T23:27:02.171Z] 23:27:02 INFO - TEST-INFO | leakcheck | tab leaked 1 XPCNativeSet
[task 2020-05-08T23:27:02.171Z] 23:27:02 INFO - TEST-INFO | leakcheck | tab leaked 1 XPCWrappedNative
[task 2020-05-08T23:27:02.172Z] 23:27:02 INFO - TEST-INFO | leakcheck | tab leaked 1 XPCWrappedNativeProto
[task 2020-05-08T23:27:02.172Z] 23:27:02 INFO - TEST-INFO | leakcheck | tab leaked 1 XPCWrappedNativeTearOff
[task 2020-05-08T23:27:02.173Z] 23:27:02 INFO - TEST-INFO | leakcheck | tab leaked 1 nsJSPrincipals
[task 2020-05-08T23:27:02.173Z] 23:27:02 INFO - TEST-INFO | leakcheck | tab leaked 3 nsTArray_base
[task 2020-05-08T23:27:02.174Z] 23:27:02 INFO - TEST-UNEXPECTED-FAIL | leakcheck | tab 464 bytes leaked (BackstagePass, XPCNativeInterface, XPCNativeMember, XPCNativeSet, XPCWrappedNative, ...)
[task 2020-05-08T23:27:02.175Z] 23:27:02 INFO -
[task 2020-05-08T23:27:02.175Z] 23:27:02 INFO - leakcheck | Processing leak log file /tmp/tmpoeHkXt.mozrunner/runtests_leaks_tab_pid7155.log
[task 2020-05-08T23:27:02.176Z] 23:27:02 INFO -
[task 2020-05-08T23:27:02.177Z] 23:27:02 INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 7155
[task 2020-05-08T23:27:02.177Z] 23:27:02 INFO -
[task 2020-05-08T23:27:02.178Z] 23:27:02 INFO - |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2020-05-08T23:27:02.178Z] 23:27:02 INFO - | | Per-Inst Leaked| Total Rem|
[task 2020-05-08T23:27:02.179Z] 23:27:02 INFO - 0 |TOTAL | 36 0| 161681 0|
[task 2020-05-08T23:27:02.180Z] 23:27:02 INFO -
[task 2020-05-08T23:27:02.180Z] 23:27:02 INFO - nsTraceRefcnt::DumpStatistics: 829 entries

Comment 1

[Andrew McCreight [:mccr8]]

This looks similar to a leak I fixed before. It seems suspicious that we hit this leak 2 out of 4 times on the landing of bug 1595046, and that was the first reappearance of that, particular given that this was known to have some possible leakiness.

Comment 2

[Tom Schuster [:evilpie]]

Definitely seems suspicious. It seems like this leak is not associated with any particular test? What do I need to run, ./mach test browser/components/privatebrowsing/test/browser ?

Comment 3

[Andrew McCreight [:mccr8]]

This happens in shutdown. Basically, the JSM loader global (I think that's the main backstage pass we have) has some associated data structures that are cyclic. If something else is holding that alive until after the final shutdown CC we do in debug builds, then we get this leak. But yes, you'll have to run the directory, or maybe the last one or two tests we have in that directory.

It looks like I confused the related leak with another leak. For the one I linked, it was due to CycleCollectedJSContext::mPendingException not being cleared when we returned to the event loop. https://bugzilla.mozilla.org/show_bug.cgi?id=1612364#c9

Comment 4

[Andrew McCreight [:mccr8]]

You might have better luck reproducing it if you run with --enable-fission and --verify. That will give you many more process shutdowns to try to hit the leak with. It looks like most, but not all, failures are in Fission.

Comment 5

[Tom Schuster [:evilpie]]

Seems like we create a lot of nsScriptErrorWithStack instances without an inner-window-id. Actually something like 95% of created instances in one of my test runs don't have such an ID. This can only happen when |win| in CreateScriptError is null. Maybe as a compromise we could set the exception value to Nothing when we have no window? This would still let us have a stack in these cases.

Comment 6

[Intermittent Failures Robot]

19 failures in 5276 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 18
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 16
• macosx1014-64: 2
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636624&startday=2020-05-04&endday=2020-05-10&tree=all

Comment 7

[Tom Schuster [:evilpie]]

Attachment: Bug 1636590 - Update devtools page stubs. r?nchevobbe

Comment 8

[Phabricator Automation]

Comment on attachment 9147180 [details]
Bug 1636590 - Update devtools page stubs. r?nchevobbe

Revision D74603 was moved to bug 1636590. Setting attachment 9147180 [details] to obsolete.

Comment 9

[Tom Schuster [:evilpie]]

So we are basically back to the reason why I had originally also added aInnerWindow to CreateScriptError:

• AsyncErrorReporter::Run doesn't have a window. But we do have an exception and stack that we want to actually keep. We do however have xpc::ErrorReport so we could use that to get the inner-window-id.
• In other places we could just pass win->WindowID().

The basically reverse approach would be to pass Nothing for the exception parameters in nsXPCComponents_Utils::ReportError (when there is no win) and here in AutoJSAPI::ReportException.

Comment 10

[Intermittent Failures Robot]

27 failures in 997 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 25
• mozilla-central: 2

Platform breakdown:

• linux1804-64: 27

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636624&startday=2020-05-11&endday=2020-05-11&tree=all

Comment 11

[Release mgmt bot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1595046

Comment 12

[Intermittent Failures Robot]

34 failures in 1049 pushes (0.032 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 34

Platform breakdown:

• linux1804-64: 31
• linux1804-64-qr: 2
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636624&startday=2020-05-12&endday=2020-05-12&tree=all

Comment 13

[Tom Schuster [:evilpie]]

Attachment: Bug 1636624 - Mitigate leaks involving exceptions with CreateSciptError. r?mccr8

Doing this kind of censoring in "CreateScriptError" turns out to be more effort
than I thought. We don't have a JSContext to create Rooted instances.
This still catches all the relavent places where we creating script-errors without inner-window-ids.

Comment 14

[Tom Schuster [:evilpie]]

With this patch I can't reproduce the leak anymore in about double the time where it reproduced before.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fd0b50a41a86
Mitigate leaks involving exceptions with CreateSciptError. r=mccr8

Comment 16

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/fd0b50a41a86

Comment 17

[Intermittent Failures Robot]

70 failures in 5488 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #17 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 65
• mozilla-central: 2
• try: 3

Platform breakdown:

• linux1804-64: 66
• linux1804-64-qr: 2
• macosx1014-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636624&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 18

[Intermittent Failures Robot]

3 failures in 4727 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 3

Platform breakdown:

• linux32: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636624&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 19

[Intermittent Failures Robot]

4 failures in 5458 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 4

Platform breakdown:

• linux32: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636624&startday=2020-05-25&endday=2020-05-31&tree=all

Comment 20

[Andrew McCreight [:mccr8]]

The esr68 failures are likely actually bug 1612364, which affected that branch but I never backported the fix (unlike this bug which does not affect esr68).