Closed Bug 1760018 Opened 3 years ago Closed 2 years ago

Intermittent leakcheck (linux debug, dom/security/test/csp/) | default 230504 bytes leaked (BackstagePass, BrowsingContext, BrowsingContextGroup, BrowsingContextWebProgress, CanonicalBrowsingContext, ...)

Categories

(Core :: Networking, defect, P5)

defect

Tracking

()

RESOLVED FIXED
114 Branch
Tracking Status
firefox114 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: mccr8)

References

Details

(Keywords: intermittent-failure, Whiteboard: [domsecurity-backlog][stockwell disable-recommended])

Attachments

(1 file)

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=371377343&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/brzNPjFDTmCNSNWhqPtMKw/runs/0/artifacts/public/logs/live_backing.log


[task 2022-03-17T12:21:36.006Z] 12:21:36     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, default process 5321
[task 2022-03-17T12:21:36.007Z] 12:21:36     INFO - 
[task 2022-03-17T12:21:36.008Z] 12:21:36     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2022-03-17T12:21:36.009Z] 12:21:36     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2022-03-17T12:21:36.010Z] 12:21:36     INFO -    0 |TOTAL                                 |       61     5712| 5731364       33|
[task 2022-03-17T12:21:36.011Z] 12:21:36     INFO -   88 |BackstagePass                         |      160      160|       1        1|
[task 2022-03-17T12:21:36.012Z] 12:21:36     INFO -  103 |BrowsingContext                       |     1160     1160|      42        1|
[task 2022-03-17T12:21:36.013Z] 12:21:36     INFO -  104 |BrowsingContextGroup                  |      280      280|      57        1|
[task 2022-03-17T12:21:36.014Z] 12:21:36     INFO -  105 |BrowsingContextWebProgress            |      144      144|      30        1|
[task 2022-03-17T12:21:36.015Z] 12:21:36     INFO -  146 |CanonicalBrowsingContext              |     1360     1360|      42        1|
[task 2022-03-17T12:21:36.015Z] 12:21:36     INFO -  161 |ChildSHistory                         |      176      176|      30        1|
[task 2022-03-17T12:21:36.016Z] 12:21:36     INFO -  197 |CondVar                               |       80      240|     679        3|
[task 2022-03-17T12:21:36.017Z] 12:21:36     INFO -  527 |JS Object                             |        8        8|       1        1|
[task 2022-03-17T12:21:36.018Z] 12:21:36     INFO -  614 |Mutex                                 |       72      288|    8731        4|
[task 2022-03-17T12:21:36.019Z] 12:21:36     INFO - 1255 |ThreadEventTarget                     |       64       64|     120        1|
[task 2022-03-17T12:21:36.020Z] 12:21:36     INFO - 1261 |ThreadTargetSink                      |       16       16|     120        1|
[task 2022-03-17T12:21:36.021Z] 12:21:36     INFO - 1264 |ThrottledEventQueue                   |       40       80|     126        2|
[task 2022-03-17T12:21:36.021Z] 12:21:36     INFO - 1265 |ThrottledEventQueue::Inner            |      272      544|     126        2|
[task 2022-03-17T12:21:36.022Z] 12:21:36     INFO - 1400 |XPCNativeInterface                    |       56       56|     344        1|
[task 2022-03-17T12:21:36.022Z] 12:21:36     INFO - 1401 |XPCNativeMember                       |       16       16|   11686        1|
[task 2022-03-17T12:21:36.023Z] 12:21:36     INFO - 1402 |XPCNativeSet                          |       32       32|     887        1|
[task 2022-03-17T12:21:36.023Z] 12:21:36     INFO - 1405 |XPCWrappedNative                      |       96       96|    7660        1|
[task 2022-03-17T12:21:36.024Z] 12:21:36     INFO - 1406 |XPCWrappedNativeProto                 |       40       40|     563        1|
[task 2022-03-17T12:21:36.026Z] 12:21:36     INFO - 1408 |XPCWrappedNativeTearOff               |       32       32|    9144        1|
[task 2022-03-17T12:21:36.026Z] 12:21:36     INFO - 1611 |nsAuthURLParser                       |       24       24|       2        1|
[task 2022-03-17T12:21:36.027Z] 12:21:36     INFO - 1837 |nsJSPrincipals                        |       24       24|    4928        1|
[task 2022-03-17T12:21:36.027Z] 12:21:36     INFO - 1932 |nsSecureBrowserUI                     |       56       56|      30        1|
[task 2022-03-17T12:21:36.028Z] 12:21:36     INFO - 1952 |nsStandardURL                         |      336      336|   11002        1|
[task 2022-03-17T12:21:36.030Z] 12:21:36     INFO - 1961 |nsStringBuffer                        |        8       16|  192015        2|
[task 2022-03-17T12:21:36.030Z] 12:21:36     INFO - 2003 |nsThread                              |      464      464|     164        1|
[task 2022-03-17T12:21:36.031Z] 12:21:36     INFO - 
[task 2022-03-17T12:21:36.032Z] 12:21:36     INFO - nsTraceRefcnt::DumpStatistics: 2120 entries
[task 2022-03-17T12:21:36.032Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 BackstagePass
[task 2022-03-17T12:21:36.033Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 BrowsingContext
[task 2022-03-17T12:21:36.034Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 BrowsingContextGroup
[task 2022-03-17T12:21:36.035Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 BrowsingContextWebProgress
[task 2022-03-17T12:21:36.035Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 CanonicalBrowsingContext
[task 2022-03-17T12:21:36.036Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 ChildSHistory
[task 2022-03-17T12:21:36.037Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 3 CondVar
[task 2022-03-17T12:21:36.038Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 JS Object
[task 2022-03-17T12:21:36.038Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 4 Mutex
[task 2022-03-17T12:21:36.039Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 ThreadEventTarget
[task 2022-03-17T12:21:36.040Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 ThreadTargetSink
[task 2022-03-17T12:21:36.040Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 2 ThrottledEventQueue
[task 2022-03-17T12:21:36.041Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 2 ThrottledEventQueue::Inner
[task 2022-03-17T12:21:36.042Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 XPCNativeInterface
[task 2022-03-17T12:21:36.042Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 XPCNativeMember
[task 2022-03-17T12:21:36.043Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 XPCNativeSet
[task 2022-03-17T12:21:36.044Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 XPCWrappedNative
[task 2022-03-17T12:21:36.045Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 XPCWrappedNativeProto
[task 2022-03-17T12:21:36.045Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 XPCWrappedNativeTearOff
[task 2022-03-17T12:21:36.046Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 nsAuthURLParser
[task 2022-03-17T12:21:36.047Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 nsJSPrincipals
[task 2022-03-17T12:21:36.047Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 nsSecureBrowserUI
[task 2022-03-17T12:21:36.048Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 nsStandardURL
[task 2022-03-17T12:21:36.049Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 2 nsStringBuffer
[task 2022-03-17T12:21:36.050Z] 12:21:36     INFO - TEST-INFO | leakcheck | default leaked 1 nsThread
[task 2022-03-17T12:21:36.050Z] 12:21:36     INFO - TEST-UNEXPECTED-FAIL | leakcheck | default 5712 bytes leaked (BackstagePass, BrowsingContext, BrowsingContextGroup, BrowsingContextWebProgress, CanonicalBrowsingContext, ...)
[task 2022-03-17T12:21:36.051Z] 12:21:36     INFO - 

Leak while running the dom/security/test/mixedcontentblocker/ tests. Possibly interesting objects in the leak are the ChildSHistory and the nsAuthURLParser.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

Update:

There have been 37 failures within the last 7 days:

  • 4 failures on Linux 18.04 x64 WebRender debug
  • 5 failures on Windows 10 x86 2004 WebRender debug
  • 16 failures on Windows 10 x64 2004 WebRender debug
  • 3 failures on Windows 11 x86 22H2 WebRender debug
  • 9 failures on Windows 11 x64 22H2 WebRender debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=407840786&repo=mozilla-central&lineNumber=118813

[task 2023-03-04T22:56:16.029Z] 22:56:16     INFO - 
[task 2023-03-04T22:56:16.030Z] 22:56:16     INFO - nsTraceRefcnt::DumpStatistics: 2173 entries
[task 2023-03-04T22:56:16.030Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 BackstagePass
[task 2023-03-04T22:56:16.030Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 BrowsingContext
[task 2023-03-04T22:56:16.031Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 BrowsingContextGroup
[task 2023-03-04T22:56:16.031Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 BrowsingContextWebProgress
[task 2023-03-04T22:56:16.031Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 CanonicalBrowsingContext
[task 2023-03-04T22:56:16.032Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 ChildSHistory
[task 2023-03-04T22:56:16.032Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 ComponentModuleLoader
[task 2023-03-04T22:56:16.032Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 3 CondVar
[task 2023-03-04T22:56:16.033Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 ExpirationTrackerObserver
[task 2023-03-04T22:56:16.033Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 JS Object
[task 2023-03-04T22:56:16.033Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 ModuleLoaderBase
[task 2023-03-04T22:56:16.034Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 4 Mutex
[task 2023-03-04T22:56:16.034Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 2 SHEntrySharedParentState
[task 2023-03-04T22:56:16.035Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 2 SessionHistoryEntry
[task 2023-03-04T22:56:16.036Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 ThreadEventTarget
[task 2023-03-04T22:56:16.036Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 ThreadTargetSink
[task 2023-03-04T22:56:16.037Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 2 ThrottledEventQueue
[task 2023-03-04T22:56:16.037Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 2 ThrottledEventQueue::Inner
[task 2023-03-04T22:56:16.038Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 2 WeakReference
[task 2023-03-04T22:56:16.038Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 XPCNativeInterface
[task 2023-03-04T22:56:16.039Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 XPCNativeMember
[task 2023-03-04T22:56:16.039Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 XPCNativeSet
[task 2023-03-04T22:56:16.039Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 XPCWrappedNative
[task 2023-03-04T22:56:16.040Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 XPCWrappedNativeProto
[task 2023-03-04T22:56:16.040Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 XPCWrappedNativeTearOff
[task 2023-03-04T22:56:16.041Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 2 mozilla::dom::HistoryEntryCounterForBr
[task 2023-03-04T22:56:16.041Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 nsAuthURLParser
[task 2023-03-04T22:56:16.042Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 3 nsJSPrincipals
[task 2023-03-04T22:56:16.042Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 nsLayoutHistoryState
[task 2023-03-04T22:56:16.043Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 nsSHistory
[task 2023-03-04T22:56:16.043Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 nsSecureBrowserUI
[task 2023-03-04T22:56:16.044Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 3 nsSimpleURI
[task 2023-03-04T22:56:16.044Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 3 nsStandardURL
[task 2023-03-04T22:56:16.045Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 14 nsStringBuffer
[task 2023-03-04T22:56:16.045Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 1 nsThread
[task 2023-03-04T22:56:16.045Z] 22:56:16     INFO - TEST-INFO | leakcheck | default leaked 2 nsWeakReference
[task 2023-03-04T22:56:16.047Z] 22:56:16     INFO - TEST-UNEXPECTED-FAIL | leakcheck | default 8536 bytes leaked (BackstagePass, BrowsingContext, BrowsingContextGroup, BrowsingContextWebProgress, CanonicalBrowsingContext, ...)
[task 2023-03-04T22:56:16.047Z] 22:56:16     INFO - 

Gian-Carlo, can you help us assign this to someone?
Thank you.

Flags: needinfo?(gpascutto)
Whiteboard: [stockwell needswork:owner]
Component: General → DOM: Security
Flags: needinfo?(gpascutto)

It looks like there was a big spike in this leak on Saturday, mostly happening in mochitest-browser-chrome-3. I looked at four of these bc3 logs, and they were all happening while running tests in the toolkit/components/places/tests/browser/ directory, so I'll move this over there.

Component: DOM: Security → Places
Product: Core → Toolkit

Nothing has changed in the browser.ini for that file since 2021, so it isn't a new test causing the issue.

There was always a parent process leak in the logs I looked at. About half of the time, there was also a content process leak. In 2 such logs I looked at, the leaking process was created while running browser_bug646422.js. In the third, it was created while running browser_bug680727.js. That may or may not mean anything.

It looks like the spike has gone away for some reason. Maybe a patch got backed out for other reasons.

The latest leaks of the last week look like they're in dom/security/test/, but there's some ones earlier in the week that have been misclassified. I'll raise this on #sheriffs.

Component: Places → DOM: Security
Product: Toolkit → Core
See Also: → 1824695

(In reply to Mark Banner (:standard8) from comment #34)

The latest leaks of the last week look like they're in dom/security/test/, but there's some ones earlier in the week that have been misclassified. I'll raise this on #sheriffs.

I was wrong, there's actually two sets of failures here. The toolkit/components/places ones which occurred only on Windows, peaked at the start of the week. The dom/security/test/csp ones were on Linux and have peaked at the end of the week.

I've therefore filed bug 1824695 to try and have that one specific to the places issues, and this one specific to the dom issues.

Summary: Intermittent leakcheck | default 5712 bytes leaked (BackstagePass, BrowsingContext, BrowsingContextGroup, BrowsingContextWebProgress, CanonicalBrowsingContext, ...) → Intermittent leakcheck (linux debug, dom/security/test/csp/) | default 230504 bytes leaked (BackstagePass, BrowsingContext, BrowsingContextGroup, BrowsingContextWebProgress, CanonicalBrowsingContext, ...)
Whiteboard: [stockwell unknown] → [stockwell unknown][domsecurity-backlog
Whiteboard: [stockwell unknown][domsecurity-backlog → [stockwell unknown][domsecurity-backlog]

Update

There have been 38 failures within the last 7 days, all of them on Linux 18.04 x64 WebRender debug.

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=412611282&repo=autoland&lineNumber=8054

[task 2023-04-16T09:12:41.142Z] 09:12:41     INFO - 
[task 2023-04-16T09:12:41.142Z] 09:12:41     INFO - nsTraceRefcnt::DumpStatistics: 2261 entries
[task 2023-04-16T09:12:41.143Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 BackstagePass
[task 2023-04-16T09:12:41.144Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 BrowsingContext
[task 2023-04-16T09:12:41.145Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 BrowsingContextGroup
[task 2023-04-16T09:12:41.145Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 BrowsingContextWebProgress
[task 2023-04-16T09:12:41.146Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 CanonicalBrowsingContext
[task 2023-04-16T09:12:41.147Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 ChildSHistory
[task 2023-04-16T09:12:41.147Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 ComponentModuleLoader
[task 2023-04-16T09:12:41.148Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 3 CondVar
[task 2023-04-16T09:12:41.149Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 ExpirationTrackerObserver
[task 2023-04-16T09:12:41.150Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 JS Object
[task 2023-04-16T09:12:41.150Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 ModuleLoaderBase
[task 2023-04-16T09:12:41.151Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 224 Mutex
[task 2023-04-16T09:12:41.152Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 71 ReferrerInfo
[task 2023-04-16T09:12:41.153Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 74 SHEntrySharedParentState
[task 2023-04-16T09:12:41.153Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 132 SessionHistoryEntry
[task 2023-04-16T09:12:41.154Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 ThreadEventTarget
[task 2023-04-16T09:12:41.155Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 ThreadTargetSink
[task 2023-04-16T09:12:41.156Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 2 ThrottledEventQueue
[task 2023-04-16T09:12:41.156Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 2 ThrottledEventQueue::Inner
[task 2023-04-16T09:12:41.157Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 2 WeakReference
[task 2023-04-16T09:12:41.158Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 XPCNativeInterface
[task 2023-04-16T09:12:41.158Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 XPCNativeMember
[task 2023-04-16T09:12:41.159Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 XPCNativeSet
[task 2023-04-16T09:12:41.159Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 XPCWrappedNative
[task 2023-04-16T09:12:41.160Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 XPCWrappedNativeProto
[task 2023-04-16T09:12:41.161Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 XPCWrappedNativeTearOff
[task 2023-04-16T09:12:41.161Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 26 mozilla::dom::HistoryEntryCounterForBr
[task 2023-04-16T09:12:41.162Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 nsAuthURLParser
[task 2023-04-16T09:12:41.162Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 73 nsCSPContext
[task 2023-04-16T09:12:41.163Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 221 nsJSPrincipals
[task 2023-04-16T09:12:41.164Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 74 nsLayoutHistoryState
[task 2023-04-16T09:12:41.164Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 nsSHistory
[task 2023-04-16T09:12:41.165Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 nsSecureBrowserUI
[task 2023-04-16T09:12:41.165Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 3 nsSimpleNestedURI
[task 2023-04-16T09:12:41.166Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 7 nsSimpleURI
[task 2023-04-16T09:12:41.167Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 439 nsStandardURL
[task 2023-04-16T09:12:41.167Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 567 nsStringBuffer
[task 2023-04-16T09:12:41.168Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 nsThread
[task 2023-04-16T09:12:41.168Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 nsTimer
[task 2023-04-16T09:12:41.169Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 1 nsTimerImpl
[task 2023-04-16T09:12:41.170Z] 09:12:41     INFO - TEST-INFO | leakcheck | default leaked 2 nsWeakReference
[task 2023-04-16T09:12:41.171Z] 09:12:41     INFO - TEST-UNEXPECTED-FAIL | leakcheck | default 230536 bytes leaked (BackstagePass, BrowsingContext, BrowsingContextGroup, BrowsingContextWebProgress, CanonicalBrowsingContext, ...)
[task 2023-04-16T09:12:41.171Z] 09:12:41     INFO - 

Last test successfully run:

[task 2023-04-16T09:12:22.944Z] 09:12:22     INFO - TEST-OK | dom/security/test/csp/test_win_open_blocked.html | took 2625ms
[task 2023-04-16T09:12:23.257Z] 09:12:23     INFO - TEST-START | dom/security/test/csp/test_xslt_inherits_csp.html
[task 2023-04-16T09:12:25.513Z] 09:12:25     INFO - TEST-PASS | dom/security/test/csp/test_xslt_inherits_csp.html | The CSP did block injected JS  
[task 2023-04-16T09:12:25.529Z] 09:12:25     INFO - GECKO(5003) | MEMORY STAT | vsize 10794MB | residentFast 175MB | heapAllocated 14MB
[task 2023-04-16T09:12:25.568Z] 09:12:25     INFO - TEST-OK | dom/security/test/csp/test_xslt_inherits_csp.html | took 2311ms

Mark, is this something you are still looking into?

Flags: needinfo?(standard8)
Whiteboard: [stockwell unknown][domsecurity-backlog] → [domsecurity-backlog][stockwell needswork:owner]

(In reply to Natalia Csoregi [:nataliaCs] from comment #38)

Mark, is this something you are still looking into?

No, these all appear to be dom/security/csp related, hence forwarding to triage owner.

Flags: needinfo?(standard8) → needinfo?(fbraun)

(In reply to Natalia Csoregi [:nataliaCs] from comment #38)

Update

There have been 38 failures within the last 7 days, all of them on Linux 18.04 x64 WebRender debug.

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=412611282&repo=autoland&lineNumber=8054

I really want to help here, but I have no idea where to even start.
From looking at this linked log and scrolling up a bit, towards line 7180 it seems that the test is crashing due to JSWindowActorChild::SendRawMessage issues and IPC Connection Errors.

How sure are we that this issue is actually related to the particular test case?
Are we even concerned about leaks if the process is crashing anyway?

Flags: needinfo?(fbraun)

OK, thanks to Tom Schuster I figured that those crashes are because we leak. Makes sense.
Next step, it seems that for analyzing the leaks we would require GC and CC logs, which this job generated but did not attach to the artifacts. What can I even do here?

Tom says you could help explaining next steps here?

Flags: needinfo?(continuation)

The failures are all in mochitest-plain-http3, which might be interesting. Also only on Linux. It is possible that the leak just shows up slightly differently on other configurations.

We don't get GC/CC logs by default in automation. leakcheck is using a different kind of counter to detect leaks.

The first step would be to get GC/CC logs locally. If that is difficult, you can get them in automation and download the logs, though that's a bit finicky. I can try taking a first look at this later today.

I noticed that a large number of tests in this directory are disabled in http3 is enabled. Maybe there's some odd test ordering issue that shows up in that case.

I managed to get CC+DMD logs for this leak on try.

The object keeping things alive is a BrowsingContext, with one missing strong reference. The missing strong reference looks to either be a reference from the object itself (at a byte offset of 1120 or 1128) or from a PLDHashTable that is a field of the same BrowsingContextGroup that the BrowsingContext has a strong reference to (at a byte offset of 64). The latter would cause a cycle, but mContexts is the only hash table that looks like it would match, and that is cycle collected already, but it isn't showing up in the CC log, so maybe that's a red herring (it could have gotten removed from the hash table but the pointer is still around or something). I'll look some more tomorrow.

The two self-references are from mDeprioritizedLoadRunner due to LinkedList weirdness. They aren't strong and they aren't actually for the BC.

I think the hash table reference is from the BC group of the BC (mContexts). The RefPtr<> dtor releases the BC but does not zero it out, so I suspect that the BC was removed, but a dead pointer was sitting around in memory. The BC group is in the CC graph, and nothing shows up for mContexts, so I think it is actually empty.

Next I'm going to try taking the DMD heap dump during the CC. It is possible that the missing heap reference to the BC went away after the BC, but before the DMD memory dump, which happens very late.

I thought I hadn't been able to reproduce the leak with refcount logging enabled, but I think I accidentally broke the leak checker, and I have actually captured logs for the leaking case.

In the leaking case, we get a release with this stack, after the shutdown CC:

<BrowsingContext> 0x7f7de45f4200 0 Release 3 [thread 0x7f7e2be2cee0]
#01: mozilla::dom::BrowsingContext::Release() [docshell/base/BrowsingContext.cpp:1782]
#02: mozilla::dom::CanonicalBrowsingContext::Release() [docshell/base/CanonicalBrowsingContext.cpp:3054]
#03: mozilla::net::ParentChannelListener::~ParentChannelListener() [netwerk/protocol/http/ParentChannelListener.cpp:49]
#04: mozilla::net::ParentChannelListener::Release() [netwerk/protocol/http/ParentChannelListener.cpp:56]
#05: mozilla::net::HttpBaseChannel::~HttpBaseChannel() [netwerk/protocol/http/HttpBaseChannel.cpp:261]
#06: mozilla::net::nsHttpChannel::~nsHttpChannel() [netwerk/protocol/http/nsHttpChannel.cpp:345]
#07: mozilla::net::nsHttpChannel::~nsHttpChannel() [netwerk/protocol/http/nsHttpChannel.cpp:324]
#08: mozilla::net::HttpBaseChannel::Release() [netwerk/protocol/http/HttpBaseChannel.cpp:404]
#09: {virtual override thunk({offset(-80)}, mozilla::net::nsHttpChannel::Release())} [netwerk/protocol/http/nsHttpChannel.cpp:0]
#10: PLDHashTable::~PLDHashTable() [xpcom/ds/PLDHashTable.cpp:293]
#11: mozilla::net::RedirectChannelRegistrar::Release() [netwerk/base/RedirectChannelRegistrar.cpp:14]
#12: nsNetShutdown() [netwerk/build/nsNetModule.cpp:240]
#13: mozilla::xpcom::StaticComponents::Shutdown()
#14: nsComponentManagerImpl::Shutdown() [xpcom/components/nsComponentManager.cpp:555]
#15: mozilla::ShutdownXPCOM(nsIServiceManager*) [xpcom/build/XPCOMInit.cpp:689]

What this shows is that after the shutdown CC, we shut down static components, including nsNet, which destroys the RedirectChannelRegistrar, which holds alive an HttpBaseChannel, which holds alive a ParentChannelListener, which holds alive our BC.

Then there's another release when we do the special shutdown GC when we are shutting down XPConnect. I think this is from the BC's reflector. This special shutdown GC stops rooting JS objects, which is how it still ends up getting destroyed. (I wonder how many leaks this is papering over...)

That leaves the BC at a refcount of 2, which matches the two remaining cycles with a ChildSHistory and a BrowsingContextWebProgress. We already ran the last CC, so we leak.

The most obvious way to fix this would be to shut down some of this networking stuff earlier, but I don't know if that is papering over some other issue or what.

Kershaw, could this kind of "leak until shutdown" of a channel via RedirectChannelRegistrar be a sign of some kind of issue, or could I just change it so that we clear out the registrar before the shutdown CC instead of after it? This is showing up with the http3 version of the Mochitests, but I don't know if this is due to the very limited set of tests we are running from this directory or if it could be due to some actual browser issue. thanks.

Flags: needinfo?(continuation) → needinfo?(kershaw)

(In reply to Andrew McCreight [:mccr8] from comment #50)

Kershaw, could this kind of "leak until shutdown" of a channel via RedirectChannelRegistrar be a sign of some kind of issue, or could I just change it so that we clear out the registrar before the shutdown CC instead of after it? This is showing up with the http3 version of the Mochitests, but I don't know if this is due to the very limited set of tests we are running from this directory or if it could be due to some actual browser issue. thanks.

Yes, I see no reason that we can't clear RedirectChannelRegistrar earlier. Maybe we can clear it when receiving profile-change-net-teardown.
I also don't think this has something to do with Http/3 tests. Maybe it's the timing change that makes this leak happens more often.

Flags: needinfo?(kershaw)

Thank you so much for looking more deeply, mccr8.
Next step appears to be looking into Networking shutdown procedures, then?

Component: DOM: Security → Networking

I have a patch to clear out the two registrars earlier. It seems like it fixes the leak. I'll put it up for review when I get a full clean try run. I just made them get cleared at XPCOMShutdownFinal via ClearOnShutdown as that seemed like the small change.

Assignee: nobody → continuation

These registrars can hold channels alive, which can hold listeners alive,
which can hold cycle collected things alive. By clearing the registrars
before the final CC instead of after the final CC, we can avoid spurious
leaks.

It would make more logical sense to do this earlier, at net teardown,
but I've made the minimal change to try to avoid stirring up weird
shutdown issues.

Pushed by amccreight@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/b012b9792225 Clean up networking registrars earlier to avoid leaks. r=necko-reviewers,kershaw
Status: REOPENED → RESOLVED
Closed: 3 years ago2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 114 Branch

This does appear to have actually gone away.

If a similar leak appears later, please file a new bug for it.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: