If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Enable leak logging for content processes on desktop

NEW
Assigned to

Status

()

Core
DOM: Content Processes
3 years ago
6 days ago

People

(Reporter: billm, Assigned: mccr8)

Tracking

(Depends on: 1 bug, Blocks: 1 bug, {meta})

unspecified
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(e10s-)

Details

(Whiteboard: [MemShrink:meta])

Attachments

(1 obsolete attachment)

(Reporter)

Description

3 years ago
It actually works right now, but the content process usually crashes so we never see any leaks.
(Assignee)

Comment 1

3 years ago
Thanks, Bill.

Just for some context here, given the plans for a single content process this really needs to work for reals in order to ship e10s.  Without it, we'll have no visibility into severe permanent leaks.  (If we were using process-per-tab it wouldn't be as big of a deal, because content process leaks would only last until somebody closed a tab, but as it is, these leaks are just as bad as pre-e10s.)
tracking-e10s: --- → ?
OS: Linux → All
Hardware: x86_64 → All
Whiteboard: [MemShrink]
Depends on: 1038943
(Assignee)

Updated

3 years ago
Depends on: 1045847
tracking-e10s: ? → +
(Assignee)

Updated

3 years ago
Depends on: 1035454
(Assignee)

Comment 2

3 years ago
Apparently bug 1035454 is enough that we start getting child process leak checking.  But there are a bunch of leaks, so bug 1052224 is going to disable the leak checking.
See Also: → bug 1052224
Whiteboard: [MemShrink] → [MemShrink:P1]
(Assignee)

Comment 3

3 years ago
try run with leak checking enabled: https://tbpl.mozilla.org/?tree=Try&rev=32bf41cea397
(ignore the M3 asserts)
Blocks: 997462, 984139
No longer blocks: 984139
Blocks: 1058354
(Assignee)

Comment 4

3 years ago
Leak logging works once bug 1035454 is landed, so the bigger issue is fixing the content process leaks, so I'm changing this to be about letting us re-enable it.  The actual enabling (once bug 1035454 is landed) is basically just going to be backing out bug 1052224.
Summary: Get leak logging working in content processes → Enable leak logging for content processes
(Assignee)

Comment 5

3 years ago
I guess I'm working on this the most, so I'll take the bug.
Assignee: nobody → continuation
Does the testharness automatically do sane things with multiple logs?
(Assignee)

Comment 7

3 years ago
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #6)
> Does the testharness automatically do sane things with multiple logs?

Yeah, it seems like it.  Check out M2 here:
  https://tbpl.mozilla.org/?tree=Try&rev=32bf41cea397
(Reporter)

Comment 8

3 years ago
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #6)
> Does the testharness automatically do sane things with multiple logs?

It seems to:
http://mxr.mozilla.org/mozilla-central/source/build/automationutils.py#376
(Assignee)

Updated

3 years ago
Depends on: 1059480
(Assignee)

Updated

3 years ago
Depends on: 1060526
(Assignee)

Comment 9

3 years ago
Created attachment 8483012 [details] [diff] [review]
Produce an error for larger child process leaks. WIP

This is a hacky work in progress, but it is useful when locally investigating leaks.  This patch makes it so that we error when a content child leaks, but only if it is larger than 5kb, which is what we always leak.
(Assignee)

Updated

3 years ago
Depends on: 831223
(Assignee)

Updated

3 years ago
Depends on: 1063312
(Assignee)

Comment 10

3 years ago
On top of my hacky pile of patches, e10s M4 has about 620kb of leaks and M5 has about 200kb.  Across a few runs, the size of the former is pretty stables, and the latter was identical in all runs, which is a good sign.
  https://tbpl.mozilla.org/?tree=Try&rev=6ce2c7636985
Does the large one include JS?  My experience with b2g was that once we got rid of the JS we were leaking the # was extremely stable.
(Assignee)

Comment 12

3 years ago
Well, it is only varying by a few hundred bytes, so that's pretty stable.

M2 and M3 hit asserts I added because they are running the GMPService in a content process, so I'm not sure how bad those are going to be once I fix that.  I haven't looked at M1 yet.
(Assignee)

Updated

3 years ago
No longer depends on: 1063312
(Assignee)

Updated

3 years ago
Blocks: 931285
(Assignee)

Updated

3 years ago
No longer blocks: 931285
Depends on: 931285
(Assignee)

Updated

3 years ago
Depends on: 1064587
(Assignee)

Updated

3 years ago
Depends on: 1065098
(Assignee)

Updated

3 years ago
No longer depends on: 1065098
See Also: → bug 1065098
(Assignee)

Updated

3 years ago
Depends on: 1065117
(Assignee)

Updated

3 years ago
Depends on: 1065536
tracking-e10s: + → m5+
(Assignee)

Updated

3 years ago
Depends on: 1067633
(Assignee)

Updated

3 years ago
Depends on: 1067664
(Assignee)

Updated

3 years ago
Blocks: 1070068
(Assignee)

Comment 13

3 years ago
B2G has a number of special issues of its own, so I'm splitting that out into a separate bug, bug 1070068.
Summary: Enable leak logging for content processes → Enable leak logging for content processes on desktop
(Assignee)

Comment 14

3 years ago
Comment on attachment 8483012 [details] [diff] [review]
Produce an error for larger child process leaks. WIP

Leak thresholds can now be configured by process type by changing options.leakThresholds, and what processes we error on when they don't produce a log can be adjusted by changing options.ignoreMissingLeaks.
Attachment #8483012 - Attachment is obsolete: true
(Assignee)

Updated

3 years ago
Depends on: 1062472
(Assignee)

Updated

3 years ago
Depends on: 1062443
(Assignee)

Comment 15

3 years ago
e10s M3 has gone from leaking about 11kb in the tab process to leaking 942kb.
(Assignee)

Comment 16

3 years ago
Good: https://hg.mozilla.org/try/rev/d7ec36a77534
Bad: https://hg.mozilla.org/try/rev/388e101e75c8

So that's only a window of a few days.  I'll try to narrow it down to a test and then a regressing changeset at some point.  I tried running the tests that are leaking URLs, but nothing useful came up.

leaking try run: https://tbpl.mozilla.org/?tree=Try&rev=5f123e75dc9c
(I could have also somehow caused the leaks with the patches I added in the try push.)
(Assignee)

Updated

3 years ago
Depends on: 1080301
(Assignee)

Updated

3 years ago
Depends on: 1081415
(Assignee)

Updated

3 years ago
Depends on: 1081453
(Assignee)

Updated

3 years ago
Depends on: 1073310
(Assignee)

Updated

3 years ago
Depends on: 1083897
(Assignee)

Updated

3 years ago
Depends on: 1087509
(Assignee)

Updated

3 years ago
Depends on: 1087518
(Assignee)

Updated

3 years ago
No longer depends on: 1062443
(Assignee)

Updated

3 years ago
No longer depends on: 1062472
(Assignee)

Updated

3 years ago
No longer depends on: 1087509
(Assignee)

Updated

3 years ago
No longer depends on: 1087518
(Assignee)

Updated

3 years ago
Depends on: 1087613
(Assignee)

Updated

3 years ago
Blocks: 1089816
(Assignee)

Updated

3 years ago
Depends on: 1080096
(Assignee)

Updated

3 years ago
No longer depends on: 1067633
(Assignee)

Updated

3 years ago
Depends on: 1067585
(Assignee)

Updated

3 years ago
No longer depends on: 1067585
(Assignee)

Updated

3 years ago
Depends on: 1091917
(Assignee)

Updated

3 years ago
No longer depends on: 1073310
(Assignee)

Updated

3 years ago
Depends on: 1103068
(Assignee)

Updated

3 years ago
Depends on: 1116261
(Assignee)

Updated

3 years ago
Depends on: 1117203
(Assignee)

Updated

3 years ago
Depends on: 845982
(Assignee)

Updated

3 years ago
Depends on: 1121670
(Assignee)

Updated

3 years ago
No longer blocks: 1089816
(Assignee)

Comment 17

3 years ago
Bug 1057908 should make us not run the GeckoMediaPluginService in the content process, where it doesn't shut down properly, and thus leaks.
Depends on: 1057908

Comment 18

3 years ago
requesting retriage on this tracker.
tracking-e10s: m5+ → ?
(Assignee)

Comment 19

3 years ago
(In reply to Jim Mathies [:jimm] from comment #18)
> requesting retriage on this tracker.

This has sort of degenerated into a tracking bug, so I'm not sure it needs to be tracked or not.  We're running XPCOM leak checking on Linux.  Last I checked, it was working on OSX, but we're not actually running it.  Windows has a known issue that needs to be fixed, but that work is being tracked in its own bug, bug 1091917.
(Assignee)

Comment 20

3 years ago
(We're not running leak checking on OSX, because it only runs in debug builds, and we're not running e10s debug OSX builds on TreeHerder.)
tracking-e10s: ? → -
Keywords: meta
(Assignee)

Updated

3 years ago
Depends on: 1150147
(Assignee)

Updated

3 years ago
Depends on: 1137999
(Assignee)

Updated

3 years ago
Depends on: 1128486
(Assignee)

Updated

3 years ago
Depends on: 1157308
(Assignee)

Updated

3 years ago
Depends on: 1157467
(Assignee)

Updated

3 years ago
Depends on: 1157468
(Assignee)

Updated

3 years ago
Depends on: 1157515
(Assignee)

Updated

3 years ago
Depends on: 1158404
(Assignee)

Updated

2 years ago
Depends on: 1193917
(Assignee)

Updated

2 years ago
Depends on: 1193922
(Assignee)

Comment 21

2 years ago
We're doing this testing now, so no need to leave it as a P1.
Whiteboard: [MemShrink:P1] → [MemShrink:meta]
(Assignee)

Updated

2 years ago
Depends on: 1215265
(Assignee)

Updated

2 years ago
Depends on: 1215684
(Assignee)

Updated

2 years ago
Depends on: 1219369
(Assignee)

Updated

6 days ago
Depends on: 1401764
(Assignee)

Updated

6 days ago
Depends on: 1401763
You need to log in before you can comment on or make changes to this bug.