Intermittent leakcheck | default process: 2432027 bytes leaked (AbstractThread, AbstractWatcher, AddonPathService, Animation, AnimationEffectReadOnly, ...)

RESOLVED DUPLICATE of bug 1348547

Status

RESOLVED DUPLICATE of bug 1348547
2 years ago
3 months ago

People

(Reporter: intermittent-bug-filer, Assigned: gbrown)

Tracking

({intermittent-failure, memory-leak})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [stockwell fixed])

Keywords: mlk
16 failures in 777 pushes (0.021 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 11
* mozilla-inbound: 3
* oak: 1
* graphics: 1

Platform breakdown:
* osx-10-10: 12
* windows8-64: 2
* windows7-32-vm: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-13&endday=2017-03-19&tree=all
(Assignee)

Comment 2

2 years ago
This happens on OSX and Windows, always e10s, always debug (of course).

There were a few isolated failures in February and early March, then a sudden increase in frequency March 17.

This leak is consistently reported at the end of devtools/client/aboutdebugging/test test runs. Many leaked URLs and leaked objects are reported.
29 failures in 136 pushes (0.213 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 14
* mozilla-inbound: 9
* graphics: 3
* mozilla-central: 2
* try: 1

Platform breakdown:
* osx-10-10: 16
* linux64: 11
* windows8-64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-20&endday=2017-03-20&tree=all
and now this is becoming frequent enough that we should look into this.

looking at the first failure on March 17th [0], I see this:
09:51:37     INFO - GECKO(1740) | WARNING: YOU ARE LEAKING THE WORLD (at least one JSRuntime and everything alive inside it, that is) AT JS_ShutDown TIME.  FIX THIS!
09:51:37     INFO - GECKO(1740) | [Parent 1740] ###!!! ASSERTION: Component Manager being held past XPCOM shutdown.: 'cnt == 0', file /home/worker/workspace/build/src/xpcom/build/XPCOMInit.cpp, line 1045


Looking at this, I did some retrigger to help narrow this down:
https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=osx%20debug%20dt&tochange=5e664d9cc14d3331264be607ae27c650352066b6&fromchange=bb5c698bff0a4d73ee015230818faacc587b07d5&group_state=expanded&selectedJob=84636043

it will take a while to get the full data, but I can follow up in here when we have it all.


[0] https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-macosx64-debug/autoland_yosemite_r7-debug_test-mochitest-e10s-devtools-chrome-1-bm106-tests1-macosx-build197.txt.gz
Component: General → General
Product: Core → Firefox
Whiteboard: [stockwell needswork]
Component: General → Developer Tools: about:debugging
:jdescottes, can you take a look at this?  With this being so recent and frequent it should be easier to debug.  I suspect this is the same root cause as bug 1348547 when we find it.
Flags: needinfo?(jdescottes)
possibly this came from bug 1348112?  collecting more retriggers to help prove that.
with more retriggers, that is likely the suspect, although it is hard to reproduce this failure- lets see if looking at that update and/or the tests can help result in figuring this out.
31 failures in 174 pushes (0.178 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 15
* autoland: 11
* try: 3
* mozilla-central: 2

Platform breakdown:
* osx-10-10: 21
* windows7-32-vm: 4
* linux64: 4
* windows8-64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-21&endday=2017-03-21&tree=all
41 failures in 186 pushes (0.22 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 23
* mozilla-inbound: 10
* graphics: 3
* try: 2
* mozilla-central: 2
* mozilla-beta: 1

Platform breakdown:
* osx-10-10: 27
* linux64: 6
* windows8-64: 4
* windows7-32-vm: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-22&endday=2017-03-22&tree=all
36 failures in 174 pushes (0.207 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 19
* mozilla-inbound: 11
* mozilla-central: 5
* graphics: 1

Platform breakdown:
* osx-10-10: 22
* linux64: 7
* windows7-32-vm: 4
* windows8-64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-23&endday=2017-03-23&tree=all
38 failures in 153 pushes (0.248 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 21
* mozilla-inbound: 16
* oak: 1

Platform breakdown:
* osx-10-10: 29
* windows8-64: 4
* windows7-32-vm: 3
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-24&endday=2017-03-24&tree=all
209 failures in 898 pushes (0.233 failures/push) were associated with this bug in the last 7 days. 

This is the #1 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 102
* mozilla-inbound: 68
* mozilla-central: 14
* try: 11
* graphics: 8
* oak: 5
* mozilla-beta: 1

Platform breakdown:
* osx-10-10: 134
* linux64: 36
* windows7-32-vm: 22
* windows8-64: 17

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-20&endday=2017-03-26&tree=all
32 failures in 141 pushes (0.227 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 21
* mozilla-inbound: 8
* graphics: 2
* mozilla-aurora: 1

Platform breakdown:
* osx-10-10: 20
* linux64: 7
* windows8-64: 3
* windows7-32-vm: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-27&endday=2017-03-27&tree=all
It is very unlikely that Bug 1348112 is the responsible for this regression. 
This bug only landed new files in devtools, which are not used at the moment.

The other suspect would be Bug 1345932, which is one of the only bugs touching to about:debugging.
It landed on 16 march, right before this spiked.
Flags: needinfo?(jdescottes)
:gbrown, as this is our #1 failure on the tree, can you take a look at this?
Flags: needinfo?(gbrown)
19 failures in 165 pushes (0.115 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 8
* autoland: 8
* mozilla-central: 2
* try: 1

Platform breakdown:
* osx-10-10: 12
* windows7-32-vm: 3
* linux64: 3
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-28&endday=2017-03-28&tree=all
(Assignee)

Comment 17

2 years ago
I don't have much experience with leaks, so I'd be happy to hand this off to someone else, but I'll give it a shot.

I'll try to reproduce on try, then see if backing out suspect changesets helps. (Bug 1345932, perhaps https://hg.mozilla.org/integration/autoland/rev/5e664d9cc14d3331264be607ae27c650352066b6, ...)
Assignee: nobody → gbrown
Flags: needinfo?(gbrown)
For the record I did some retriggers on autoland before / after the changesets from Bug 1345932 were merged:
- https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=eb818ab6e5797c51bd7efabaa378ba54ae596529&selectedJob=86915238
- https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=5f4fd0bbc9a0797cd87e3b47a4845c99f2dfcc7c&selectedJob=86915281

And I didn't get any failure for both jobs, so maybe it is unrelated to this bug.
32 failures in 188 pushes (0.17 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 20
* mozilla-inbound: 6
* try: 4
* mozilla-central: 1
* graphics: 1

Platform breakdown:
* osx-10-10: 16
* linux64: 10
* windows7-32-vm: 5
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-29&endday=2017-03-29&tree=all
30 failures in 123 pushes (0.244 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 12
* autoland: 9
* try: 6
* oak: 2
* mozilla-central: 1

Platform breakdown:
* linux64: 13
* osx-10-10: 8
* windows7-32-vm: 6
* windows8-64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-30&endday=2017-03-30&tree=all
(Assignee)

Comment 21

2 years ago
It looks like the increase in content processes is responsible for the increase in frequency of this leak.

https://treeherder.mozilla.org/#/jobs?repo=try&author=gbrown@mozilla.com&tochange=101430a5866394f8cac4e7943c08d707a54f125a&fromchange=ac47424769e7ac4c043846809a189f3c02466088

The top-most push is 1 content process, the middle 4 content processes, the bottom 2 content processes. I got 3 leaks with 4 content processes, 0 leaks with 1 or 2 content processes. (I manipulated dom.ipc.processCount in all.js for this).

https://bugzilla.mozilla.org/show_bug.cgi?id=1348547#c17 is worth quoting here:

(Blake Kaplan (:mrbkap) from comment #17)
> That is very surprising. Service worker debugging doesn't work in e10s-multi
> and these tests attempt to force a single content process [1][2]. So bug
> 1336398 really should have little to no effect on these tests. That suggests
> a bug in our process selector.
> 
> [1]
> http://searchfox.org/mozilla-central/rev/
> 72fe012899a1b27d34838ab463ad1ae5b116d76b/devtools/client/aboutdebugging/test/
> browser_service_workers_status.js#15
> [2]
> http://searchfox.org/mozilla-central/rev/
> 72fe012899a1b27d34838ab463ad1ae5b116d76b/devtools/client/aboutdebugging/test/
> head.js#425,433
See Also: → bug 1348547
35 failures in 146 pushes (0.24 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 21
* mozilla-inbound: 11
* try: 2
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 19
* linux64: 10
* windows8-64: 3
* windows7-32-vm: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-31&endday=2017-03-31&tree=all
17 failures in 51 pushes (0.333 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 8
* mozilla-inbound: 6
* mozilla-central: 3

Platform breakdown:
* linux64: 10
* osx-10-10: 5
* windows8-64: 1
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-04-01&endday=2017-04-01&tree=all
198 failures in 845 pushes (0.234 failures/push) were associated with this bug in the last 7 days. 

This is the #4 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 101
* mozilla-inbound: 60
* mozilla-central: 16
* try: 15
* graphics: 3
* oak: 2
* mozilla-aurora: 1

Platform breakdown:
* osx-10-10: 88
* linux64: 67
* windows7-32-vm: 28
* windows8-64: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-03-27&endday=2017-04-02&tree=all
:gbrown, do you know what is causing these leaks?   this is quite frequent and I would like to see us fix the issue or disable the offending tests this week- do you need help bisecting the leak to the root test failures?
Flags: needinfo?(gbrown)
(Assignee)

Comment 26

2 years ago
I do not know the root cause of the leaks. 

I have not bisected to find a test or group of tests that is correlated with the leak.

The leak increased in frequency with additional content processes and goes away with fewer content processes - comment 21.

There is a potential fix under discussion in bug 1348547. I'll be working on that today.
Flags: needinfo?(gbrown)
(Assignee)

Comment 27

2 years ago
(In reply to Geoff Brown [:gbrown] from comment #26)
> I have not bisected to find a test or group of tests that is correlated with
> the leak.

Except of course, we know it is in the aboutdebugging tests, and there's https://bugzilla.mozilla.org/show_bug.cgi?id=1348547#c17.
25 failures in 147 pushes (0.17 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 12
* autoland: 9
* try: 3
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 19
* windows7-32-vm: 4
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-04-03&endday=2017-04-03&tree=all
(Assignee)

Comment 29

2 years ago
I am hoping that the change landing in bug 1348547 will have a big impact on this bug.
22 failures in 134 pushes (0.164 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 10
* mozilla-inbound: 7
* try: 3
* pine: 1
* oak: 1

Platform breakdown:
* osx-10-10: 9
* windows7-32-vm: 6
* linux64: 5
* windows8-64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-04-04&endday=2017-04-04&tree=all
18 failures in 151 pushes (0.119 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* try: 8
* autoland: 8
* mozilla-central: 1
* graphics: 1

Platform breakdown:
* osx-10-10: 8
* linux64: 6
* windows7-32-vm: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-04-05&endday=2017-04-05&tree=all
(Assignee)

Comment 32

2 years ago
(In reply to Geoff Brown [:gbrown] from comment #29)
> I am hoping that the change landing in bug 1348547 will have a big impact on this bug.

And indeed it is happening much less frequently...but leaks continue, and look just the same. I'll bisect the aboutdebugging tests...
(Assignee)

Comment 33

2 years ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2cac2be219d351ea504c03d45f4c7a35a9d7a2f7 has 3 such instances. In every case, the leaks follow bug 1256544 - "browser_service_workers_start.js | Test timed out".
79 failures in 867 pushes (0.091 failures/push) were associated with this bug in the last 7 days. 

This is the #20 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 33
* mozilla-inbound: 23
* try: 16
* mozilla-central: 4
* pine: 1
* oak: 1
* graphics: 1

Platform breakdown:
* osx-10-10: 44
* windows7-32-vm: 17
* linux64: 16
* windows8-64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-04-03&endday=2017-04-09&tree=all
:gbrown, can we disable browser_service_workers_start.js to mitigate this failure?
Flags: needinfo?(gbrown)
(Assignee)

Comment 36

2 years ago
All recent failures have been on try. I'm just keeping this open to monitor for a few more days, but I'm hoping I can dup this to bug 1348547.
Flags: needinfo?(gbrown)
(Assignee)

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → DUPLICATE
Whiteboard: [stockwell needswork] → [stockwell fixed]
Duplicate of bug: 1348547
5 failures in 817 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 2
* try: 1
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* osx-10-10: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-04-17&endday=2017-04-23&tree=all
1 failures in 892 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-06-19&endday=2017-06-25&tree=all
2 failures in 718 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 1
* autoland: 1

Platform breakdown:
* windows7-32-vm: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-06-26&endday=2017-07-02&tree=all
1 failures in 656 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 1

Platform breakdown:
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-07-03&endday=2017-07-09&tree=all
3 failures in 720 pushes (0.004 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 1
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 1
* osx-10-10: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-07-10&endday=2017-07-16&tree=all
1 failures in 1008 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1329836&startday=2017-07-24&endday=2017-07-30&tree=all

Updated

3 months ago
Product: Firefox → DevTools
You need to log in before you can comment on or make changes to this bug.