1289336 - Intermittent leakcheck | default process: 8326 / 11212 bytes leaked (CondVar, ConsoleReportCollector, DataStorage, FetchDriver, HttpBaseChannel, ...)

Status: RESOLVED INCOMPLETE

(Toolkit :: Add-ons Manager, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: tomcat [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=32591102&repo=mozilla-inbound

http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win32-debug/1469502122/mozilla-inbound_win7_ix-debug_test-mochitest-browser-chrome-1-bm119-tests1-windows-build194.txt.gz

Comment 1

[BMO Automation]

Bulk assigning P3 to all open intermittent bugs without a priority set in Firefox components per bug 1298978.

Comment 2

[Intermittent Failures Robot]

20 failures in 146 pushes (0.137 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* try: 11
* autoland: 5
* mozilla-inbound: 4

Platform breakdown:
* osx-10-10: 12
* linux64: 4
* windows7-32-vm: 2
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1289336&startday=2017-05-02&endday=2017-05-02&tree=all

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

and a fun leak to debug!

Doing some retriggers:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=osx%20debug%20e10s%20browser-chrome-4&tochange=084bebaffd26e5a0e3aa71b0751b95bce6ac356f&fromchange=fd049b7bf3d233fa51227992f1cbd8bfcdad1e98&selectedJob=95528072

I suspect we will have to find a way to determine if this is a specific directory causing the leak on shutdown, likewise the specific failure rate.

Comment 4

[Intermittent Failures Robot]

42 failures in 770 pushes (0.055 failures/push) were associated with this bug in the last 7 days. 

This is the #26 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 16
* try: 11
* mozilla-inbound: 10
* mozilla-central: 4
* oak: 1

Platform breakdown:
* osx-10-10: 34
* windows7-32-vm: 3
* linux64: 3
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1289336&startday=2017-05-01&endday=2017-05-07&tree=all

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

retriggered:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=osx%20debug%20e10s%20browser-chrome-4&tochange=084bebaffd26e5a0e3aa71b0751b95bce6ac356f&fromchange=a55987f8d963b51d62d853c6d2400bf1df8c8ede&selectedJob=97093534

it looks like our main issue is in shutting down the browser after running browser/base/content/test/appUpdate, and a root cause doesn't look too feasible.



that leaves us with 13 tests to find the leak:
https://dxr.mozilla.org/mozilla-central/source/browser/base/content/test/appUpdate/browser.ini?q=path%3Abrowser%2Fbase%2Fcontent%2Ftest%2FappUpdate%2Fbrowser.ini&redirect_type=single

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

focusing on osx debug browser-chrome e10s, I will do some try pushes to narrow this down.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

this doesn't look so clear cut here:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&fromchange=441924b2f3f5e42db568eeb89d1b84b53832cfac&filter-searchStr=browser-chrome&selectedJob=97413472

the only leaks I saw where early on:
https://hg.mozilla.org/try/rev/3f113b53df4b9fd4067e2d9cd3e4d71deb5be263

but I would have expected to see more leaks in subsequent pushes

Comment 8

[Intermittent Failures Robot]

12 failures in 879 pushes (0.014 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 7
* autoland: 4
* graphics: 1

Platform breakdown:
* osx-10-10: 12

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1289336&startday=2017-05-08&endday=2017-05-14&tree=all

Comment 9

[Intermittent Failures Robot]

34 failures in 777 pushes (0.044 failures/push) were associated with this bug in the last 7 days. 

This is the #37 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 16
* mozilla-inbound: 13
* try: 2
* mozilla-central: 2
* graphics: 1

Platform breakdown:
* osx-10-10: 34

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1289336&startday=2017-05-15&endday=2017-05-21&tree=all

Comment 10

[Robert Strong (they/them - no direct email)]

The original log in comment #0 did not run any of the application update code. It did run extension update code which might be why this was filed under app update
chrome://mozapps/content/extensions/update.xul

Going off of https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1289336&startday=2017-05-14&endday=2017-05-21&tree=trunk
I found there were logs that did run app update code as well as others that did not.

Comment 11

[Robert Strong (they/them - no direct email)]

Since it appears that this bug was first found when running extension / add-on manager tests moving to add-ons manager even though there might be something similar going on with app update.

Comment 12

[Intermittent Failures Robot]

23 failures in 891 pushes (0.026 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 14
* mozilla-inbound: 8
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 23

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1289336&startday=2017-05-22&endday=2017-05-28&tree=all

Comment 13

[Intermittent Failures Robot]

8 failures in 820 pushes (0.01 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 4
* mozilla-inbound: 2
* mozilla-central: 1
* graphics: 1

Platform breakdown:
* osx-10-10: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1289336&startday=2017-05-29&endday=2017-06-04&tree=all

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

still a low failure rate, specifically on osx debug, not worth investigating as it would be hard to reproduce.

Comment 15

[Intermittent Failures Robot]

11 failures in 864 pushes (0.013 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 7
* mozilla-inbound: 2
* try: 1
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1289336&startday=2017-06-05&endday=2017-06-11&tree=all

Comment 16

[Intermittent Failures Robot]

1 failures in 720 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1289336&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 17

[Intermittent Failures Robot]

1 failures in 718 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2018-03-26&endday=2018-04-01&tree=trunk

Comment 18

[Stefan Hindli [:stefan_hindli]]

https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&fromchange=fbb68a1fb7ce707f1be9a44e7eb3990efc8e74ea&selectedJob=214986445

Comment 19

[Intermittent Failures Robot]

14 failures in 2808 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 4
* autoland: 5
* mozilla-central: 4
* try: 1

Platform breakdown:
* linux32: 10
* linux64: 3
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2018-11-26&endday=2018-12-02&tree=all

Comment 20

[Andreea Pavel [:apavel]]

Retriggered to find where the spike comes from

 https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=214930810&searchStr=linux%2Cdebug%2Cmochitests%2Ctest-linux32%2Fdebug-mochitest-chrome-3%2Cm%28c3%29&tochange=17a1195583aadff908496e3a1448f4cd6e57089f&fromchange=0ffc19c364e82f893b636540c19da0aa6ca36cc2

Comment 21

[Intermittent Failures Robot]

17 failures in 1862 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 1
* mozilla-central: 6
* mozilla-inbound: 2
* autoland: 8

Platform breakdown:
* osx-10-10: 1
* linux32: 8
* linux64: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2018-12-03&endday=2018-12-09&tree=all

Comment 22

[Intermittent Failures Robot]

23 failures in 2485 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* try: 4
* mozilla-beta: 3
* autoland: 9
* mozilla-inbound: 5
* mozilla-central: 2

Platform breakdown:
* linux32: 8
* osx-10-10: 4
* linux64: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2018-12-10&endday=2018-12-16&tree=all

Comment 23

[Intermittent Failures Robot]

17 failures in 2648 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 4
* mozilla-beta: 1
* try: 1
* mozilla-inbound: 5
* autoland: 6

Platform breakdown:
* linux64: 4
* osx-10-10: 4
* linux32: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2018-12-17&endday=2018-12-23&tree=all

Comment 24

[Intermittent Failures Robot]

4 failures in 1008 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 1
* autoland: 1
* mozilla-central: 2

Platform breakdown:
* linux64: 1
* linux32: 2
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2018-12-24&endday=2018-12-30&tree=all

Comment 25

[Intermittent Failures Robot]

7 failures in 1668 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 1
* autoland: 3
* mozilla-central: 3

Platform breakdown:
* linux32: 2
* osx-10-10: 4
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2018-12-31&endday=2019-01-06&tree=all

Comment 26

[Intermittent Failures Robot]

1 failures in 2383 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2019-01-21&endday=2019-01-27&tree=all

Comment 27

[Intermittent Failures Robot]

3 failures in 2628 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-central: 2

Platform breakdown:

• osx-10-10: 1
• linux64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2019-01-28&endday=2019-02-03&tree=all

Comment 28

[Intermittent Failures Robot]

4 failures in 2658 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform breakdown:

• linux32: 2
• osx-10-10: 1
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2019-02-04&endday=2019-02-10&tree=all

Comment 29

[Intermittent Failures Robot]

3 failures in 2832 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-inbound: 2

Platform breakdown:

• linux32: 2
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2019-02-11&endday=2019-02-17&tree=all

Comment 30

[Intermittent Failures Robot]

1 failures in 2602 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1289336&startday=2019-02-18&endday=2019-02-24&tree=all

Comment 31

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup