Intermittent leakcheck | default process: 600 bytes leaked (CondVar, Mutex, nsRunnable, nsTArray_base, nsThread, ...)

RESOLVED FIXED in Firefox 47

Status

()

Core
WebRTC: Audio/Video
P1
normal
Rank:
10
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: nigelb, Assigned: jesup)

Tracking

({intermittent-failure, mlk})

Trunk
mozilla49
intermittent-failure, mlk
Points:
---

Firefox Tracking Flags

(firefox47 fixed, firefox48 fixed, firefox49 fixed)

Details

Comment hidden (empty)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)

Comment 3

2 years ago
7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 2
* mozilla-aurora: 2
* b2g-inbound: 2
* fx-team: 1

Platform breakdown:
* windows7-32: 4
* osx-10-6: 1
* linux64: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2015-11-16&endday=2015-11-22&tree=all

Comment 4

2 years ago
23 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 7
* fx-team: 6
* mozilla-central: 4
* b2g-inbound: 4
* try: 2

Platform breakdown:
* windows7-32: 21
* osx-10-6: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2015-11-23&endday=2015-11-29&tree=all

Comment 5

2 years ago
34 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 11
* fx-team: 11
* mozilla-central: 6
* b2g-inbound: 4
* try: 2

Platform breakdown:
* windows7-32: 33
* osx-10-6: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2015-11-30&endday=2015-12-06&tree=all

Comment 6

2 years ago
34 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 23
* fx-team: 9
* mozilla-central: 2

Platform breakdown:
* linux32: 15
* linux64: 13
* windows7-32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-02-25&endday=2016-02-25&tree=all

Comment 7

2 years ago
50 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 27
* fx-team: 15
* try: 5
* mozilla-central: 3

Platform breakdown:
* linux32: 29
* linux64: 14
* windows7-32: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-02-26&endday=2016-02-26&tree=all
This is currently the #2 leak on OrangeFactor.

Leak stack:
 11:22:53     INFO -  #00: mozilla::CondVar::CondVar(mozilla::Mutex&, char const*) [xpcom/glue/CondVar.h:47]
 11:22:53     INFO -  #01: nsThread::nsThread(nsThread::MainThreadFlag, unsigned int) [xpcom/glue/nsTArray.h:834]
 11:22:53     INFO -  #02: nsThreadManager::NewThread(unsigned int, unsigned int, nsIThread**) [xpcom/threads/nsThreadManager.cpp:252]
 11:22:53     INFO -  #03: NS_NewThread(nsIThread**, nsIRunnable*, unsigned int) [xpcom/glue/nsThreadUtils.cpp:72]
 11:22:53     INFO -  #04: mozilla::camera::GetCamerasChild() [xpcom/glue/nsThreadUtils.h:76]
 11:22:53     INFO -  #05: mozilla::MediaEngineWebRTC::EnumerateVideoDevices(mozilla::dom::MediaSourceEnum, nsTArray<RefPtr<mozilla::MediaEngineVideoSource> >*) [dom/media/systemservices/CamerasChild.h:133]
 11:22:53     INFO -  #06: mozilla::media::LambdaTask<mozilla::MediaManager::EnumerateRawDevices(uint64_t, mozilla::dom::MediaSourceEnum, mozilla::dom::MediaSourceEnum, bool, bool)::<lambda()> >::Run [dom/media/MediaManager.cpp:1053]
 11:22:53     INFO -  #07: MessageLoop::RunTask(Task*) [ipc/chromium/src/base/message_loop.cc:365]
 11:22:53     INFO -  #08: MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&) [ipc/chromium/src/base/message_loop.cc:375]
 11:22:53     INFO -  #09: MessageLoop::DoWork() [ipc/chromium/src/base/message_loop.cc:459]
 11:22:53     INFO -  #10: mozilla::ipc::DoWorkRunnable::Run() [ipc/glue/MessagePump.cpp:221]
 11:22:53     INFO -  #11: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1018]
 11:22:53     INFO -  #12: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/glue/nsThreadUtils.cpp:297]
 11:22:53     INFO -  #13: mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:355]
 11:22:53     INFO -  #14: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:235]
 11:22:53     INFO -  #15: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:520]
 11:22:53     INFO -  #16: base::Thread::ThreadMain() [ipc/chromium/src/base/thread.cc:175]
 11:22:53     INFO -  #17: ThreadFunc [ipc/chromium/src/base/platform_thread_posix.cc:38]
 11:22:53     INFO -  #18: libpthread.so.0 + 0x7e9a
11:22:53 INFO - #19: libc.so.6 + 0xf338d
Component: General → WebRTC: Audio/Video
Keywords: mlk
Product: Firefox → Core
See Also: → bug 1251123

Comment 9

2 years ago
19 automation job failures were associated with this bug yesterday.

Repository breakdown:
* try: 15
* mozilla-inbound: 4

Platform breakdown:
* linux32: 10
* linux64: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-02-27&endday=2016-02-27&tree=all

Comment 10

2 years ago
155 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 63
* try: 51
* fx-team: 35
* mozilla-central: 6

Platform breakdown:
* linux32: 84
* linux64: 51
* windows7-32: 20

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-02-22&endday=2016-02-28&tree=all
Did this start spiking after bug 1249313?
Flags: needinfo?(ryanvm)
That looks like a very distinct possibility.
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&endday=2016-02-29&startday=2016-02-01&tree=trunk
Flags: needinfo?(ryanvm)

Comment 13

2 years ago
44 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 26
* fx-team: 10
* try: 5
* mozilla-central: 3

Platform breakdown:
* linux64: 26
* linux32: 16
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-02-29&endday=2016-02-29&tree=all
(Assignee)

Updated

2 years ago
backlog: --- → webrtc/webaudio+
Rank: 10
Priority: -- → P1
I'm ok with backing out bug 1249313 to get this orange under control. I think there's no functional impact with the current MediaManager workings.

The patch in that bug is likely correct, but if it's causing so much sherrifing pain I think we can pull it till we figure out why it's making the orange worse.
Flags: needinfo?(ryanvm)
Flags: needinfo?(ryanvm)

Comment 15

2 years ago
19 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 12
* fx-team: 6
* try: 1

Platform breakdown:
* linux32: 13
* linux64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-03-01&endday=2016-03-01&tree=all
(Assignee)

Comment 16

2 years ago
(In reply to Gian-Carlo Pascutto [:gcp] from comment #14)
> I'm ok with backing out bug 1249313 to get this orange under control. I
> think there's no functional impact with the current MediaManager workings.
> 
> The patch in that bug is likely correct, but if it's causing so much
> sherrifing pain I think we can pull it till we figure out why it's making
> the orange worse.

I think we should back out, especially given "no functional impact".  Also, is bug 1251123 a dup of this?  If so, let's mark it as such.
Assignee: nobody → gpascutto
Flags: needinfo?(gpascutto)
It was already backed out.
Flags: needinfo?(gpascutto)
Duplicate of this bug: 1251123
Bug 1252647 may or may not be a dupe too.
FWIW I've been trying to track this down to no avail so far, because it was happening almost permanently with bug 1250990 (before the backout of course, trying again now to see if it has improved).

As I understand it, the Camera IPC thread gets runnables both from the IPC thread and the MediaManager thread.

Seems there's occasionally (almost permanently with bug 1250990) an outstanding runnable at time of termination. I think we'd need to instrument more to see where it is coming from. There's certainly a lot of code to ensure this doesn't happen, so it's hard to tell what it could be.
(In reply to Jan-Ivar Bruaroey [:jib] from comment #20)
> Seems there's occasionally (almost permanently with bug 1250990) an
> outstanding runnable at time of termination. I think we'd need to instrument
> more to see where it is coming from. There's certainly a lot of code to
> ensure this doesn't happen, so it's hard to tell what it could be.

Bug 1252647 seems to point to the ThreadDestructor being a possibility, although who knows there might be two leaks :-/

Comment 22

2 years ago
75 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 40
* fx-team: 16
* try: 13
* mozilla-central: 6

Platform breakdown:
* linux64: 35
* linux32: 33
* windows7-32: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-02-29&endday=2016-03-06&tree=all

Comment 23

2 years ago
9 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 5
* fx-team: 3
* try: 1

Platform breakdown:
* windows7-32: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-03-07&endday=2016-03-13&tree=all

Comment 24

2 years ago
9 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6
* fx-team: 2
* mozilla-aurora: 1

Platform breakdown:
* windows7-32: 8
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-03-14&endday=2016-03-20&tree=all
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a8d25beb97c6

Comment 26

2 years ago
13 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 5
* fx-team: 4
* mozilla-central: 2
* try: 1
* ash: 1

Platform breakdown:
* windows7-32: 12
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-03-21&endday=2016-03-27&tree=all
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9d352764f15b

Comment 28

2 years ago
16 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 5
* mozilla-aurora: 4
* try: 2
* mozilla-beta: 2
* fx-team: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-03-28&endday=2016-04-03&tree=all

Comment 29

2 years ago
18 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 11
* fx-team: 4
* try: 2
* ash: 1

Platform breakdown:
* windows7-32: 15
* linux64: 2
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-04-04&endday=2016-04-10&tree=all

Comment 30

2 years ago
35 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 13
* fx-team: 11
* try: 9
* mozilla-central: 2

Platform breakdown:
* windows7-32: 33
* linux64: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-04-11&endday=2016-04-17&tree=all

Comment 31

2 years ago
27 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 17
* fx-team: 4
* mozilla-aurora: 3
* try: 1
* mozilla-release: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 21
* linux64: 3
* linux32: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-04-18&endday=2016-04-24&tree=all
Looking good so far since bug 1267600 landed.
Depends on: 1267600
No OrangeFactor hits for this since bug 1267600 landed. Calling this fixed!
Assignee: gpascutto → rjesup
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox44: affected → ---
status-firefox47: --- → affected
status-firefox48: --- → affected
status-firefox49: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Target Milestone: mozilla48 → mozilla49

Comment 34

2 years ago
5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 2
* try: 1
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* windows7-32: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-04-25&endday=2016-05-01&tree=all

Comment 35

2 years ago
6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 6

Platform breakdown:
* windows7-32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-05-10&endday=2016-05-16&tree=all
status-firefox47: affected → fixed
status-firefox48: affected → fixed

Comment 36

2 years ago
5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 3
* mozilla-beta: 2

Platform breakdown:
* windows7-32: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-05-16&endday=2016-05-22&tree=all

Comment 37

2 years ago
15 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 12
* mozilla-beta: 3

Platform breakdown:
* windows7-32: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1207431&startday=2016-05-23&endday=2016-05-29&tree=all
You need to log in before you can comment on or make changes to this bug.