Excessive memory usage from stopped transceivers
Categories
(Core :: WebRTC, defect, P2)
Tracking
()
People
(Reporter: jib, Unassigned)
References
Details
Attachments
(1 file)
21.11 KB,
text/plain
|
Details |
STRs:
- Open https://jsfiddle.net/jib1/p6b8gjuL/ in Nightly
- Hover over tab to note down process id of the tab
- Open Activity Monitor (or equivalent on non-OSX) and note memory usage of process (e.g.
250 MB
). - Click the
1000
button and wait many seconds until the word1000
is outputted on the page. - Check memory usage of process again (e.g.
1.45 GB
). - On dev tools
Memory
tab, click "Take snapshot". (e.g.2.45 MB
). - On about:memory, click
Measure
and look for process id (e.g. 95% isheap-unclassified
)
Expected result: Megabytes
Actual result: Gigabytes
This isn't a leak, since most of the memory is returned when the tab is closed, and pc.getTransceivers()
is holding on to 2000 stopped transceivers.
But those JS objects only make up 2.46 MB vs 2.01 MB = 450 KB total (from dev tools Memory):
2000 RTCRtpSender (96 KB)
2000 RTCRtpReceiver (96 KB)
2000 MediaStreamTrack (96 KB)
2000 RTCRtpTransceiver (96 KB)
1000 TranceiverImpl (48 KB)
or 225 bytes per pc.
The gigabytes must come from the c++ backend: (1410-250 = 930 MB) / 2000 = 580 KB per pc.
This may prove problematic for certain WebRTC use cases (conferences where participants join and leave a lot).
This would be fixed by bug 1568296 which lets us free stopped transceivers over time.
Until then, is this amount of memory usage expected or excessive? Does it point to something we want to fix or optimize?
Filing this mostly as a health report, looking for feedback.
Reporter | ||
Updated•5 years ago
|
Reporter | ||
Updated•5 years ago
|
Comment 1•5 years ago
|
||
I'm guessing that most of this memory utilization is related to webrtc.org objects. There might be some way to free those up more aggressively when the transceiver is stopped. Dan?
Do we have a memory profile for this?
Comment 2•5 years ago
|
||
DMD could probably shine some light on what exactly is allocated here.
I don't know if it's expected without seeing more data. Intuitively it does sound high.
Comment 3•5 years ago
|
||
I'll have a look at the webrtc.org memory usage using a DMD build like Andreas suggested.
Comment 4•5 years ago
|
||
FYI I saw a crash in release (linux) with jib's fiddle and wanted to check it out. It reveals a bit.
In debug on linux I hit this assertion failure after clicking "1000" and waiting a while (it wasn't done yet):
[warn] evutil_make_internal_pipe_: pipe: Too many open files
[err] evsig_init_: socketpair: Too many open files
Assertion failure: isEmpty() (failing this assertion means this LinkedList's creator is buggy: it should have removed all this list's elements before the list's destruction), at /home/pehrsons/dev/m-c-3/obj-x86_64-pc-linux-gnu/dist/include/mozilla/LinkedList.h:433
#0 0x00007f4bf2f03b72 in mozilla::LinkedList<mozilla::(anonymous namespace)::RegistryEntries>::~LinkedList() (this=0x7f4bf5d27d10 <mozilla::(anonymous namespace)::GetRegistryEntries()::sEntries>)
at /home/pehrsons/dev/m-c-3/obj-x86_64-pc-linux-gnu/dist/include/mozilla/LinkedList.h:430
#1 0x00007f4bff955041 in __run_exit_handlers (status=1, listp=0x7f4bffcfd718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#2 0x00007f4bff95513a in __GI_exit (status=<optimized out>) at exit.c:139
#3 0x00007f4bef669fdf in event_exit (errcode=1) at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/log.c:102
#4 0x00007f4bef66a07e in event_sock_err (eval=-3152256, sock=<optimized out>, fmt=<optimized out>) at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/log.c:140
#5 0x00007f4bef66abc2 in evsig_init_ (base=0x7f4ada241400) at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/signal.c:187
#6 0x00007f4bef64dd44 in epoll_init (base=0x7f4ada241400) at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/epoll.c:220
#7 0x00007f4bef6544df in event_base_new_with_config (cfg=<optimized out>) at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/event.c:652
#8 0x00007f4bef654683 in event_base_new () at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/event.c:485
#9 0x00007f4bf2aae827 in rtc::TaskQueue::Impl::Impl(char const*, rtc::TaskQueue*, rtc::TaskQueue::Priority) (this=0x7f4ad4dfbe90, queue_name=0x7f4becdd11f0 <webrtc::(anonymous namespace)::kIncomingQueueName> "IncomingVideoStream", queue=0x0, priority=rtc::TaskQueue::Priority::HIGH) at /home/pehrsons/dev/m-c-3/media/webrtc/trunk/webrtc/rtc_base/task_queue_libevent.cc:279
#10 0x00007f4bf2aaf6b0 in rtc::RefCountedObject<rtc::TaskQueue::Impl>::RefCountedObject<char const*&, rtc::TaskQueue*, rtc::TaskQueue::Priority&>(char const*&, rtc::TaskQueue*&&, rtc::TaskQueue::Priority&) (this=0x7f4ad4dfbe90, p0=<optimized out>, p1=<optimized out>, args=<optimized out>) at /home/pehrsons/dev/m-c-3/media/webrtc/trunk/webrtc/rtc_base/refcountedobject.h:31
#11 0x00007f4bf2aaf6b0 in rtc::TaskQueue::TaskQueue(char const*, rtc::TaskQueue::Priority) (this=0x7f4ad6cfa0c8, queue_name=0x7f4bffcff8b0 <_IO_stdfile_2_lock> "", priority=(unknown: -1740032160))
at /home/pehrsons/dev/m-c-3/media/webrtc/trunk/webrtc/rtc_base/task_queue_libevent.cc:490
#12 0x00007f4bf2acabbe in webrtc::internal::VideoReceiveStream::Start() (this=0x7f4ad36fb000) at /home/pehrsons/dev/m-c-3/media/webrtc/trunk/webrtc/video/video_receive_stream.cc:193
#13 0x00007f4befbb7ec6 in mozilla::WebrtcVideoConduit::StartReceivingLocked() (this=0x7f4baa1b2000) at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:2151
#14 0x00007f4befbba94c in mozilla::WebrtcVideoConduit::ConfigureRecvMediaCodecs(std::vector<mozilla::UniquePtr<mozilla::VideoCodecConfig, mozilla::DefaultDelete<mozilla::VideoCodecConfig> >, std::allocator<mozilla::UniquePtr<mozilla::VideoCodecConfig, mozilla::DefaultDelete<mozilla::VideoCodecConfig> > > > const&) (this=0x7f4baa1b2000, codecConfigList=...)
at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1599
#15 0x00007f4befbf9e0a in mozilla::TransceiverImpl::UpdateVideoConduit() (this=0x7f4baa1acea0) at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/TransceiverImpl.cpp:827
#16 0x00007f4befbf7ab5 in mozilla::TransceiverImpl::UpdateConduit() (this=0x7f4baa1acea0) at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/TransceiverImpl.cpp:218
#17 0x00007f4befbf182b in mozilla::PeerConnectionMedia::UpdateMediaPipelines() (this=<optimized out>) at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/PeerConnectionMedia.cpp:326
#18 0x00007f4befbf1f94 in mozilla::PeerConnectionImpl::SetSignalingState_m(mozilla::dom::RTCSignalingState, bool) (this=0x7f4bc8f87800, aSignalingState=<optimized out>, rollback=<optimized out>)
at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:2248
#19 0x00007f4befbeeb58 in mozilla::PeerConnectionImpl::SetLocalDescription(int, char const*) (this=0x7f4bc8f87800, aAction=<optimized out>, aSDP=<optimized out>)
at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:1304
#20 0x00007f4bf06c6041 in mozilla::PeerConnectionImpl::SetLocalDescription(int, nsTSubstring<char16_t> const&, mozilla::ErrorResult&) (this=0x7f4bffcfe680 <_IO_2_1_stderr_>, aAction=-3147600, aSDP=
..., rv=...) at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.h:261
I have 910 threads at this poing. A subset:
883 Thread 22789.29972 (IncomingVideoSt) 0x0000000070000002 in ?? ()
884 Thread 22789.29973 (DecodingThread) 0x0000000070000002 in ?? ()
885 Thread 22789.30021 (IncomingVideoSt) 0x0000000070000002 in ?? ()
886 Thread 22789.30022 (DecodingThread) 0x0000000070000002 in ?? ()
887 Thread 22789.30023 (IncomingVideoSt) 0x0000000070000002 in ?? ()
888 Thread 22789.30024 (DecodingThread) 0x0000000070000002 in ?? ()
889 Thread 22789.30025 (IncomingVideoSt) 0x0000000070000002 in ?? ()
890 Thread 22789.30026 (DecodingThread) 0x0000000070000002 in ?? ()
891 Thread 22789.30074 (IncomingVideoSt) 0x0000000070000002 in ?? ()
892 Thread 22789.30075 (DecodingThread) 0x0000000070000002 in ?? ()
893 Thread 22789.30076 (IncomingVideoSt) 0x0000000070000002 in ?? ()
894 Thread 22789.30077 (DecodingThread) 0x0000000070000002 in ?? ()
It seems like a reasonable assumption that their stacks make up a large part of that excessive memory usage.
Each rtc::TaskQueue
seems to keep its own thread. If they were instead backed by a threadpool like our TaskQueues, that would go a long way.
I could easily upload a recording of this to pernosco if anyone would be interested in that.
Comment 5•5 years ago
|
||
I also see the crash that Andreas hit if I click the 1000 button on the fiddle. Attached is the output from DMD if I use the 100 button. I truncated the output once the reported allocations were less than 1% and they were no longer WebRTC related.
Comment 6•5 years ago
|
||
The thread stack size is 256kb. We already reduced it once as part of the webrtc.org branch 64 update, because we were hitting problems with intermittent OOMs on the web platform tests: https://phabricator.services.mozilla.com/D11090. At that time, I filed Bug 1505509 about the thread build up.
Comment 7•5 years ago
|
||
(In reply to Byron Campen [:bwc] from comment #1)
I'm guessing that most of this memory utilization is related to webrtc.org objects. There might be some way to free those up more aggressively when the transceiver is stopped. Dan?
Can we free the conduits when the transceiver is stopped, and recreate them again if necessary? (I'm assuming a stopped transceiver can be restarted, or we would already be doing this...)
We could also push this down a level and add calls to the conduits to free the associated streams and recreate them if necessary. We already support recreating streams as part of reconfiguring encoders.
Reporter | ||
Comment 8•5 years ago
|
||
(I'm assuming a stopped transceiver can be restarted, or we would already be doing this...)
transceiver.stop()
is terminal, which means stopped transceivers can never be restarted.
Their underlying m-lines however may be recycled, but from a JS pov at least that's a new transceiver object with a fresh mid
. Dunno how we handle that in our implementation. Byron?
Comment 9•5 years ago
|
||
In all of our internal representations, stop() is terminal too. So, TransceiverImpl is dead once it is stopped.
Comment 10•5 years ago
|
||
I'll look at freeing the conduits when the transceiver is stopped. I'll leave this bug as is in case we want to follow up on other possibilities, e.g. the thread pool that Andreas suggested.
Rolling through this in triage -- could someone more familiar with the area throw a priority on this, please?
Updated•5 years ago
|
Comment 12•4 years ago
|
||
The Firefox tab crashes with the jsfiddle with Firefox 78.0.1 on Ubuntu 18.04.
I don't see the crashid on about:crashes
I can't reproduce with Windows 7.
https://jsfiddle.net/jib1/p6b8gjuL/
Updated•2 years ago
|
Updated•2 years ago
|
Description
•