Open Bug 1580301 Opened 5 years ago Updated 2 years ago

Excessive memory usage from stopped transceivers

Categories

(Core :: WebRTC, defect, P2)

Product:
Core
Component:
WebRTC
Type:
defect

Tracking

()

People

(Reporter: jib, Unassigned)

References

Details

Attachments

(1 file)

Output from DMD
21.11 KB, text/plain
Details

STRs:

  1. Open https://jsfiddle.net/jib1/p6b8gjuL/ in Nightly
  2. Hover over tab to note down process id of the tab
  3. Open Activity Monitor (or equivalent on non-OSX) and note memory usage of process (e.g. 250 MB).
  4. Click the 1000 button and wait many seconds until the word 1000 is outputted on the page.
  5. Check memory usage of process again (e.g. 1.45 GB).
  6. On dev tools Memory tab, click "Take snapshot". (e.g. 2.45 MB).
  7. On about:memory, click Measure and look for process id (e.g. 95% is heap-unclassified)

Expected result: Megabytes
Actual result: Gigabytes

This isn't a leak, since most of the memory is returned when the tab is closed, and pc.getTransceivers() is holding on to 2000 stopped transceivers.

But those JS objects only make up 2.46 MB vs 2.01 MB = 450 KB total (from dev tools Memory):

2000 RTCRtpSender (96 KB)
2000 RTCRtpReceiver (96 KB)
2000 MediaStreamTrack (96 KB)
2000 RTCRtpTransceiver (96 KB)
1000 TranceiverImpl (48 KB)

or 225 bytes per pc.

The gigabytes must come from the c++ backend: (1410-250 = 930 MB) / 2000 = 580 KB per pc.

This may prove problematic for certain WebRTC use cases (conferences where participants join and leave a lot).

This would be fixed by bug 1568296 which lets us free stopped transceivers over time.

Until then, is this amount of memory usage expected or excessive? Does it point to something we want to fix or optimize?

Filing this mostly as a health report, looking for feedback.

Flags: needinfo?(docfaraday)
Flags: needinfo?(apehrson)

I'm guessing that most of this memory utilization is related to webrtc.org objects. There might be some way to free those up more aggressively when the transceiver is stopped. Dan?

Do we have a memory profile for this?

Flags: needinfo?(docfaraday) → needinfo?(dminor)

DMD could probably shine some light on what exactly is allocated here.

I don't know if it's expected without seeing more data. Intuitively it does sound high.

Flags: needinfo?(apehrson)

I'll have a look at the webrtc.org memory usage using a DMD build like Andreas suggested.

Flags: needinfo?(dminor)

FYI I saw a crash in release (linux) with jib's fiddle and wanted to check it out. It reveals a bit.

In debug on linux I hit this assertion failure after clicking "1000" and waiting a while (it wasn't done yet):

[warn] evutil_make_internal_pipe_: pipe: Too many open files                                                                                                                                                
[err] evsig_init_: socketpair: Too many open files                                                                                                                                                          
Assertion failure: isEmpty() (failing this assertion means this LinkedList's creator is buggy: it should have removed all this list's elements before the list's destruction), at /home/pehrsons/dev/m-c-3/obj-x86_64-pc-linux-gnu/dist/include/mozilla/LinkedList.h:433
#0  0x00007f4bf2f03b72 in mozilla::LinkedList<mozilla::(anonymous namespace)::RegistryEntries>::~LinkedList() (this=0x7f4bf5d27d10 <mozilla::(anonymous namespace)::GetRegistryEntries()::sEntries>)
    at /home/pehrsons/dev/m-c-3/obj-x86_64-pc-linux-gnu/dist/include/mozilla/LinkedList.h:430
#1  0x00007f4bff955041 in __run_exit_handlers (status=1, listp=0x7f4bffcfd718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#2  0x00007f4bff95513a in __GI_exit (status=<optimized out>) at exit.c:139
#3  0x00007f4bef669fdf in event_exit (errcode=1) at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/log.c:102
#4  0x00007f4bef66a07e in event_sock_err (eval=-3152256, sock=<optimized out>, fmt=<optimized out>) at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/log.c:140
#5  0x00007f4bef66abc2 in evsig_init_ (base=0x7f4ada241400) at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/signal.c:187
#6  0x00007f4bef64dd44 in epoll_init (base=0x7f4ada241400) at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/epoll.c:220
#7  0x00007f4bef6544df in event_base_new_with_config (cfg=<optimized out>) at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/event.c:652
#8  0x00007f4bef654683 in event_base_new () at /home/pehrsons/dev/m-c-3/ipc/chromium/src/third_party/libevent/event.c:485
#9  0x00007f4bf2aae827 in rtc::TaskQueue::Impl::Impl(char const*, rtc::TaskQueue*, rtc::TaskQueue::Priority) (this=0x7f4ad4dfbe90, queue_name=0x7f4becdd11f0 <webrtc::(anonymous namespace)::kIncomingQueueName> "IncomingVideoStream", queue=0x0, priority=rtc::TaskQueue::Priority::HIGH) at /home/pehrsons/dev/m-c-3/media/webrtc/trunk/webrtc/rtc_base/task_queue_libevent.cc:279
#10 0x00007f4bf2aaf6b0 in rtc::RefCountedObject<rtc::TaskQueue::Impl>::RefCountedObject<char const*&, rtc::TaskQueue*, rtc::TaskQueue::Priority&>(char const*&, rtc::TaskQueue*&&, rtc::TaskQueue::Priority&) (this=0x7f4ad4dfbe90, p0=<optimized out>, p1=<optimized out>, args=<optimized out>) at /home/pehrsons/dev/m-c-3/media/webrtc/trunk/webrtc/rtc_base/refcountedobject.h:31
#11 0x00007f4bf2aaf6b0 in rtc::TaskQueue::TaskQueue(char const*, rtc::TaskQueue::Priority) (this=0x7f4ad6cfa0c8, queue_name=0x7f4bffcff8b0 <_IO_stdfile_2_lock> "", priority=(unknown: -1740032160))
    at /home/pehrsons/dev/m-c-3/media/webrtc/trunk/webrtc/rtc_base/task_queue_libevent.cc:490
#12 0x00007f4bf2acabbe in webrtc::internal::VideoReceiveStream::Start() (this=0x7f4ad36fb000) at /home/pehrsons/dev/m-c-3/media/webrtc/trunk/webrtc/video/video_receive_stream.cc:193
#13 0x00007f4befbb7ec6 in mozilla::WebrtcVideoConduit::StartReceivingLocked() (this=0x7f4baa1b2000) at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:2151
#14 0x00007f4befbba94c in mozilla::WebrtcVideoConduit::ConfigureRecvMediaCodecs(std::vector<mozilla::UniquePtr<mozilla::VideoCodecConfig, mozilla::DefaultDelete<mozilla::VideoCodecConfig> >, std::allocator<mozilla::UniquePtr<mozilla::VideoCodecConfig, mozilla::DefaultDelete<mozilla::VideoCodecConfig> > > > const&) (this=0x7f4baa1b2000, codecConfigList=...)
    at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1599
#15 0x00007f4befbf9e0a in mozilla::TransceiverImpl::UpdateVideoConduit() (this=0x7f4baa1acea0) at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/TransceiverImpl.cpp:827
#16 0x00007f4befbf7ab5 in mozilla::TransceiverImpl::UpdateConduit() (this=0x7f4baa1acea0) at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/TransceiverImpl.cpp:218
#17 0x00007f4befbf182b in mozilla::PeerConnectionMedia::UpdateMediaPipelines() (this=<optimized out>) at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/PeerConnectionMedia.cpp:326
#18 0x00007f4befbf1f94 in mozilla::PeerConnectionImpl::SetSignalingState_m(mozilla::dom::RTCSignalingState, bool) (this=0x7f4bc8f87800, aSignalingState=<optimized out>, rollback=<optimized out>)
    at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:2248
#19 0x00007f4befbeeb58 in mozilla::PeerConnectionImpl::SetLocalDescription(int, char const*) (this=0x7f4bc8f87800, aAction=<optimized out>, aSDP=<optimized out>)
    at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:1304
#20 0x00007f4bf06c6041 in mozilla::PeerConnectionImpl::SetLocalDescription(int, nsTSubstring<char16_t> const&, mozilla::ErrorResult&) (this=0x7f4bffcfe680 <_IO_2_1_stderr_>, aAction=-3147600, aSDP=
    ..., rv=...) at /home/pehrsons/dev/m-c-3/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.h:261

I have 910 threads at this poing. A subset:

  883  Thread 22789.29972 (IncomingVideoSt) 0x0000000070000002 in ?? ()
  884  Thread 22789.29973 (DecodingThread) 0x0000000070000002 in ?? ()
  885  Thread 22789.30021 (IncomingVideoSt) 0x0000000070000002 in ?? ()
  886  Thread 22789.30022 (DecodingThread) 0x0000000070000002 in ?? ()
  887  Thread 22789.30023 (IncomingVideoSt) 0x0000000070000002 in ?? ()
  888  Thread 22789.30024 (DecodingThread) 0x0000000070000002 in ?? ()
  889  Thread 22789.30025 (IncomingVideoSt) 0x0000000070000002 in ?? ()
  890  Thread 22789.30026 (DecodingThread) 0x0000000070000002 in ?? ()
  891  Thread 22789.30074 (IncomingVideoSt) 0x0000000070000002 in ?? ()
  892  Thread 22789.30075 (DecodingThread) 0x0000000070000002 in ?? ()
  893  Thread 22789.30076 (IncomingVideoSt) 0x0000000070000002 in ?? ()
  894  Thread 22789.30077 (DecodingThread) 0x0000000070000002 in ?? ()

It seems like a reasonable assumption that their stacks make up a large part of that excessive memory usage.

Each rtc::TaskQueue seems to keep its own thread. If they were instead backed by a threadpool like our TaskQueues, that would go a long way.

I could easily upload a recording of this to pernosco if anyone would be interested in that.

Attached file Output from DMD —

I also see the crash that Andreas hit if I click the 1000 button on the fiddle. Attached is the output from DMD if I use the 100 button. I truncated the output once the reported allocations were less than 1% and they were no longer WebRTC related.

The thread stack size is 256kb. We already reduced it once as part of the webrtc.org branch 64 update, because we were hitting problems with intermittent OOMs on the web platform tests: https://phabricator.services.mozilla.com/D11090. At that time, I filed Bug 1505509 about the thread build up.

See Also: → 1505509

(In reply to Byron Campen [:bwc] from comment #1)

I'm guessing that most of this memory utilization is related to webrtc.org objects. There might be some way to free those up more aggressively when the transceiver is stopped. Dan?

Can we free the conduits when the transceiver is stopped, and recreate them again if necessary? (I'm assuming a stopped transceiver can be restarted, or we would already be doing this...)

We could also push this down a level and add calls to the conduits to free the associated streams and recreate them if necessary. We already support recreating streams as part of reconfiguring encoders.

(I'm assuming a stopped transceiver can be restarted, or we would already be doing this...)

transceiver.stop() is terminal, which means stopped transceivers can never be restarted.

Their underlying m-lines however may be recycled, but from a JS pov at least that's a new transceiver object with a fresh mid. Dunno how we handle that in our implementation. Byron?

Flags: needinfo?(docfaraday)

In all of our internal representations, stop() is terminal too. So, TransceiverImpl is dead once it is stopped.

Flags: needinfo?(docfaraday)
Depends on: 1580524

I'll look at freeing the conduits when the transceiver is stopped. I'll leave this bug as is in case we want to follow up on other possibilities, e.g. the thread pool that Andreas suggested.

Rolling through this in triage -- could someone more familiar with the area throw a priority on this, please?

Priority: -- → P2

The Firefox tab crashes with the jsfiddle with Firefox 78.0.1 on Ubuntu 18.04.
I don't see the crashid on about:crashes
I can't reproduce with Windows 7.
https://jsfiddle.net/jib1/p6b8gjuL/

Severity: normal → S3
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: