Closed Bug 1247244 Opened 5 years ago Closed 2 years ago

High CPU Usage in Jitsi WebRTC Call

Categories

(Core :: WebRTC, defect, P2)

43 Branch
Unspecified
macOS
defect

Tracking

()

RESOLVED INACTIVE
Blocking Flags:

People

(Reporter: abhinav.khaware, Assigned: drno)

References

Details

Attachments

(7 files)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36

Steps to reproduce:

1. Joined Jitsi session as 3 participants, link - https://meet.jit.si/abhiNOW
2. Joined same session from Macbook (details attached as zip file - MacBook Pro Abhinav.zip)
3. All 4 participants could see/hear each other.


Actual results:

Opened Activity Monitor on mac and found CPU usage to be high and Idle to be very low, 0.87% in the screenshot attached (see Jitsi on Firefox - high CPU.png). The audio from other participants was breaking up as a result and Macbook was becoming unresponsive.

Attaching signaling logs - FF43_log.txt, 
about:webrtc page output - about-webrtc.txt,
Debug log - WebRTC.log
Screenshot of Activity Monitor - Jitsi on Firefox - high CPU.png
Macbook details - MacBook Pro Abhinav.zip


Expected results:

With three other participants in the Jitsi session, audio shouldn't have broken up (Macbook shouldn't have high CPU usage).
Component: Untriaged → WebRTC
Product: Firefox → Core
Attached file FF43_log.txt.zip
Attached file about-webrtc.txt
Attached file WebRTC.log.zip
OS: Unspecified → Mac OS X
We have recently fixed a similar problem in bug 1224845. The fix for it is available in Firefox Nightly. Even though your call does not seem to involved IPv6 the bug fix from bug 1224845 might still improve/fix a similar issue. As the fix is available in Firefox Nightly would it be possible for you to test with Firefox Nightly 47, assuming the problem is reproducible for you.
I tried on Firefox Nightly 47.0a1 (2016-02-16) and can still reproduce the issue on the same Macbook (but updated to Mac OS X 10.11.3). Attaching Jitsi on Firefox Nightly.png
Please let me know if more information or logs are needed from our side.
Thanks for the reminder.

Yes I can repro it myself. Even a two party call between Windows and Mac produces way over 200% CPU on my Mac, but below 100% on Windows.

I'll start investigating.
Assignee: nobody → drno
Status: UNCONFIRMED → NEW
backlog: --- → webrtc/webaudio+
Rank: 21
Ever confirmed: true
Priority: -- → P2
Interesting: it is over 200% when Jitsi fails to render the remote video. When the two party call is up and running is uses "only" between 100% and 200%.
Hi Nils, I hope you are well.  Have you been able to make any progress on this issue?

For a little context, we (Blackboard) have introduced our new version of the Collaborate product using WebRTC (https://www.youtube.com/playlist?list=PLontYaReEU1thrYkZR4tynKpyvuTB_dkD).  Right now we only support Chrome for WebRTC and fall back to Flash for Firefox, IE and Safari.  We've had Firefox WebRTC support in "beta" for quite a while and have a fair bit of demand for the experience upgrade.  We have some Universities exclusively supporting Firefox. This issue is the last significant issue we'd like to see resolved before releasing our support for Firefox WebRTC.  We would really like to get these customers going on the best Firefox experience possible.

Cheers,
Carl Marrelli
Sr. Product Manager
Blackboard Collaborate
Sorry Carl that I did not follow up on this for such a long time, and that I failed to realize that this affects your service. I'll have a closer look next week. Let me know if you need higher priority.
Nils -- Padenot is also looking at reducing our CPU usage with nical (reducing our number of copies).  I'd like to make this a low P1 for now (P1, 19) and bump the ranking once your current work is done.  I think our CPU usage is playing in heavily to our perceived quality, especially for users with older machines.
Rank: 21 → 19
Priority: P2 → P1
Hi Nils, I appreciate the response!  I agree with Maire in her comment.  In fact, the quality of our WebRTC based application is quite poor on Macs (even the latest Macs) with this issue.  This is precisely why we have not been able to release Firefox WebRTC support.  

Next week is great!  I can't say I understand your prioritization enough to make an informed comment.  But we are very keen on having this resolved as soon as possible.  We can help test from any release channel you like and get loads of users running our own beta as well.
Carl - since you point out macs, is this something where you're screensharing a retina display?  Or screensharing on a Mac Air?

Retina displays (unless you limit the screencapture resolution, which you can do) means a LOT of bits to send on each update, and a LOT of CPU to encode (and decode).

Can you provide links/logins where we could experience this ourselves (and take profiles)?

Thanks
Flags: needinfo?(carl.marrelli)
Hi Randall,

We see the issue in just a 3-4 person video call.  We checked in Jitsi to ensure it wasn't just us.  So the original post here was about high CPU just doing a video call in Jitsi.  I think Abhinav posted some good info then.

Our application can be accessed here:
https://apollodemo01.bbcollab.com/collab/ui/session/guest/CD5B58A98E868AE4BA1510C750F388CC

You should just need to run standard audio/video calls to reproduce the high CPU utilization.
Hi Nils, 
I wanted to express support for Carl's request to see the Firefox WebRTC issues resolved. The University of Montana is one of those universities exclusively supporting Firefox so we see this as a critical issue. We've been involved with Carl's effort to make Collaborate an accessible system and being able to move away from Flash is important in that regard. I just wanted to thank you for your work on this issue.

Marlene Zentz
Senior Instructional Designer/Accessibility Specialist
University of Montana
Flags: needinfo?(carl.marrelli)
Thanks Carl for the link. That makes it a lot easier to try to reproduce the problem.

Unfortunately today I'm not able to reproduce the problem with the provided blackboard link.

I started a conference from my MacBook Pro laptop. I joined that conference with one Linux machine and two Windows laptops.
I see just below 100% CPU usage according to Mac's Activity Monitor. Which is significantly less then Abhinav reported with his screen shots. And I'm assuming that 80-100% CPU usage on a 2013 MacBook Pro should be okay to use the service.

I tried Firefox 48 (with and without e10s), Firefox 45 and even Firefox 43. For compassing I also joined the conference with Google Chrome from the MacBook. And for Chrome Activity Monitor shows me one Chrome Helper process with 60% CPU usage, plus at least one more with around 20%. So in total it looks to me like Chrome is using less resources, but not significantly less.

Now on my two Windows laptops the Windows Task Manager shows me around 10-20% CPU usage, which is incredibly low. But that could also be caused different ways of calculating the system usage.
On my Linux machine (Ubuntu 15) top tell me Firefox uses 180-200% CPU usage. That appears to be a lot. And I'm planing to have a look into that. But it does not sound like you are interested in Linux improvements :-)

I'll try to start the conference from one of the other machines and just join with the Mac. And I'll also try again just with the Jitsi video bridge, as I saw high CPU usage on that when I initially tried to reproduce the problem.

Please let me know if you spot anything why I would not be able to reproduce this.
Joining an existing 3 party call with my Mac also results in ~80-90% CPU usage.
Joining a running 3 party call on meet.jit.si with my Mac results in the same ~80% CPU usage.

Note: the SPD on the meet.jit.si is a lot shorter then on blackboard. Jitsi seems to add m-sections as needed, where the blackboard appears to pre-populate m-sections for non-existing participants.
Another note worthy observation: meet.jit.si shows "only" 80-100% CPU usage on Linux (compared to the 180-200% I saw on blackboard).
I am not sure how the fact Blackboard starts with 5 users, and the consequent larger SDP, right from the start is relevant. What does Jitsi do if you are the sixth person to join the conference? Surely you end up in the same place.

Your observation does seem to imply that the CPU usage is possibly dependent on the number of m-sections, is that a clue?


As an aside, we set up the maximum number of m-sections right from the start simply as an optimisation. We then do not require the logic for detecting the addition/removal of participants and the renegotiation of media via SDP each time. That applies to both server and client in our architecture.
Blackboard's conferencing server forwards the voice and video of up to 5 people on these pre-allocated channels. This means we can quickly switch people on these channels without having to re-negotiate the SDP.

We can still reproduce the CPU issues and can provide any logs you need or arrange a time to work on it directly with you.
Issue reproducible on Firefox 45.0.2 (25 Apr 2016), see screenshot attached.
So I did some profiling on my Linux machine to get an idea if anything in our WebRTC code is causing the high CPU usage. But the result is that more then 50% of the CPU time is spend in the cairo lib drawing things on the screen.
I spend last week probably chasing the wrong thing. But recently we got some more hints from a similar report, which lets me currently investigate some kind of race condition when decoding multiple video streams.
Depends on: 1273206
See Also: → 1281035
My understanding from communication outside of this bug report is that we can close this issue now as it appears to be fixed in FF 49. Correct?
Flags: needinfo?(abhinav.khaware)
I can still reproduce 170% CPU usage on Jitsi on FX 49.0a2 (2016-07-21), OS X 10.11.
I tried on FF 49 and FF 50 on Blackboard session and issue is reproducible on both now. CPU Usage is ~200% for 3 video streams. Please let me know if more information or logging is required. (I don't have a virtual mic or cam on MacBook).
Flags: needinfo?(abhinav.khaware)
High CPU issue reproduced on Aug 10, 2016 with Blackboard Session.

Steps to reproduce:

1. Join session https://apollodemo01.bbcollab.com/collab/ui/session/guest/8A5D356DFE1118DC9196240915B8CD23 first as 3 users with cam and mic on.  
2. Join session as Firefox user (49 or 50), start cam and mic.
3. Let the session run and talk between the participants.

The %CPU Usage for Firefox process goes to ~200%. MacBook Activity Monitor shows %idle below 4% in both Firefox 49 and 50. Trace files (with screenshots of Activity Monitor and MacBook details) are available here - https://drive.google.com/drive/folders/0BzQI6MLbgP7vQV9pcUxJR1I2LUk
Mass change P1->P2 to align with new Mozilla triage process
Priority: P1 → P2
For future reference: I think one thing we only found out later is that the Firefox performance on the integrated Intel graphic cards is really bad. Although it shouldn't be like that on MacBook Pro's it helps to switch to the discrete Nvidia or AMD card instead.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INACTIVE
You need to log in before you can comment on or make changes to this bug.