1703522 - Video from Screen Capture API has low frame rate

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[koen.francois]

Attachment: Recording Firefox.mov

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36

Steps to reproduce:

• Go to the following link: https://yari-demos.prod.mdn.mozit.cloud/en-US/docs/Web/API/Screen_Capture_API/Using_Screen_Capture/_samples_/Simple_screen_capture
• Capture any window or the entire screen (has no impact)
• Move your mouse

System information:
Macbook Pro 2018 2.6GHz, 16gb, Radeon Pro 560X
OSX 11.2.3
No external devices/displays connected.

Actual results:

Frames seem to be dropped. Not all mouse movements are captured, resulting in a choppy video.

Expected results:

Video should be smooth. You can compare the resulting videos between Chrome & Firefox.

Comment 1

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Core::WebRTC: Audio/Video' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Hi Francois, thanks for reporting! What dimensions and frame rate do you see if you run https://jsfiddle.net/jib1/5e1cbpyt/show ?

I get 3072 x 1920 x ~8 in Firefox. In Chrome I get 1536 x 960 x ~18, which may partly explain the frame rate difference (Firefox returns full Retina res).

This does not appear to be a regression.

Comment 3

[koen.francois]

Hi Jan-Ivar,

On Firefox I get 3840 x 2400 x ~4 compared to 1920 x 1200 x ~24 on Chrome.
When recording a smaller window of 1600 x 1000 in Firefox I get improved frame rates of ~12-14 which seems low still.

I understand that this may not be a performance regression, perhaps this can move to an enhancement (or an OSX or hardware-related defect) if you agree?
If not, 1703991 should already be a big improvement. 🙌

Thank you for creating that ticket & for your efforts in reproducing this!

Comment 4

[Paul Adenot (:padenot)]

This is a performance issue we've always had. I'm trying to find time to have a look and overhaul our usage of surfaces. For now we're doing to many copies and it causes this issue.

Comment 6

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Sorry I meant to assign myself to bug 1703991 instead, thinking that was the cause of this bug, which on closer examination appears unlikely.

I can still try to take a look. Paul, do you have more info on where in the code you think the problem you allude to in comment 4 lies?

Comment 7

[Paul Adenot (:padenot)]

I only see smoke, not the cause of the fire. On Linux: sudo perf top --sort comm,dso shows that the capture thread takes a lot of CPU, and also Xorg, when capturing. On macOS with Instruments, something similar is shown, full of libyuv scaling/memcpy/allocator. In general the allocator and memcpy use lots of CPU, and it should not be the case, we should retain the frames, maybe scaling them in the process. The frame rate is always single digit, even on computers that are impossibly fast (28 threads core i9 linux box, 12 threads core i9 macbook pro).

The Firefox profiler cannot be used to measure this, because the threads aren't registered. We should do it, but that's a separate concern, and it might well be that the CPU being used is in other processes, because we're not using system-level APIs correctly.

Comment 8

[Paul Adenot (:padenot)]

It's smoother when the machine is intentionally almost idle, but there is some latency though.

Comment 9

[Jeff Muizelaar [:jrmuizel]]

I had a look at this in Instruments with "Record Waiting Threads" and "Record Kernel Callstacks" turned on. In this profile I see 63% of the CaptureFrame time being spent waiting in CGDisplayCreateImage/SLDisplayCreateImage. Much of the rest of the time is spent in memcpy. Some of it likely doing zero-fault-on-demand.

While looking more closely I realized that we don't have the IOSurface capture path enabled. Chrome enabled since 2018: https://source.chromium.org/chromium/chromium/src/+/09cd5826e743af9dbcbde8ab36f73f5e0bd55f6c

Turning that on cuts the time spent CaptureFrame from ~80ms to ~22ms.

The code in DesktopCaptureImpl::ProcessIter() is also a bit weird. _maxFPSNeeded is actually 1000 / _requestedCapability.maxFPS so a better name would might be minimumCaptureIntervalMs. We currently wait for max(minimumCaptureIntervalMs, sleepTime) which prevents us from being able to hit the maxFPS if capturing takes a non-zero amount of time.

This should instead probably be max(minimumCaptureIntervalMs - processTime, sleepTime)

Comment 10

[Jeff Muizelaar [:jrmuizel]]

Attachment: Bug 1703522. Allow IOSurface capture by default.

This cuts the time spent in CaptureFrame for me from ~80ms to ~22ms.
It was enable by default in Chromium in 2018 via
https://source.chromium.org/chromium/chromium/src/+/09cd5826e743af9dbcbde8ab36f73f5e0bd55f6c

Comment 11

[Jeff Muizelaar [:jrmuizel]]

When using the IOSurfaces, half of the capture time is now in ConvertToI420 mostly servicing page faults presumably from zero-fill-on-demand because we allocate a fresh buffer every frame: https://searchfox.org/mozilla-central/rev/9769b513e38ee4f5df9d5d1eff55ff7cdc8cbf81/dom/media/systemservices/video_engine/desktop_capture_impl.cc#479

Comment 12

[Jeff Muizelaar [:jrmuizel]]

With the remaining time mostly in copies and buffer allocation/deallocation

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4efaf464a679
Allow IOSurface capture by default. r=pehrsons

Comment 14

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/4efaf464a679

Comment 15

[Jeff Muizelaar [:jrmuizel]]

Release Note Request (optional, but appreciated)
[Why is this notable]: Screencapture in WebRTC on macOS can now capture at higher frame rates
[Affects Firefox for Android]: Nope
[Suggested wording]: Lower CPU usage and increased frame rates during WebRTC screen capture on macOS

Comment 16

[Pascal Chevrel (relman team) -> :pascalc]

Should that note be merged with the existing note about general improvements in WebRTC following the libwebrtc upgrade or should it be separate?
https://www.mozilla.org/en-US/firefox/106.0beta/releasenotes/

Currently we have:

Major improvements of our WebRTC capabilities (libwebrtc library upgraded from version 86 to 103) with multiple improvements:

• Many RTP performance and reliability improvements.
• Windows and Wayland screen sharing improvements.
• Richer statistics.

Comment 17

[Jeff Muizelaar [:jrmuizel]]

I think it's fine to fold it in

Comment 18

[Pascal Chevrel (relman team) -> :pascalc]

Note folded into the existing webrtc note on Beta.