1918096 - Hook up an SCContentSharingPicker MVP integration

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, task)

Description

[Andreas Pehrson [:pehrsons]]

We already have Pipewire desktop capture using a system dialog. Let's hook up macOS >= 15 similarly, behind a pref, for now. This let's us integrate SCContentSharingPicker without much other patches (e.g. frontend), and provides an avenue for potential uplift into 131.

Comment 1

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Integrate SCContentSharingPicker into ScreenCapturerSck. r?#webrtc-reviewers

Comment 2

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Generate libwebrtc moz.build files. r?#webrtc-reviewers

Comment 3

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Wire up SCContentSharingPicker using Pipewire desktop capture UX. r?#webrtc-reviewers!, r?grulja

Comment 4

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Reject gUM/gDM request if video track ends before first frame. r?#webrtc-reviewers

This case gets hit if the SCContentSharingPicker is cancelled by the user.

Comment 5

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - In ScreenCapturerSck improve dictionary ergonomics. r?#webrtc-reviewers

Comment 6

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - In ScreenCapturerSck adapt to sources that change resolution. r?#webrtc-reviewers

If the resolution is smaller than the allocated surface, we crop.
If the resolution is larger than the allocated surface, we reconfigure with a
larger surface.

The allocated surface has the size we told it to have in the
SCStreamConfiguration, in pixels.

If the source is a screen, the size is pretty static. Changing display settings
could affect it.
If the source is a window, the size is initially the size of the window. The
user resizing the window affects the size.
If the source is multiple windows, the size initially seems to be that of the
display they're sitting on. Windows across multiple displays are not captured
into the same surface, as of macOS 14.

Comment 7

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - In ScreenCapturerSck skip some processing when possible. r?#webrtc-reviewers

No need to update the last frame unless it has changed.

Comment 8

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Enable SCContentSharingPicker by default (limited to macOS 15). r?#webrtc-reviewers

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/86a5d2db126e
Integrate SCContentSharingPicker into ScreenCapturerSck. r=webrtc-reviewers,padenot,dbaker
https://hg.mozilla.org/integration/autoland/rev/c8c8702310d4
Generate libwebrtc moz.build files. r=webrtc-reviewers,ng,dbaker

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/eb9b355882a9
Wire up SCContentSharingPicker using Pipewire desktop capture UX. r=grulja,webrtc-reviewers,dbaker
https://hg.mozilla.org/integration/autoland/rev/97e405de0635
Reject gUM/gDM request if video track ends before first frame. r=webrtc-reviewers,dbaker

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/aabf0bd94f28
In ScreenCapturerSck improve dictionary ergonomics. r=webrtc-reviewers,ng
https://hg.mozilla.org/integration/autoland/rev/bc6215142fea
In ScreenCapturerSck adapt to sources that change resolution. r=webrtc-reviewers,ng
https://hg.mozilla.org/integration/autoland/rev/8d52f2fd5754
In ScreenCapturerSck skip some processing when possible. r=webrtc-reviewers,ng

Comment 12

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/86a5d2db126e
https://hg.mozilla.org/mozilla-central/rev/c8c8702310d4
https://hg.mozilla.org/mozilla-central/rev/eb9b355882a9
https://hg.mozilla.org/mozilla-central/rev/97e405de0635

Comment 13

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Make SckPickerHandle thread safe. r?#webrtc-reviewers, r?padenot

We cannot guarantee that the thread it is used on is static, as both the
VideoCapture and DesktopCapture threads can come and go.

Comment 14

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/aabf0bd94f28
https://hg.mozilla.org/mozilla-central/rev/bc6215142fea
https://hg.mozilla.org/mozilla-central/rev/8d52f2fd5754

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cfcdd5339805
Make SckPickerHandle thread safe. r=padenot

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/07dd5fcced62
Enable SCContentSharingPicker by default (limited to macOS 15). r=webrtc-reviewers,dbaker

Comment 17

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/cfcdd5339805
https://hg.mozilla.org/mozilla-central/rev/07dd5fcced62

Comment 18

[Jim Mathies [:jimm]]

[Tracking Requested - why for this release]:
We would like to uplift this work (plus the fixes in the two regressions) once it's had more bake time.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

FWIW, if this is going to go to ESR128, it'll need a rebased patch stack to do so. I'm also not super thrilled about taking this large of a set of changes there, so more explanation as to why it's necessary would be appreciated.

Comment 20

[Andreas Pehrson [:pehrsons]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #19)

FWIW, if this is going to go to ESR128, it'll need a rebased patch stack to do so. I'm also not super thrilled about taking this large of a set of changes there, so more explanation as to why it's necessary would be appreciated.

This is mainly to avoid the new system alerts in macOS 15 that are mentioned in their release notes (see ScreenCaptureKit Deprecations). There's some code in here affecting the existing desktop capture framework, but note most of it is behind a pref and a macOS 15 runtime check. We also would like this to have more time to bake in release before uplifting.

Comment 21

[Donal Meehan [:dmeehan]]

:pehrsons next week is the final week before esr128.5 goes to build.
Is this ready for an esr128 uplift request?

Comment 22

[Donal Meehan [:dmeehan]]

Will reach out in Slack to get an uplift request in time for 128.6 at least

Comment 23

[Andreas Pehrson [:pehrsons]]

I'm on this now.

Comment 24

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Integrate SCContentSharingPicker into ScreenCapturerSck.

This works by creating ScreenCapturerSck with a sck_allow_system_picker
DesktopCaptureOptions flag.

The flow is similar to the Pipewire capturer in that SelectSource has no effect.
Start() brings up the system picker, and:

• automatically starts capture on selection, also reconfigures on future
  selection changes.
• signals a stop by setting the permanent error flag on cancel or error.

The system picker is configured for both single display and multiple windows to
give users as much power over what they share as possible. There is an
application mode also, but other browsers are not allowing access to that as of
now.

Original Revision: https://phabricator.services.mozilla.com/D221851

Comment 25

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Generate libwebrtc moz.build files.

Original Revision: https://phabricator.services.mozilla.com/D221852

Comment 26

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Wire up SCContentSharingPicker using Pipewire desktop capture UX.

Original Revision: https://phabricator.services.mozilla.com/D221853

Comment 27

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Reject gUM/gDM request if video track ends before first frame.

This case gets hit if the SCContentSharingPicker is cancelled by the user.

Original Revision: https://phabricator.services.mozilla.com/D221854

Comment 28

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - In ScreenCapturerSck improve dictionary ergonomics.

Original Revision: https://phabricator.services.mozilla.com/D221940

Comment 29

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - In ScreenCapturerSck adapt to sources that change resolution.

If the resolution is smaller than the allocated surface, we crop.
If the resolution is larger than the allocated surface, we reconfigure with a
larger surface.

The allocated surface has the size we told it to have in the
SCStreamConfiguration, in pixels.

If the source is a screen, the size is pretty static. Changing display settings
could affect it.
If the source is a window, the size is initially the size of the window. The
user resizing the window affects the size.
If the source is multiple windows, the size initially seems to be that of the
display they're sitting on. Windows across multiple displays are not captured
into the same surface, as of macOS 14.

Original Revision: https://phabricator.services.mozilla.com/D221941

Comment 30

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - In ScreenCapturerSck skip some processing when possible.

No need to update the last frame unless it has changed.

Original Revision: https://phabricator.services.mozilla.com/D221942

Comment 31

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Make SckPickerHandle thread safe.

We cannot guarantee that the thread it is used on is static, as both the
VideoCapture and DesktopCapture threads can come and go.

Original Revision: https://phabricator.services.mozilla.com/D222082

Comment 32

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1918096 - Enable SCContentSharingPicker by default (limited to macOS 15).

Original Revision: https://phabricator.services.mozilla.com/D221997

Comment 33

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr128/rev/763672311790
https://hg.mozilla.org/releases/mozilla-esr128/rev/7137a430a3e2
https://hg.mozilla.org/releases/mozilla-esr128/rev/f3041571a5af
https://hg.mozilla.org/releases/mozilla-esr128/rev/c363120c7250
https://hg.mozilla.org/releases/mozilla-esr128/rev/627e494ce7d1
https://hg.mozilla.org/releases/mozilla-esr128/rev/9dcc048d54bb
https://hg.mozilla.org/releases/mozilla-esr128/rev/19419915bd0c
https://hg.mozilla.org/releases/mozilla-esr128/rev/438abd29c801
https://hg.mozilla.org/releases/mozilla-esr128/rev/3aa7437729f5