1692873 - is screenshare simulcast supported?

Status: RESOLVED FIXED

(Core :: WebRTC, defect, P2)

Description

[Francesco Durighetto]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36

Steps to reproduce:

I'm trying to publish a screensharing stream in simulcast but only one ssrc is received by the SFU.
Also, when I use replacetrack with a screen track in a correctly published webcam simulcast stream, the number of layers (temporal and spatials) drops to 1 until the webcam stream is replaced again.

this can also be tested in https://github.com/fippo/simulcast-playground/blob/gh-pages/rid-as-mid.html changing line 35 to do a getDisplayMedia()

Actual results:

Only one ssrc is sent to other party and no simulcast is being published.

Expected results:

At least couple of temporal layer should be published by Firefox when screensharing in simulcast.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Privacy: Anti-Tracking' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Francesco Durighetto]

(In reply to Release mgmt bot [:sylvestre / :calixte / :marco for bugbug] from comment #1)

The Bugbug bot thinks this bug should belong to the 'Core::Privacy: Anti-Tracking' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 3

[Nils Ohlmeier [:drno]]

I think it should work. But it sounds like we have a bug there. It might be that Firefox old version of libwebrtc prevents simulcast of screen shares... I think I might have seen some code like that.

Comment 4

[Michael Froman [:mjf]]

(In reply to Francesco Durighetto from comment #0)

this can also be tested in https://github.com/fippo/simulcast-playground/blob/gh-pages/rid-as-mid.html changing line 35 to do a getDisplayMedia()

Francesco, would you mind providing more details around this? I've tried this, but must not be doing exactly what you are because I get nothing at all on the screen after modifying line 35.

Comment 5

[Francesco Durighetto]

(In reply to Michael Froman [:mjf] from comment #4)

(In reply to Francesco Durighetto from comment #0)

this can also be tested in https://github.com/fippo/simulcast-playground/blob/gh-pages/rid-as-mid.html changing line 35 to do a getDisplayMedia()

Francesco, would you mind providing more details around this? I've tried this, but must not be doing exactly what you are because I get nothing at all on the screen after modifying line 35.

You can try to copy the following code into an HTML page: https://pastebin.com/duKPSGmN

note that not always the video is subscribed and displayed in the tabs. reload the page few times to get an idea of what I'm assuming is a bug

Comment 6

[Michael Froman [:mjf]]

I've tried this with our in-progress libwebrtc update and did not have luck (insta-crash). I think we're not quite far enough along for this particular use case. I see this as interesting in the log when running a debug build:

[Child 20601, Main Thread] WARNING: NS_ENSURE_TRUE(info) failed: file {path-to-source}/extensions/permissions/PermissionDelegateHandler.cpp:348

I'll capture screen shots of the html in Comment 5 so we can have additional conversations.

Comment 7

[Michael Froman [:mjf]]

Attachment: test.html served with python3 -m http.server

Comment 8

[Michael Froman [:mjf]]

FYI: I'm using the attached test.html file in a directory, served by running python3 -m http.server.

Comment 9

[Michael Froman [:mjf]]

Attachment: results from test.html

Here is what I see when using test.html when locally served. This doesn't seem to work at all in Chrome. I'm not exactly sure what I'm supposed to see.

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: FirefoxChromeSimulcastComparison.png

I'm able to repro:

• with this fiddle, I get just one (large) video,
• whereas in Chrome, or s/getDisplayMedia/getUserMedia/ it works: shows 3 videos (small, medium, large).

Comment 11

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

This does not appear to be a regression. I was able to go back to 2018-08-01 (using an older fiddle to go past bug 1405495). But past that I run into some stricter tests we used to have that trip up fippo's simulcast hack: DOMException: "Answer changes mid for level, was 'sdparta_0', now '0'"

Comment 12

[Francesco Durighetto]

I think it is safe to at least switch the bug to confirmed

Comment 13

[Byron Campen [:bwc]]

I'm going to guess that there's no chance of this working until we're done with bug 1654112. It is possible that bug 1654112 will be sufficient, since we do not differentiate between screenshare and camera in our simulcast negotiation/configuration code, I think.

Comment 14

[streamer45]

Hi, wondering if there was an update on this or a plan to get it fixed anytime soon? Thank you.

Comment 15

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Here's an updated fiddle showing the problem more clearly. Chrome sends 4,2,1, Safari sends 1,1,1, (likely only a defaults problem), while Firefox only sends 4 for some reason.

Comment 17

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1692873 - Enable simulcast for screensharing sources. r?jib

Comment 18

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1692873 - Test simulcast screenshare in webrtc/simulcast/screenshare.https.html WPT. r?bwc

Depends on D189663

Comment 19

[Andreas Pehrson [:pehrsons]]

Let's get this landed.

Comment 20

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1692873 - Add VideoConduit gtests for screensharing. r?#webrtc-reviewers

Comment 21

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1692873 - Add mochitest for simulcast screensharing. r?jib!

Comment 22

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1692873 - Create smaller video elements for playback in simulcast tests. r?jib

For debugging purposes, when the test fails in CI and creates a screenshot.
gDM sources tend to be large, and for the sake of this test they should be
encoded large too. This shrinks the dimensions of the rendered video tracks
instead.

Comment 23

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1692873 - Use the same number of temporal layers for both screensharing and non-screensharing simulcast. r?#webrtc-reviewers

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/88a33c282af1
Test simulcast screenshare in webrtc/simulcast/screenshare.https.html WPT. r=bwc
https://hg.mozilla.org/integration/autoland/rev/154f072ea936
Add VideoConduit gtests for screensharing. r=webrtc-reviewers,ng
https://hg.mozilla.org/integration/autoland/rev/fd77389cbc70
Add mochitest for simulcast screensharing. r=jib
https://hg.mozilla.org/integration/autoland/rev/d88f2751aa16
Create smaller video elements for playback in simulcast tests. r=jib
https://hg.mozilla.org/integration/autoland/rev/26a05286fa42
Enable simulcast for screensharing sources. r=jib
https://hg.mozilla.org/integration/autoland/rev/d4850b36e2ff
Use the same number of temporal layers for both screensharing and non-screensharing simulcast. r=webrtc-reviewers,ng

Comment 25

[Andreas Pehrson [:pehrsons]]

Adding dev-doc-needed because it would be good to get a note in the release notes for developers, under WebRTC, that "MediaStreamTracks for screen and window capture, i.e. originating from getDisplayMedia, can now be encoded as multiple simulcast layers when using VP8."

Comment 26

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/49266 for changes under testing/web-platform/tests

Comment 27

[amarc]

Backed out for causing wpt failures @ screenshare.https.html

• Backout link
• Push with failures
• Failure Log
• Failure line:

TEST-UNEXPECTED-FAIL | /webrtc/simulcast/screenshare.https.html | Basic simulcast setup with two spatial layers - promise_test: Unhandled rejection with value: object "TypeError: navigator.mediaDevices.getDisplayMedia is not a function"

Comment 28

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR was closed without merging

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/efbb9e1af880
Test simulcast screenshare in webrtc/simulcast/screenshare.https.html WPT. r=bwc
https://hg.mozilla.org/integration/autoland/rev/c547e3d451c5
Add VideoConduit gtests for screensharing. r=webrtc-reviewers,ng
https://hg.mozilla.org/integration/autoland/rev/a93266847c8c
Add mochitest for simulcast screensharing. r=jib
https://hg.mozilla.org/integration/autoland/rev/1b2720a24c44
Create smaller video elements for playback in simulcast tests. r=jib
https://hg.mozilla.org/integration/autoland/rev/70fa4bed6f07
Enable simulcast for screensharing sources. r=jib
https://hg.mozilla.org/integration/autoland/rev/304638224e18
Use the same number of temporal layers for both screensharing and non-screensharing simulcast. r=webrtc-reviewers,ng

Comment 30

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/efbb9e1af880
https://hg.mozilla.org/mozilla-central/rev/c547e3d451c5
https://hg.mozilla.org/mozilla-central/rev/a93266847c8c
https://hg.mozilla.org/mozilla-central/rev/1b2720a24c44
https://hg.mozilla.org/mozilla-central/rev/70fa4bed6f07
https://hg.mozilla.org/mozilla-central/rev/304638224e18

Comment 31

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 32

[Hamish Willee]

FF134 MDN work for this can be tracked in https://github.com/mdn/content/issues/36914

Unfortunately I have no idea how to even start with this. I know what simulcast means, but not what the APIs a developer users to implement it are - existing MDN docs do not even define the term.

Can you help scope the problem for me:

1. Is this new functionality or a bug fix?
2. If new functionality, what WebRTC interfaces were added/removed etc.
3. Can you propose a release note you'd like for the MDN release notes?

Depending on your answer I'll see if I can find someone in the docs team with more WebRTC experience than me to help.

Comment 33

[Andreas Pehrson [:pehrsons]]

(In reply to Hamish Willee from comment #32)

1. Is this new functionality or a bug fix?

More of a bug fix. We have supported simulcast with VP8 for non-screensharing video for the longest time, but didn't for screensharing video for historical reasons relating to an external library. With this bug we treat both types of video the same, though they're handled a bit differently under the hood.

2. If new functionality, what WebRTC interfaces were added/removed etc.

n/a

3. Can you propose a release note you'd like for the MDN release notes?

See comment 25!

Thanks

Comment 34

[Hamish Willee]

Thanks very much. I added the release note in https://github.com/mdn/content/pull/37020