Closed Bug 1752559 Opened 3 years ago Closed 2 years ago

WebRTC fails to share the screen with VP9

Categories

(Core :: WebRTC: Audio/Video, defect, P2)

Product:
Core
Component:
WebRTC: Audio/Video
Version:
Firefox 96
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
109 Branch
Project Flags:
Webcompat Priority P3
Tracking Flags:
Tracking Status
firefox-esr91 --- unaffected
firefox-esr102 --- verified
firefox96 --- wontfix
firefox97 --- wontfix
firefox98 --- wontfix
firefox99 --- wontfix
firefox100 --- wontfix
firefox107 --- wontfix
firefox108 --- wontfix
firefox109 --- verified

People

(Reporter: pierre.bodilis+github, Assigned: pehrsons)

References

(Regression)

Details

(Keywords: regression)

Attachments

(2 files, 1 obsolete file)

Bug 1752559 - Test that a screencapture track can be encoded by all supported codecs. r?bwc!
48 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-esr102+
Details | Review
Bug 1752559 - Never configure SVC layers for VP9 screencaptures. r?bwc!
48 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-esr102+
Details | Review

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:96.0) Gecko/20100101 Firefox/96.0

Steps to reproduce:

start a webrtc session, using getDisplayMedia as the source stream for the peerconnection. Provided response from server selects VP9 as video codec
Tested with Firefox 96.0, and beta 97

Can be reproduiced with https://www.webrtc-experiment.com/Pluginfree-Screen-Sharing

the video constraints on video source
settings: {
"frameRate": 5,
"height": 544,
"width": 839
}

the offer from FF:
v=0
o=mozilla...THIS_IS_SDPARTA-97.0 3510005339791607615 0 IN IP4 0.0.0.0
s=-
t=0 0
a=sendrecv
a=fingerprint:sha-256 CB:B7:EC:CD:A5:8A:2A:37:4B:20:6C:3A:C5:D8:2D:56:A4:C0:32:9D:82:67:0D:87:55:CD:30:5B:C9:02:B6:19
a=ice-options:trickle
a=msid-semantic: WMS *
a=group:BUNDLE 0
m=video 49778 UDP/TLS/RTP/SAVPF 120 124 121 125 126 127 97 98
c=IN IP4 192.168.1.83
a=rtpmap:120 VP8/90000
a=rtpmap:124 rtx/90000
a=rtpmap:121 VP9/90000
a=rtpmap:125 rtx/90000
a=rtpmap:126 H264/90000
a=rtpmap:127 rtx/90000
a=rtpmap:97 H264/90000
a=rtpmap:98 rtx/90000
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1
a=fmtp:120 max-fs=12288;max-fr=60
a=fmtp:124 apt=120
a=fmtp:121 max-fs=12288;max-fr=60
a=fmtp:125 apt=121
a=fmtp:127 apt=126
a=fmtp:98 apt=97
a=rtcp:37014 IN IP4 192.168.1.83
a=rtcp-fb:120 nack
a=rtcp-fb:120 nack pli
a=rtcp-fb:120 ccm fir
a=rtcp-fb:120 goog-remb
a=rtcp-fb:120 transport-cc
a=rtcp-fb:121 nack
a=rtcp-fb:121 nack pli
a=rtcp-fb:121 ccm fir
a=rtcp-fb:121 goog-remb
a=rtcp-fb:121 transport-cc
a=rtcp-fb:126 nack
a=rtcp-fb:126 nack pli
a=rtcp-fb:126 ccm fir
a=rtcp-fb:126 goog-remb
a=rtcp-fb:126 transport-cc
a=rtcp-fb:97 nack
a=rtcp-fb:97 nack pli
a=rtcp-fb:97 ccm fir
a=rtcp-fb:97 goog-remb
a=rtcp-fb:97 transport-cc
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:4 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:5 urn:ietf:params:rtp-hdrext:toffset
a=extmap:6/recvonly http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=setup:actpass
a=mid:0
a=msid:{a8525744-a310-4155-bcf2-42bc9ca9f443} {c36be92a-1fc0-4288-851e-4332e29e042b}
a=sendonly
a=ice-ufrag:2d8291b0
a=ice-pwd:c127d09248896e5c35ea13f1323d513c
a=candidate:0 1 UDP 2122187007 192.168.1.83 49778 typ host
a=candidate:1 1 UDP 2121990399 10.42.0.1 37306 typ host
a=candidate:2 1 UDP 2122055935 172.17.0.1 49501 typ host
a=candidate:3 1 UDP 2122121471 192.168.45.12 50513 typ host
a=candidate:4 1 UDP 2122252543 2a01:e34:ec61:e380:1e8e:4483:fced:752b 37543 typ host
a=candidate:5 1 TCP 2105458943 192.168.1.83 9 typ host tcptype active
a=candidate:6 1 TCP 2105262335 10.42.0.1 9 typ host tcptype active
a=candidate:7 1 TCP 2105327871 172.17.0.1 9 typ host tcptype active
a=candidate:8 1 TCP 2105393407 192.168.45.12 9 typ host tcptype active
a=candidate:9 1 TCP 2105524479 2a01:e34:ec61:e380:1e8e:4483:fced:752b 9 typ host tcptype active
a=candidate:0 2 UDP 2122187006 192.168.1.83 37014 typ host
a=candidate:1 2 UDP 2121990398 10.42.0.1 46027 typ host
a=candidate:2 2 UDP 2122055934 172.17.0.1 36831 typ host
a=candidate:3 2 UDP 2122121470 192.168.45.12 42267 typ host
a=candidate:4 2 UDP 2122252542 2a01:e34:ec61:e380:1e8e:4483:fced:752b 49724 typ host
a=candidate:5 2 TCP 2105458942 192.168.1.83 9 typ host tcptype active
a=candidate:6 2 TCP 2105262334 10.42.0.1 9 typ host tcptype active
a=candidate:7 2 TCP 2105327870 172.17.0.1 9 typ host tcptype active
a=candidate:8 2 TCP 2105393406 192.168.45.12 9 typ host tcptype active
a=candidate:9 2 TCP 2105524478 2a01:e34:ec61:e380:1e8e:4483:fced:752b 9 typ host tcptype active
a=end-of-candidates
a=ssrc:2909849795 cname:{47184a8c-1603-4a2f-bf3e-8e90f0433dcb}
a=ssrc:1437227869 cname:{47184a8c-1603-4a2f-bf3e-8e90f0433dcb}
a=ssrc-group:FID 2909849795 1437227869
a=rtcp-mux
a=rtcp-rsize

the response from server:
v=0
o=- 1643385141411 1643385141411 IN IP4 192.168.15.215
s=-
t=0 0
a=ice-lite
m=video 20018 UDP/TLS/RTP/SAVPF 121
c=IN IP4 192.168.15.215
a=rtpmap:121 VP9/90000
a=rtcp-fb:121 ccm fir
a=rtcp-fb:121 goog-remb
a=rtcp-fb:121 nack
a=rtcp-fb:121 nack pli
a=rtcp-fb:121 transport-cc
a=rtcp-mux
a=rtcp-rsize
a=ice-ufrag:O7AfEkRBUSVxSmkm
a=ice-pwd:KDpJYCDYJI24SWqwRQClZhKJ
a=fingerprint:sha-256 B3:C4:B8:C2:6D:7C:2E:FD:6B:BF:C5:8F:AD:11:CE:B5:27:08:1B:40:58:97:C7:AF:9B:63:E9:7E:03:DD:F7:E0
a=setup:passive
a=fmtp:121 max-fs=12288;max-fr=60
a=candidate:0 1 udp 2130706431 192.168.15.215 20018 typ host
a=candidate:0 2 udp 2130706430 192.168.15.215 20018 typ host
a=candidate:1 1 tcp 2130706433 192.168.15.215 9999 typ host tcptype passive
a=candidate:1 2 tcp 2130706432 192.168.15.215 9999 typ host tcptype passive
a=recvonly

Actual results:

No video is sent by the browser.

We can see an initialization error in firefox logs ( MOZ_LOG="webrtc_trace:5,signaling:5,mtransport:5,MediaManager:5,GetUserMedia:5")

Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_encoder.cc:1150): Video frame parameters changed: dimensions=839x544, texture=0.
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (vp9_impl.cc:1708): Webrtc quality scaler for vp9 is enabled.
[Child 11941: WebrtcWorker #1]: I/signaling [WebrtcWorker #1|WebrtcVideoSessionConduit] VideoStreamFactory.cpp:204: CreateEncoderStreams Input frame 839x544, RID scaling to 839x544
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_encoder.cc:705): ReconfigureEncoder:
Simulcast streams:
0: 839x544 fps: 60 min_kbps: 200 target_kbps: 800 max_kbps: 1000 max_fps: 60 max_qp: 56 num_tl: 1 active: false
Spatial layers:
0: 839x544 fps: 5 min_kbps: 30 target_kbps: 150 max_kbps: 250 max_qp: 0 num_tl: 1 active: true
1: 839x544 fps: 10 min_kbps: 200 target_kbps: 350 max_kbps: 500 max_qp: 0 num_tl: 1 active: true

[Child 11941: WebrtcCallThread #1]: D/webrtc_trace (video_source_sink_controller.cc:76): Pushing SourceSink restrictions: max_fps=60 max_pixel_count=2147483647 target_pixel_count=null
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (vp9_impl.cc:599): Flexible mode is required for screenshare with several spatial layers
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_encoder.cc:779): Failed to initialize the encoder associated with codec type: VP9 (2)
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_adapter.cc:230): Resetting restrictions
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_encoder.cc:810): Failed to configure encoder.
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (resource_adaptation_processor.cc:132): Registered resource "EncoderUsageResource".
[Child 11941: WebrtcWorker #2]: D/webrtc_trace (bitrate_allocator.cc:523): UpdateAllocationLimits : total_requested_min_bitrate: 30 kbps, total_requested_padding_bitrate: 0 bps, total_requested_max_bitrate: 750 kbps
[Child 11941: WebrtcWorker #2]: D/webrtc_trace (pacing_controller.cc:231): bwe:pacer_updated pacing_kbps=750 padding_budget_kbps=0
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (quality_scaler.cc:222): QP thresholds: low: 149, high: 205
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (resource_adaptation_processor.cc:132): Registered resource "QualityScalerResource".
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_encoder.cc:1290): Encoder settings changed from EncoderInfo { ScalingSettings { min_pixels_per_frame = 57600 }, requested_resolution_alignment = 1, apply_alignment_to_all_simulcast_layers = 0, supports_native_handle = 0, implementation_name = 'unknown', has_trusted_rate_controller = 0, is_hardware_accelerated = 1, has_internal_source = 0, fps_allocation = [[ 1] ], resolution_bitrate_limits = [] , supports_simulcast = 0} to EncoderInfo { ScalingSettings { Thresholds { low = 149, high = 205}, min_pixels_per_frame = 57600 }, requested_resolution_alignment = 1, apply_alignment_to_all_simulcast_layers = 0, supports_native_handle = 0, implementation_name = 'libvpx', has_trusted_rate_controller = 0, is_hardware_accelerated = 0, has_internal_source = 0, fps_allocation = [[ 1] ], resolution_bitrate_limits = [] , supports_simulcast = 0}
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_encoder.cc:1442): Failed to encode frame. Error code: -7
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_encoder.cc:1712): OnBitrateUpdated, bitrate 300000 stable bitrate = 300000 link allocation bitrate = 300000 packet loss 0 rtt 0
[Child 11941: WebrtcWorker #2]: V/signaling [WebrtcWorker #2|WebrtcVideoSessionConduit] VideoConduit.cpp:1376: WebrtcVideoConduit 7f0a059b9e00 SendVideoFrame (send SSRC 2726938730 (0xa289c86a))
[Child 11941: WebrtcWorker #2]: D/webrtc_trace (video_stream_encoder.cc:1442): Failed to encode frame. Error code: -7
[Child 11941: WebrtcWorker #2]: V/signaling [WebrtcWorker #2|WebrtcVideoSessionConduit] VideoConduit.cpp:1376: WebrtcVideoConduit 7f0a059b9e00 SendVideoFrame (send SSRC 2726938730 (0xa289c86a))
[Child 11941: WebrtcWorker #2]: D/webrtc_trace (video_stream_encoder.cc:1442): Failed to encode frame. Error code: -7
[Child 11941: WebrtcWorker #2]: V/signaling [WebrtcWorker #2|WebrtcVideoSessionConduit] VideoConduit.cpp:1376: WebrtcVideoConduit 7f0a059b9e00 SendVideoFrame (send SSRC 2726938730 (0xa289c86a))
[Child 11941: WebrtcWorker #2]: D/webrtc_trace (video_stream_encoder.cc:1442): Failed to encode frame. Error code: -7
[Child 11941: WebrtcWorker #1]: V/signaling [WebrtcWorker #1|WebrtcVideoSessionConduit] VideoConduit.cpp:1376: WebrtcVideoConduit 7f0a059b9e00 SendVideoFrame (send SSRC 2726938730 (0xa289c86a))
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_encoder.cc:1442): Failed to encode frame. Error code: -7
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (probe_controller.cc:381): kWaitingForProbingResult: timeout
[Child 11941: WebrtcWorker #1]: V/signaling [WebrtcWorker #1|WebrtcVideoSessionConduit] VideoConduit.cpp:1376: WebrtcVideoConduit 7f0a059b9e00 SendVideoFrame (send SSRC 2726938730 (0xa289c86a))
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_encoder.cc:1442): Failed to encode frame. Error code: -7
[Child 11941: WebrtcWorker #1]: V/signaling [WebrtcWorker #1|WebrtcVideoSessionConduit] VideoConduit.cpp:1376: WebrtcVideoConduit 7f0a059b9e00 SendVideoFrame (send SSRC 2726938730 (0xa289c86a))
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_encoder.cc:1442): Failed to encode frame. Error code: -7

Expected results:

VP9 encoder should be successfully created and the video stream be sent

The Bugbug bot thinks this bug should belong to the 'Core::WebRTC: Audio/Video' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Component: Untriaged → WebRTC: Audio/Video
Product: Firefox → Core

Problem is reproduceable with FF 90, but not with FF 85

I see this as reproducible for 96, not 95 and earlier, so I ran mozregression on my Ubuntu 20.04 machine:

 9:34.14 INFO: Last good revision: c8fdcf75317d40a3ead539a959f748c940d0a5c9
 9:34.14 INFO: First bad revision: 1495ca5ef535f8ad692a3a579ca42eddc14f39a8
 9:34.14 INFO: Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=c8fdcf75317d40a3ead539a959f748c940d0a5c9&tochange=1495ca5ef535f8ad692a3a579ca42eddc14f39a8

 9:36.07 INFO: ************* Switching to autoland by process of elimination (no branch detected in commit message)
 9:36.85 ERROR: Unable to exploit the merge commit. Origin branch is mozilla-central, and the commit message for 1495ca5e was:
Bug 1729367 - P7 - restore mac PID tracking using new API;r=mjf a=webrtc-update

Differential Revision: https://phabricator.services.mozilla.com/D129721

This appears to be fallout from landing the libwebrtc update.

Updated STR:

  1. Go to https://www.webrtc-experiment.com/Pluginfree-Screen-Sharing
  2. Click the "Share Your Screen" button, and allow sharing a window.
  3. After the window contents are visible in the page, click the "Copy & Share This Private Room Link" link.
  4. This opens a new tab or window.
  5. The window contents should be visible on the new tab/window.

This also reproduces on macOS. I haven't looked into the info in the logs.

:jib, any thoughts?

Severity: -- → S3
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(jib)
Priority: -- → P2

[Child 11941: WebrtcWorker #1]: D/webrtc_trace (vp9_impl.cc:599): Flexible mode is required for screenshare with several spatial layers
[Child 11941: WebrtcWorker #1]: D/webrtc_trace (video_stream_encoder.cc:779): Failed to initialize the encoder associated with codec type: VP9 (2)

This appears to be here. Has this been fixed upstream?

Simulcast streams:
0: 839x544 fps: 60 min_kbps: 200 target_kbps: 800 max_kbps: 1000 max_fps: 60 max_qp: 56 num_tl: 1 active: false
Spatial layers:
0: 839x544 fps: 5 min_kbps: 30 target_kbps: 150 max_kbps: 250 max_qp: 0 num_tl: 1 active: true
1: 839x544 fps: 10 min_kbps: 200 target_kbps: 350 max_kbps: 500 max_qp: 0 num_tl: 1 active: true

I googled and found this https://webrtchacks.com/chrome-vp9-svc/ - Could the combo of VP9 and a screen-sharing source be triggering some unexpected upstream code here?

Flags: needinfo?(jib) → needinfo?(docfaraday)
Keywords: regression
Regressed by: 1654112

Unsure if this is fixed upstream, but we could just set that bool (vp9_settings.flexibleMode) to true in here:

https://searchfox.org/mozilla-central/rev/09819cb9d36b7052d715cbfe392216dcb5c0709e/dom/media/webrtc/libwebrtcglue/VideoConduit.cpp#210-225

We could make that contingent on both |is_screencast|, and the length of |aConfig.mEncodings|, although I wonder if there are more situations where we would want to set this.

Flags: needinfo?(docfaraday)

Set release status flags based on info from the regressing bug 1654112

status-firefox96: --- → affected
status-firefox97: --- → affected
status-firefox98: --- → affected
status-firefox-esr91: --- → unaffected
Has Regression Range: --- → yes

Jan-Ivar, can we get somebody assigned to this?

Flags: needinfo?(jib)
Webcompat Priority: --- → ?

This is only breaking a lab site, and we're not yet aware of any production apps that break. So this is a P3 for us at the moment.

Webcompat Priority: ? → P3

This breaks some stuff with our customers... We have to use a workaround to use VP8 instead when using screensharing.

Still seeing this even with the lib updates.

Blocks: webrtc-triage
Flags: needinfo?(jib)
Flags: needinfo?(mfroman)
Flags: needinfo?(mfroman)
Severity: S3 → S2
Summary: webrtc: fail to share screen with VP9 → WebRTC fails to share the screen with VP9

Andreas, can you take a look? Looks like VP9 outgoing is broken.

Flags: needinfo?(apehrson)

I seem able to reproduce this using https://jsfiddle.net/jib1/02j8q6h9/177/ and these steps:

  1. Leave ☑ vp9 checked
  2. Click the gDM! button and pick something to screen share (e.g. Entire Desktop) and allow

Expected result (works with vp8):

  • remote video appears (the one on the left) and stats flow

Actual result:

  • remote video and stats never appear
Assignee: nobody → apehrson
Flags: needinfo?(apehrson)
Status: NEW → ASSIGNED
No longer blocks: webrtc-triage

By just turning on flexible mode we are still failing this DCHECK because we have 2 spatial layers.

Attachment #9307356 - Attachment is obsolete: true
Pushed by pehrsons@gmail.com: https://hg.mozilla.org/integration/autoland/rev/30d2f71c3f6b Test that a screencapture track can be encoded by all supported codecs. r=bwc https://hg.mozilla.org/integration/autoland/rev/5a5d382dd664 Never configure SVC layers for VP9 screencaptures. r=bwc,jib

https://hg.mozilla.org/mozilla-central/rev/30d2f71c3f6b
https://hg.mozilla.org/mozilla-central/rev/5a5d382dd664

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
status-firefox109: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 109 Branch

Comment on attachment 9307579 [details]
Bug 1752559 - Never configure SVC layers for VP9 screencaptures. r?bwc!

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Very low-risk fix for a (non-default) encoding mode of a screen capture video track
  • User impact if declined: We won't send any video when applications try to use VP9 (non-default) to encode screen capture video tracks
  • Fix Landed on Version: 109
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Simple configuration change
Attachment #9307579 - Flags: approval-mozilla-esr102?
Attachment #9307354 - Flags: approval-mozilla-esr102?

Comment on attachment 9307354 [details]
Bug 1752559 - Test that a screencapture track can be encoded by all supported codecs. r?bwc!

Approved 102.7esr.

Attachment #9307354 - Flags: approval-mozilla-esr102? → approval-mozilla-esr102+
Attachment #9307579 - Flags: approval-mozilla-esr102? → approval-mozilla-esr102+
Flags: qe-verify+

Reproduced the issue on Firefox 98.0a1 (2022-01-28) under macOS 11.7.2 and Ubuntu 22.04 by following the STR from Comment 13.

The issue is fixed on Firefox 109.0 and Firefox 102.7.0esr. Tests were performed on macOS 11.7.2, Ubuntu 22.04 and Windows 10.

Status: RESOLVED → VERIFIED
status-firefox109: fixed → verified
status-firefox-esr102: fixed → verified
Flags: qe-verify+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: