2019841 - Some wpt 2pass subtests incorrectly excluded from and some less2pass included in interop-2026

Status: RESOLVED FIXED

(Core :: WebRTC, task)

Description

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: image.png

With wpt.fyi having updated, hindsight is improved, and I either found a couple subtests I missed (or some of these are intermittent?):

count=2(status:pass) ?rest:

• RTCPeerConnection-addIceCandidate.html?rest

  • addIceCandidate after close should reject with InvalidStateError
  • addIceCandidate should not recognize relayProtocol or url

• RTCPeerConnection-getStats.https.html?rest

  • getStats() with track not added to connection should reject with InvalidAccessError
  • getStats() with track associated with both sender and receiver should reject with InvalidAccessError
  • getStats() audio contains inbound-rtp stats
  • getStats(track) should not work if multiple senders have the same track
  • getStats succeeds on a closed peerconnection

• RTCPeerConnection-iceGatheringState.html?rest

  • renegotiation that closes all transports should result in ICE gathering state "new"

• RTCRtpParameters-codec.html?rest

  • <nothing>

The last one changed between when I first pasted the top list (see screenshot) to now as I'm typing in this description hours later.

This suggests that the results are moving. Might be intermittent results?

Or incorrect wpt.fyi filtering behavior... e.g. https://wpt.fyi/results/webrtc/RTCRtpParameters-codec.html%3Frest?label=master&product=firefox&product=chrome&product=safari&q=%3Frest%20count%3D2%28status%3Apass%29%20 shows me less2pass and 3pass...

Comment 1

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Also, conversely

count<2(status:pass) ?interop-2026 (addIceCandidate details):

• RTCPeerConnection-addIceCandidate.html?interop-2026
  • addIceCandidate({"candidate":"","sdpMid":null,"sdpMLineIndex":null}) adds a=end-of-candidates to both m-sections
  • addIceCandidate(undefined) adds a=end-of-candidates to both m-sections
  • addIceCandidate(null) adds a=end-of-candidates to both m-sections
  • addIceCandidate({}) adds a=end-of-candidates to both m-sections
  • addIceCandidate({}) in stable should work, and add a=end-of-candidates to both m-sections
  • addIceCandidate({usernameFragment: usernameFragment1, sdpMid: sdpMid1}) should work, and add a=end-of-candidates to the first m-section
  • addIceCandidate({usernameFragment: usernameFragment2, sdpMLineIndex: 1}) should work, and add a=end-of-candidates to the first m-section
  • addIceCandidate({usernameFragment: "no such ufrag"}) should not work
  • Add with empty candidate string (end of candidates) should succeed
  • Add candidate with invalid usernameFragment should reject with OperationError
  • Add candidate with sdpMid belonging to different usernameFragment should reject with OperationError

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: image2.png

This is interesting: the count=2(status:pass) ?rest results from comment 0 are intermittent. I'm going to start documenting this. (see image below)

RTCDataChannel-GC.html?rest

• Control: detected remote PC being closed using a data channel

RTCRtpTransceiver.https.html?rest

• track with audio gets unmuted when packets flow.

Comment 3

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: image3.png

Comment 4

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: image4.png

• RTCPeerConnection-remote-track-properties.https.html?rest
  • Remote audio track ID is different on different PCs
  • Remote video track ID is different on different PCs

Comment 5

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: image5.png

Comment 6

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

RTCRtpTransceiver-setCodecPreferences.html

• setCodecPreferences should accept audio codecs regardless of mimeType case
• setCodecPreferences should accept video codecs regardless of mimeType case

Comment 7

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 2019841 - Calibrate which webrtc subtests are part of interop-2026 based on 2pass. r?bwc

Comment 8

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Another observation from ?rest is that these 9 100% passing ?rest tests did not need splitting:

RTCDataChannel-GC.html?rest 			1 / 1 	1 / 1 	1 / 1
RTCDtlsTransport-state.html?rest 		1 / 1 	1 / 1 	1 / 1
RTCDTMFSender-ontonechange.https.html?rest	11 / 11	11 / 11	11 / 11
RTCPeerConnection-constructor.html?rest 	12 / 12	12 / 12	12 / 12
RTCRtpSender-replaceTrack.https.html?rest 	9 / 9 	9 / 9 	9 / 9
RTCRtpTransceiver-stop.html?rest 		7 / 7 	7 / 7 	7 / 7
RTCSctpTransport-maxChannels.html?rest 		1 / 1 	1 / 1 	1 / 1
rtp-stats-lifetime.https.html?rest 		8 / 8 	8 / 8 	8 / 8
toJSON.html?rest 				1 / 1 	1 / 1 	1 / 1

Comment 9

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Leaving RTCSctpTransport-maxChannels.html?rest alone as I plan to add rollback subtests there for https://github.com/w3c/webrtc-pc/pull/3094.

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 2019841 - add WPT test for SctpTransport exposure and rollback during negotiation. r?bwc

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/431d8781968d
https://hg.mozilla.org/integration/autoland/rev/13f2d8eb57fa
Calibrate which webrtc subtests are part of interop-2026 based on 2pass. r=bwc
https://github.com/mozilla-firefox/firefox/commit/fb00a65e47c4
https://hg.mozilla.org/integration/autoland/rev/8588387a3413
add WPT test for SctpTransport exposure and rollback during negotiation. r=bwc

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/30a171d905d1
https://hg.mozilla.org/integration/autoland/rev/efd90137e15e
Revert "Bug 2019841 - add WPT test for SctpTransport exposure and rollback during negotiation. r=bwc" for causing wpt failures at replaceTrack.

Comment 13

[Silaghi Andreea]

Backed out for causing wpt and TVw failures at webrtc/RTCRtpSender-replaceTrack.
Backout Link
Push with failures
Failure Log
Failure Log for TVw1
Failure Log for TVw2
Failure line TEST-UNEXPECTED-TIMEOUT | /webrtc/RTCRtpSender-replaceTrack.https.html | ReplaceTrack transmits the new track not the old track - Test timed out

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/c72b576a248c
https://hg.mozilla.org/integration/autoland/rev/02c6cc15d044
Calibrate which webrtc subtests are part of interop-2026 based on 2pass. r=bwc
https://github.com/mozilla-firefox/firefox/commit/f9026cfcc0cc
https://hg.mozilla.org/integration/autoland/rev/50d04a0ff89a
add WPT test for SctpTransport exposure and rollback during negotiation. r=bwc

Comment 15

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/58432 for changes under testing/web-platform/tests

Comment 16

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/02c6cc15d044
https://hg.mozilla.org/mozilla-central/rev/50d04a0ff89a

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/d4b72688750e
https://hg.mozilla.org/integration/autoland/rev/60ed5000cb8a
[wpt PR 58432] - [Gecko Bug 2019841] Calibrate which webrtc subtests are part of interop-2026 based on 2pass + test for webrrtc-pc#3094, a=testonly

Comment 18

[rperta]

https://hg.mozilla.org/mozilla-central/rev/60ed5000cb8a