1294405 - WebRTC H264 profile sets the "set2" constraint, which isn't needed and causes pain in negotiation

Status: NEW

(Core :: WebRTC: Signaling, defect, P3)

Description

[Randell Jesup [:jesup] (needinfo me)]

We have some interop issues with H264 negotiation to resolve.

We're using 42E01F for profile-level-id for H264 in WebRTC.  Breaking that down, it's Constrained Baseline (42C0) + the set2 constraint (Extended Profile) and Level 3.1.  No one uses Extended Profile, and it really doesn't add anything here.  There may be a couple of subtle differences in the SPS/PPS for 42E0, but most of the Extended differences apply only to B-frames.  (A.2.3 in the H264 spec)

It does however complicate negotiation due to how profiles are handled - if someone else supports normal Constrained Baseline (42C0), and we support Constrained Baseline Extended, it will fail matching in the RFC 6184 rules.  Combine this with the face that many people haven't followed all the rules (and we are), and things get confusing for interop.

For example, see https://blogs.windows.com/msedgedev/2016/05/27/previewing-h-264avc-for-ortc/

We inherited the 42E0xx part from the initial import of SIPCC, so when ehugg turned on H264 negotiation, it came along with no explanation (I checked old trees before we ripped that code out).

Solution:
1) Modify our negotiation code when generating answers to accept 42E0 or 42C0 and return what was offered (this is fine to do spec-wise).  
2) Modify it to accept an answer of 42C0 for a payload where we have 42E0 (non-compliant spec-wise, but that would already be a non-compliant answer).
3) Offer all H264 payloads in two variants: 42C0 and 42E0 (42c0 first).  This would let us interop with existing versions of Firefox, since they would refuse to answer if 42C0 if the only version offered.  So it would look like this:

m=video 12345 UDP/TLS/RTP/SAVPF 120 126 97 125 98
a=fmtp:126 profile-level-id=42c01f;level-asymmetry-allowed=1;packetization-mode=1
a=fmtp:97 profile-level-id=42c01f;level-asymmetry-allowed=1
a=fmtp:125 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=fmtp:98 profile-level-id=42e01f;level-asymmetry-allowed=1
a=fmtp:120 max-fs=12288;max-fr=60
...

Then, eventually (after current ESR releases "age out", remove 42E0 from what we offer.  Keep the rest of the mods.  We file a followup-bug to do so.

We *could* allow 4280 (non-constrained baseline) responses probably, presuming OpenH264 supports plain Baseline (probably).  I'm somewhat loath to offer it as well, since most people who would answer us with it will totally ignore the constraint bits anyways.  There's very little coding efficiency gained by Baseline over Constrained Baseline I believe.

Comment 1

[Nils Ohlmeier [:drno]]

For clarification purpose:

(In reply to Randell Jesup [:jesup] from comment #0)
> Solution:
> 1) Modify our negotiation code when generating answers to accept 42E0 or
> 42C0 and return what was offered (this is fine to do spec-wise).  
> 2) Modify it to accept an answer of 42C0 for a payload where we have 42E0
> (non-compliant spec-wise, but that would already be a non-compliant answer).

It sounds like 1 or 2. Or at least 2 is optional, as long as we implement 1 here, because it only takes care of broken answers.

> 3) Offer all H264 payloads in two variants: 42C0 and 42E0 (42c0 first). 
> This would let us interop with existing versions of Firefox, since they
> would refuse to answer if 42C0 if the only version offered.  So it would
> look like this:

And 3 is mandatory, because that covers the offer side of things.

If I understand it correctly I would suggest to implement 1 and 3 only. But I don't really have a good sense for how many clients would need 2 to interop with Firefox.

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Nils Ohlmeier [:drno] from comment #1)
> For clarification purpose:
> 
> (In reply to Randell Jesup [:jesup] from comment #0)
> > Solution:
> > 1) Modify our negotiation code when generating answers to accept 42E0 or
> > 42C0 and return what was offered (this is fine to do spec-wise).  
> > 2) Modify it to accept an answer of 42C0 for a payload where we have 42E0
> > (non-compliant spec-wise, but that would already be a non-compliant answer).
> 
> It sounds like 1 or 2. Or at least 2 is optional, as long as we implement 1
> here, because it only takes care of broken answers.

2 is optional, but from experience more implementations than not totally ignore the profile we offer and just send back their default.  We'll get a lot less confused users if we do that (at the cost of not forcing them to fix their impls - we could complain in the console, though that can be annoying to do).  And that still won't fix people that send back 4280 or 4200, for example.

There is something to be said for forcing people to fix it, of course.

Comment 3

[mzanaty]

H.264 profile negotiation can certainly cause confusion and interop issues. The issue here is Firefox fails to match compatible profiles when creating or receiving answers. There is no issue in creating offers, profile-level-id=42E01F is perfectly fine for interop and spec compliance.

First, a cheat sheet for this profile mess.

BP = Baseline Profile, CBP = Constrained BP, MP = Main Profile, XP = Extended Profile
Other acronyms are tools in the profiles.

Constraint Set 0 = BP = CBP + (FMO,ASO,RS)
Constraint Set 1 = MP = CBP + (AC,IL,BI)
Constraint Set 2 = XP =  BP + (DP,IL,BI,SI,SP)

Signaling multiple constraint sets means the common intersection of them all. So 0+1=CBP=0+1+2, since their common intersection is CBP. Signaling 42E0XX means BP (42) with BP+MP+XP (0+1+2) constraints, which is CBP. So is 4240XX and 42C0XX.

Table 5 in RFC 6184 is the spec for matching compatible profiles.
https://tools.ietf.org/html/rfc6184#section-8.1

Firefox could update its rules to match the spec when creating or receiving answers. But that may still cause interop issues, since many others may signal 4200XX or 4280XX, which is BP not CBP, even if they can't do full BP (with FMO,ASO,RS). While they are technically out of spec, please don't force other people to fix this, just be lenient and accept 42XXXX. That is the best solution for interop.

TLDR: Offer 42E0XX (no change) or 42C0XX or 4240XX (gratuitous change), all are the same. Accept any 42XXXX offer or answer. Create answers that echo the 42XXXX offer. This will interop best with all peers whether they are in or out of spec.

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

Profile-level-id matching *is* hairy, but I'm not sure we're reading the table the same way.

I always have read this:
    //        CB          42 (B)             x1xx0000
    //           same as: 4D (M)             1xxx0000
    //           same as: 58 (E)             11xx0000
as meaning 42+x1xx is the same as 4D+1xxx is the same as 58+11xx.

It also means (as you say) that 42E0 should be the "same" as 42C0, and in fact that is what we've implemented:
https://dxr.mozilla.org/mozilla-central/source/media/webrtc/signaling/src/jsep/JsepCodecDescription.h#548
If I modify pc_test.html to replace 42e0 in the offer with 42c0, we accept it (with 42e0) and calls work.

I mis-read some reports as our not handling this correctly (and probably mis-read A.2.3 a bit; it wasn't clear to me that the result of these intersections was 0 difference).  So we don't *have* to change, but 42C0 may improve our interop a bit (due to other's broken negotation code), as commonly people use 42C0 for CBP.

As you also say (and what triggered me to look at this) is that quite a few out there use 4280 (Baseline) and assume that this is no different than CBP.  As you say (and as I indicate in #3 above) answering 4280's with 4280 will help (even answering with 42e0 would likely help - most of them don't check the profile).

Since we do match 42e0 with 42c0 correctly, we *could* switch to 42c0 at any time without going through the rigamarole I indicated with duplicate payloads; we'd be able to talk to any older Firefox.  We probably should however mirror the offer even when they "match" - answer 42c0 with 42c0 not 42e0 (again, to help non-compliant impls).

So the remaining actions would be:
1) answer using the same 42xx as was offered
2) perhaps switch offers to 42c0
3) probably accept 42xx and echo it (i.e. 4280) even though out-of-spec (makes #1 moot)
4) complain in browser console (and signaling logs) about out-of-spec offers

H264 profiles make my head hurt.... ;-)

Comment 5

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.