1316448 - Missing support for RTP CVO extension

Status: NEW

(Core :: WebRTC: Signaling, defect, P4)

Description

[aterziev]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0
Build ID: 20161019084923

Steps to reproduce:

1. WebRTC at iOS receives a call from WebRTC at Firefox 49.0.2
2. WebRTC at iOS answers the call and establishes a video chat with the firefox browser

Note:
WebRTC at iOS was built as static library from master at commit number# 6ceab083225dfa485bb48f9e43231c216acccbaf
Firefox was running Mac OS 10.11
Video feed is properly oriented for a call between iOS and Chrome 54

Some investigations & thoughts
Recent WebRTC builds support rtp extension for CVO ( a=extmap:4 urn:3gpp:video-orientation). SDP coming from firefox does not indicate that it supports this extension. I guess if the ff 49 uses a recent WebRTC build, it should offer this extension in its SDP





Actual results:

When the iPhone device is oriented in portrait, local video feed at ios device and remote video feed at firefox client is rotated at 270 degrees



Expected results:

When the iPhone device is oriented in portrait, local video feed at ios device and remote video feed at firefox client to be properly rotated

Comment 1

[Alex Chronopoulos [:achronop]]

This contains a custom build of WebRTC. Is it something that we can support. Also reporter investigation mentions a problem with SDP. Does it make sense?

Comment 2

[Nils Ohlmeier [:drno]]

Firefox 49 (and even all version up to 52) use webrtc.org code from
 WEBRTC_SVNVERSION 7864
 WEBRTC_PULL_DATE "Wed Dec 10 12:23:33 EST 2014"
which does not have support for CVO.

We are currently working on integrating webrtc.org branch 49. But that will probably land in Firefox 53. And I believe even that has no support for CVO.

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Note also that CVO should not be used if the extension isn't successfully negotiated; if you're using CVO without negotiation (or without checking if it succeeded, you'll get problems like this.  When CVO is not negotiated, you should rotate the captured image before encoding (or the webrtc.org code should).  Perhaps you're telling it to use CVO always, and so it isn't rotating for you.

Comment 4

[zevarito]

Any update on this?, looks like this is missing either stable/dev releases, the problem is that new versions of WebRTC seems to have dropped previous method to handle rotation in favor of CVO, so no matter if the extension is not negotiated Firefox will not be able handle rotation properly.

Comment 5

[Nils Ohlmeier [:drno]]

Looks like our current webrtc.org 49 branch has some support for CVO http://searchfox.org/mozilla-central/search?q=symbol:_ZN6webrtc12RtpExtension14kVideoRotationE&redirect=false

Randell: any idea how much effort it would be to activate this? Is it simply signaling it, like we do now with abs send time, or would it require also changes to the encoding and/or rendering side?

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

It would require a fair bit of hookup to the code outside of the webrtc.org stack, first to have this info come along from the camera (lots of layers to touch), and then to deal with handling the "should we rotate" on the reception side which involves knowing about the screen/media-element orientation.  Signaling it in the webrtc.org code is the easy part

Comment 7

[zevarito]

Randell: Do you know if it is possible somehow to fallback to the previous behavior by not signaling CVO (not enough) and tell WebRTC from clients (say IOS/Android) to send right WxH, HxW for each frame? I've posted a comment on their issue but not answer yet. Thanks.

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

(In reply to zevarito from comment #7)
> Randell: Do you know if it is possible somehow to fallback to the previous
> behavior by not signaling CVO (not enough) and tell WebRTC from clients (say
> IOS/Android) to send right WxH, HxW for each frame? I've posted a comment on
> their issue but not answer yet. Thanks.

I can't force them to send something other than not negotiating CVO, which is what we do today (and have always done).  It clearly sounds like their bug.  We might be able to negotiate CVO but never send it (or never send anything bug "0 degrees") without much more work than the SDP work and some control surfaces in VideoConduit.cpp.  Webrtc.org will receive with CVO; the trick will be to ensure when that happens the webrtc.org code does the rotate, or we'll have to do it on the OnFrame()/etc call.

Comment 9

[zevarito]

Randell: thanks for the update, I will keep digging on how this can be fixed/workaround from WebRTC side.
Please update this issue when CVO gets implemented, Thanks!

Comment 10

[Bulk Bug Changes for mreavy's org]

Mass change P3->P4 to align with new Mozilla triage process.

Comment 12

[Francesco Durighetto]

any update on this topic?

I might be wrong but supporting this extension will help mobile client to do less work (especially on vp8 codec which is not very well hardware accelerated optimized) on rotating frame when encoding.

Comment 13

[Serj Lavrin]

Gentle ping