1249098 - Firefox always sends full band Opus audio even though the negotiation specifies wideband audio

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P1)

Description

[rdaware]

When WebRTC negotiates the Opus audio with Firefox version 44, FF will always send FB (full band) stereo audio, even though the negotiation specifies WB (wide band) mono.
The first byte (TOC header) will have the value of 0x7c, instead of the 0x48 value that we expected.

Our endpoint sends the following SDP answer to FF. 
m=audio 9000 RTP/SAVPF 111\r\n
a=rtpmap:111 opus/48000/2\r\n
a=ptime:20\r\n
a=maxptime:20\r\n
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\n
a=setup:passive\r\n
a=rtcp-mux\r\n
a=sendonly\r\n
a=fmtp:111 minptime=20
a=fmtp:111 maxplaybackrate=16000; sprop-maxcapturerate=16000; maxaveragebitrate=40000; useinbandfec=1; usedtx=1

This problem doesn’t exist in FF version 43 or earlier versions.

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

The change in 44 was due to changing the getUserMedia/MSG sample rate from 16Khz to 32Khz

Comment 2

[Maire Reavy [:mreavy] Plz needinfo]

We were aware of this bug, but it has been low priority because it had no "real world" impacts.  A compliant Opus decoder will decode FB even if it expects WB.  IIUC the reason this bug has become an issue is because a  legacy device is using a very old version of Opus (a version prior to 1.0).  I need to know how common this device is in the field?  Is this a device that's just in a lab?  Or is it still popular in the field?  How likely is it that those devices will try to talk with Cisco Spark outside of a lab?

Comment 3

[duanpei]

The impact to our servers is big. For our audio servers which support 16 kHz sampling, we have to do Opus-decoding, down-sampling and Opus-encoding again. The quality could be an issue. The biggest impact is the CPU cost in our servers.

Comment 4

[Byron Campen [:bwc]]

Attachment: MozReview Request: Bug 1249098: Support maxplaybackrate for opus. r=jesup

Review commit: https://reviewboard.mozilla.org/r/35533/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/35533/

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8720978 [details]
MozReview Request: Bug 1249098: Support maxplaybackrate for opus. r=jesup

https://reviewboard.mozilla.org/r/35533/#review32201

::: media/webrtc/signaling/src/sdp/SdpAttribute.h:1244
(Diff revision 1)
> +      maxplaybackrate(48000)

Can we hoist this to a #define/etc somewhere?  We have that here and in CodecConfig.h and maybe elsewhere

Comment 6

[Byron Campen [:bwc]]

Comment on attachment 8720978 [details]
MozReview Request: Bug 1249098: Support maxplaybackrate for opus. r=jesup

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/35533/diff/1-2/

Comment 7

[Byron Campen [:bwc]]

Comment on attachment 8720978 [details]
MozReview Request: Bug 1249098: Support maxplaybackrate for opus. r=jesup

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/35533/diff/2-3/

Comment 8

[Byron Campen [:bwc]]

Can you verify that one of the builds here does what you expect?

http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-3f5f2ff13beff674e444f3a73663da296d2727eb/

Comment 9

[Maire Reavy [:mreavy] Plz needinfo]

Hi Paul -- Please see Comment 8.  We want to verify that this fix resolves the problem.  Any help you can give Rahul and his team to get this verified ASAP is much appreciated. Thanks.

Comment 10

[duanpei]

We are doing verification now. Will post the results shortly.

Comment 11

[Paul E. Jones]

Sure, I'll request that this is checked.

Comment 12

[rdaware]

We tested on Firefox nightly 47.0a1 (2016-02-25) version.
Firefox still sends out full band opus audio when we negotiate for wide band audio.

Comment 13

[Maire Reavy [:mreavy] Plz needinfo]

(In reply to rdaware from comment #12)
> We tested on Firefox nightly 47.0a1 (2016-02-25) version.
> Firefox still sends out full band opus audio when we negotiate for wide band
> audio.

The candidate fix is not in Nightly yet.  You need to test with the build Byron referenced in Comment 8.

Comment 14

[rdaware]

I tested with build given by Byron. We are able to receive wide band audio but we are receiving stereo even though we are negotiating for mono.

Comment 15

[Byron Campen [:bwc]]

(In reply to rdaware from comment #14)
> I tested with build given by Byron. We are able to receive wide band audio
> but we are receiving stereo even though we are negotiating for mono.

Mono can be a separate bug.

Comment 16

[Maire Reavy [:mreavy] Plz needinfo]

I agree with Byron that mono can be a separate bug, but we (the collective "we") are running out of time in the Fx47 cycle for any additional work -- whether in this bug or a separate one.  

Rahul/Paul -- How important is it to support negotiating and sending mono?  Do you *need* that ASAP as well?  Is there anything else you need related to negotiating Opus?  I need firm answers to these questions immediately.  Thanks.

Comment 17

[Paul E. Jones]

What the team is telling me is that the existing clients cannot play stereo audio to the user.  Would it be possible to force mono and then resolve the stereo/mono issue as another bug?

Comment 18

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6e6aaff5337b

Comment 19

[duanpei]

From our tests, in the previous version, FF43, the default channels is set to Mono but in FF 44, it is set to Stereo. This part of code must also have been changed from FF 43 to FF 44.

Comment 20

[Randell Jesup [:jesup] (needinfo me)]

(In reply to duanpei from comment #19)
> From our tests, in the previous version, FF43, the default channels is set
> to Mono but in FF 44, it is set to Stereo. This part of code must also have
> been changed from FF 43 to FF 44.

We increased the bandwidth allocated to Opus around then; and the Opus encoder will cut features (like Stereo) if the bandwidth is too low.  I don't think we changed the codec channels setting.

Comment 21

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/6e6aaff5337b

Comment 22

[Byron Campen [:bwc]]

Comment on attachment 8720978 [details]
MozReview Request: Bug 1249098: Support maxplaybackrate for opus. r=jesup

Approval Request Comment
[Feature/regressing bug #]:

   Bug 953265 (sort of)

[User impact if declined]:

   Wasted bandwidth when interoping with endpoints that cannot make use of full-band opus, or even failure to interop with endpoints that cannot handle full-band opus.

[Describe test coverage new/current, TreeHerder]:

   There are some new unit-tests that cover the signaling aspects.

[Risks and why]: 

   Fairly low, mostly rote modifications, although a fair number.

[String/UUID change made/needed]:

   None.

Comment 23

[Liz Henry (:lizzard)]

Comment on attachment 8720978 [details]
MozReview Request: Bug 1249098: Support maxplaybackrate for opus. r=jesup

It would be good to fix this bandwidth issue, ok to uplift to aurora. 
Some extra test coverage added

Comment 24

[Wes Kocher (:KWierso)]

I'm hitting conflicts trying to apply this to aurora. Can we get a rebased patch?

Comment 25

[Byron Campen [:bwc]]

Attachment: (aurora backport) Support maxplaybackrate for opus

aurora backport

Comment 26

[Wes Kocher (:KWierso)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/40b47939ca7c