Opus payload type mis-match results in broken audio

RESOLVED FIXED in Firefox 68

Status

()

defect
P3
normal
Rank:
25
RESOLVED FIXED
3 years ago
2 months ago

People

(Reporter: mpietras, Assigned: bwc)

Tracking

47 Branch
mozilla68
Points:
---

Firefox Tracking Flags

(firefox68 fixed)

Details

Attachments

(4 attachments)

Reporter

Description

3 years ago
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36

Steps to reproduce:

Create bidirectional WebRTC audio with Firefox.  SDP offer from Firefox looks like this:

v=0
o=mozilla...THIS_IS_SDPARTA-47.0.1 6228214425744129508 0 IN IP4 0.0.0.0
s=-
t=0 0
a=fingerprint:sha-256 BB:F9:EA:B8:58:7E:58:4E:70:D2:4C:78:85:88:73:D7:A6:BC:A2:34:30:BC:FA:0B:20:55:2B:91:5B:00:6A:FD
a=ice-options:trickle
a=msid-semantic:WMS *
m=audio 9 UDP/TLS/RTP/SAVPF 109 9 0 8
c=IN IP4 0.0.0.0
a=sendrecv
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=fmtp:109 maxplaybackrate=48000;stereo=1
a=ice-pwd:5c2fb89eab2fc7e60867511899e16e71
a=ice-ufrag:b9babff0
a=mid:sdparta_0
a=msid:{29755e16-b28c-4233-a32b-1aa5c27f612e} {f9856fff-a400-4bf2-9446-405466efb014}
a=rtcp-mux
a=rtpmap:109 opus/48000/2
a=rtpmap:9 G722/8000/1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=setup:actpass
a=ssrc:2987989451 cname:{72c3a255-1c35-4a66-b75c-768a3c8f3557}

Note that the payload type for Opus is 109.

SDP Answer looks like this:

v=0
o=company 2547 2548 IN IP4 209.236.99.123
s=company
t=0 0
m=audio 40074 RTP/SAVPF 102 9 0 8 101 
c=IN IP4 209.236.99.123
a=ice-lite
a=ice-ufrag:hcE2w84B
a=ice-pwd:iMc9m4dLuTf4ONhiFUsEwmzOzeQ5mVHE
a=candidate:704553097 1 udp 2113937151 209.236.99.123 40074 typ host generation 0
a=fingerprint:sha-256 34:13:BB:CB:97:F6:EA:15:4D:FF:12:82:A2:45:BA:0B:1E:64:D7:E4:A9:A2:BE:22:DA:B5:BC:9C:2C:16:11:B0
a=rtpmap:102 opus/48000/2
a=ptime:20
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
a=sendrecv
a=rtcp-mux

Note that the Opus payload type is 102.  There is a fixes bug on non-matching payload types https://bugzilla.mozilla.org/show_bug.cgi?id=1221473 which if I'm reading it right seems to indicate it went into 41 and I'm testing with 47.0.1.



Actual results:

Opus audio sent from Firefox is Opus with payload type 102 and plays correctly at receiving end.  Opus audio sent to Firefox is Opus with payload type 109 and plays all garbled.

Note that this exact setup plays correctly bidirectionally with Chrome.


Expected results:

Opus should have played back cleanly.

One thing I note is in the WebRTC log it seems like there is a completely different payload type set internally:

19236; Channel::Init() L16 (107/8000/1/128000) has been added to the RTP/RTCP receiver
19236; Channel::Init() L16 (108/16000/1/256000) has been added to the RTP/RTCP receiver
19236; Channel::Init() L16 (109/32000/1/512000) has been added to the RTP/RTCP receiver
19236; Channel::Init() L16 (111/8000/2/128000) has been added to the RTP/RTCP receiver
19236; Channel::Init() L16 (112/16000/2/256000) has been added to the RTP/RTCP receiver
19236; Channel::Init() L16 (113/32000/2/512000) has been added to the RTP/RTCP receiver
19236; Channel::Init() PCMU (0/8000/1/64000) has been added to the RTP/RTCP receiver
19236; Channel::SetSendCodec()
19236; Channel::Init() PCMA (8/8000/1/64000) has been added to the RTP/RTCP receiver
19236; Channel::Init() PCMU (110/8000/2/64000) has been added to the RTP/RTCP receiver
19236; Channel::Init() PCMA (118/8000/2/64000) has been added to the RTP/RTCP receiver
19236; Channel::Init() G722 (9/16000/1/64000) has been added to the RTP/RTCP receiver
19236; Channel::Init() G722 (119/16000/2/64000) has been added to the RTP/RTCP receiver
19236; Channel::Init() opus (120/48000/2/64000) has been added to the RTP/RTCP receiver
19236; Channel::Init() CN (13/8000/1/0) has been added to the RTP/RTCP receiver
19236; Channel::Init() CN (98/16000/1/0) has been added to the RTP/RTCP receiver
19236; Channel::Init() CN (99/32000/1/0) has been added to the RTP/RTCP receiver
19236; Channel::Init() telephone-event (106/8000/1/0) has been added to the RTP/RTCP receiver

That seems to indicate the Opus payload type is expected to be 120 (not 109) that was signaled in the Offer.  I'm wondering if Firefox is interpreting the data as L16 (indicated as payload type 109 in the receiver list)?

Individual packet processing seems to go correctly:

16148; ExternalPlayoutGetData(speechData10ms=0x1f58eab8, samplingFreqHz=32000,  current_delay_ms=0)
16148; Channel::NeededFrequency(id=2)
16148; ReceiveFrequency()
16148; PlayoutFrequency()
16148; UpdateToMix(mixList,rampOutList,mixParticipantList,3)
16148; Channel::GetAudioFrame(id=2)
16148; (neteq_impl.cc:163): GetAudio
16148; (decision_logic.cc:131): Buffers: 0 packets * 22 samples/packet + 94 samples in sync buffer = 94
16148; (neteq_impl.cc:699): GetDecision returned operation=2 and 0 packet(s)
16148; (neteq_impl.cc:812): Sync buffer (1 channel(s)): insert 0 samples, extract 320 samples
16148; (neteq_impl.cc:166): Produced 320 samples/channel for 1 channel(s)
16148; PlayoutFrequency()
16148; GetAdditionalAudio(additionalFramesList)
16148; UpdateMixedStatus(mixedParticipantsMap)
16148; MixFromList(mixedAudio, audioFrameList)
16148; MixAnonomouslyFromList(mixedAudio, audioFrameList)
16148; MixAnonomouslyFromList(mixedAudio, audioFrameList)
16148; UpdateVADPositiveParticipants(mixList)
16148; OutputMixer::NewMixedAudio(id=2, size=0)
16148; ClearAudioFrameList(audioFrameList)
16148; ClearAudioFrameList(audioFrameList)
16148; ClearAudioFrameList(audioFrameList)
16148; OutputMixer::GetMixedAudio(sample_rate_hz=32000, num_channels=1)

but there's no indication (to me) about what codec was used for decode.

Updated

3 years ago
Component: Untriaged → WebRTC: Signaling
Product: Firefox → Core
Just to be clear: I'm assuming your actual bug report is that the audio is garbled. Correct?
Reporter

Comment 2

3 years ago
Yes, correct.  I was just trying to dig up as much info as I could, and found what looked to me like could be an issue.
Summary: WebRTC SDP offer payload type for Opus doesn't match internal WebRTC log → Opus payload type mis-match results in broken audio
Is that the *entire* SDP?  (i.e. was video offered?  Was it accepted or rejected in the answer?)  If not, can you give the entire SDP?

I'll note this because offers from Firefox will default to using 120 for VP8.  What is generating the logs you're dumping?  If video was offered, did the answer have the audio and video m= lines in the same order?

The Answer has a small violation of spec in that you added a payload in the answer (101/telephone-event), though that shouldn't have any impact on this problem

Can you save about:webrtc (there's a button to save it) and attach that here?

Also, if possible, start firefox with the env variable NSPR_LOG_MODULES set to signaling:4 and NSPR_LOG_FILE set to a temp file, and upload that log after running the test.  Thanks
Flags: needinfo?(mpietras)
Whiteboard: [Needinfo 2016/07/22]
Reporter

Comment 4

3 years ago
Yes that is the entire SDP, video was not offered.  I copied that from the developer console... I dumped the SDP to the debug console in the success callback of the peer-connection createOffer.

I'm not sure how the 101/telephone-event is a violation of the spec... you mean WebRTC spec?

The logs were done with Firefox environment variable set with signaling 5 which I think is more verbose:
set MOZ_LOG=signaling:5,mtransport:5;webrtc_trace:65535;MediaManager:5,GetUserMedia:5

Unfortunately when I did environment variables for logging, did the about:webrtc, cleared history and log, started debug mode and AEC Logging, did my WebRTC call, Firefox faulted.  It generated a small seemingly useless file GeckoChildCrash14048.extra with the contents:

SystemMemoryUsePercentage=23
TotalVirtualMemory=4294836224
AvailableVirtualMemory=3835199488
TotalPageFile=39331602432
AvailablePageFile=31265021952
TotalPhysicalMemory=34231328768
AvailablePhysicalMemory=26217148416

I'll play with it some and see what more I can get.
Flags: needinfo?(mpietras)
Reporter

Comment 5

3 years ago
So it looks like the AEC logging is what faulted it.  I turned that off, I'm attaching WebRTC_1.log now.
Reporter

Comment 6

3 years ago
(In reply to mpietras from comment #4)
> I'm not sure how the 101/telephone-event is a violation of the spec... you
> mean WebRTC spec?

Just to address this separately: telephone-event is not violating the spec by itself. But the fact that you added a codec into the answer which was not offered by FF is a spec violation.
This looks to me like in fact the incoming packets with payload type 109 are treated as L16 packets:

Channel::OnInitializeDecoder(id=131072, payloadType=109, payloadName=L16, frequency=32000, channels=1, rate=512000)
(neteq_impl.cc:182): webrtc::NetEqImpl::RegisterPayloadType: static_cast<int>(rtp_payload_type)=109, codec=10
Status: UNCONFIRMED → NEW
backlog: --- → webrtc/webaudio+
Rank: 25
Ever confirmed: true
Priority: -- → P2
Whiteboard: [Needinfo 2016/07/22]
Reporter

Comment 9

3 years ago
(In reply to Nils Ohlmeier [:drno] from comment #7)
> (In reply to mpietras from comment #4)
> > I'm not sure how the 101/telephone-event is a violation of the spec... you
> > mean WebRTC spec?
> 
> Just to address this separately: telephone-event is not violating the spec
> by itself. But the fact that you added a codec into the answer which was not
> offered by FF is a spec violation.

Right, OK thanks, good catch.
Reporter

Comment 10

3 years ago
Wondering if there's a known workaround for this issue or if we should continue to use just G.711?  I'd much rather be using Opus of course...
Mass change P2->P3 to align with new Mozilla triage process.
Priority: P2 → P3

Comment 12

3 months ago

I am facing a similar issue with latest stable and nightly build. when payload type in local and remote SDP does not match, firefox is playing nothing.

SDP
Local SDP (Offer)

v=0
o=mozilla...THIS_IS_SDPARTA-67.0 702656407228142757 0 IN IP4 0.0.0.0
s=-
t=0 0
a=sendrecv
a=fingerprint:sha-256 A3:71:3B:36:68:DA:F7:E4:ED:83:1D:19:E4:60:98:07:8D:29:5C:09:72:FC:05:62:A9:E7:A9:90:29:7F:54:35
a=group:BUNDLE 0
a=ice-options:trickle
a=msid-semantic:WMS *
m=audio 63814 UDP/TLS/RTP/SAVPF 109 9 0 8 101
c=IN IP4 xxx.xxx.xxx.xxx
a=candidate:0 1 UDP 2122252543 xxx.xxx.xxx.xxx 63814 typ host
a=candidate:1 1 UDP 2122187007 xxx.xxx.xxx.xxx 49917 typ host
a=candidate:2 1 TCP 2105524479 xxx.xxx.xxx.xxx 9 typ host tcptype active
a=candidate:3 1 TCP 2105458943 xxx.xxx.xxx.xxx 9 typ host tcptype active
a=candidate:0 2 UDP 2122252542 xxx.xxx.xxx.xxx 50130 typ host
a=candidate:1 2 UDP 2122187006 xxx.xxx.xxx.xxx 56326 typ host
a=candidate:2 2 TCP 2105524478 xxx.xxx.xxx.xxx 9 typ host tcptype active
a=candidate:3 2 TCP 2105458942 xxx.xxx.xxx.xxx 9 typ host tcptype active
a=sendrecv
a=end-of-candidates
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2/recvonly urn:ietf:params:rtp-hdrext:csrc-audio-level
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=fmtp:109 maxplaybackrate=48000;stereo=1;useinbandfec=1
a=fmtp:101 0-15
a=ice-pwd:917fd51a0a7280cc59d1f10a34b45c96
a=ice-ufrag:d3488322
a=mid:0
a=msid:{6519e41c-ce45-4e4b-8791-d9ee860a8a18} {ad1f02e9-450b-6648-a19d-f8d83587092a}
a=rtcp:50130 IN IP4 xxx.xxx.xxx.xxx
a=rtcp-mux
a=rtpmap:109 opus/48000/2
a=rtpmap:9 G722/8000/1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000
a=setup:actpass
a=ssrc:2341447803 cname:{0969b691-9368-5849-9ed1-74cf0b2fddb9}

Remote SDP (Answer)

v=0
o=root 972696436 469494478 IN IP4 xxx.xxx.xxx.xxx
s=-
t=0 0
a=sendrecv
a=ice-lite
m=audio 7205 UDP/TLS/RTP/SAVPF 107 101
c=IN IP4 xxx.xxx.xxx.xxx
a=candidate:874625322 1 udp 2130706431 xxx.xxx.xxx.xxx 7205 typ host
a=sendrecv
a=fingerprint:sha-256 0A:B1:D8:D4:33:E9:0B:BB:2E:6D:08:E7:D9:77:D0:1D:C6:8C:62:9E:58:0A:7D:84:D4:7B:2D:79:48:47:EA:40
a=fmtp:107 maxplaybackrate=0;stereo=0;useinbandfec=1
a=fmtp:101 0-16
a=ice-pwd:MH2Ej7aZqnHtqmbsFoL1pp2H
a=ice-ufrag:pFpYkDmxFDza9nMU
a=maxptime:20
a=mid:0
a=ptime:20
a=rtcp-mux
a=rtpmap:107 opus/48000/2
a=rtpmap:101 telephone-event/8000
a=setup:active
a=ssrc:2575757703 cname:JSz91vC3l3UI1/et

RTP Stats:

inbound_rtp_audio_1

Local: 17:30:12 GMT+0800 (Hong Kong Standard Time) inbound-rtp SSRC: 1964845081 Received: 269 packets (34.77 Kb) Lost: 0 Jitter: 0
outbound_rtp_audio_0

Local: 17:30:12 GMT+0800 (Hong Kong Standard Time) outbound-rtp SSRC: 2341447803 Sent: 274 packets (24.93 Kb)
Assignee

Updated

3 months ago
Assignee: nobody → docfaraday

Comment 19

2 months ago
Pushed by bcampen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e7003b05592a
Part 0: Add some unit-tests related to payload type asymmetry. r=mjf
https://hg.mozilla.org/integration/autoland/rev/f06d16709069
Part 1: Do not set recv tracks' payload types based on the remote SDP. Some related simplifications/fixes. r=mjf
https://hg.mozilla.org/integration/autoland/rev/9fcab041c5d7
Part 2: Fix longstanding bug where rollback could wipe out the codecs in a transceiver, that was being hidden by a bug fixed in part 1. r=mjf
You need to log in before you can comment on or make changes to this bug.