Closed Bug 1288105 Opened 10 years ago Closed 7 years ago

Opus payload type mis-match results in broken audio

Categories

(Core :: WebRTC: Signaling, defect, P3)

Product:
Core
Component:
WebRTC: Signaling
Version:
47 Branch
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla68
Tracking Flags:
Tracking Status
firefox68 --- fixed
Blocking Flags:
backlog webrtc/webaudio+

People

(Reporter: mpietras, Assigned: bwc)

References

Details

Attachments

(1 file, 3 external files)

zip of log of call with garbled audio
502.18 KB, application/x-zip-compressed
Details
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 Steps to reproduce: Create bidirectional WebRTC audio with Firefox. SDP offer from Firefox looks like this: v=0 o=mozilla...THIS_IS_SDPARTA-47.0.1 6228214425744129508 0 IN IP4 0.0.0.0 s=- t=0 0 a=fingerprint:sha-256 BB:F9:EA:B8:58:7E:58:4E:70:D2:4C:78:85:88:73:D7:A6:BC:A2:34:30:BC:FA:0B:20:55:2B:91:5B:00:6A:FD a=ice-options:trickle a=msid-semantic:WMS * m=audio 9 UDP/TLS/RTP/SAVPF 109 9 0 8 c=IN IP4 0.0.0.0 a=sendrecv a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=fmtp:109 maxplaybackrate=48000;stereo=1 a=ice-pwd:5c2fb89eab2fc7e60867511899e16e71 a=ice-ufrag:b9babff0 a=mid:sdparta_0 a=msid:{29755e16-b28c-4233-a32b-1aa5c27f612e} {f9856fff-a400-4bf2-9446-405466efb014} a=rtcp-mux a=rtpmap:109 opus/48000/2 a=rtpmap:9 G722/8000/1 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=setup:actpass a=ssrc:2987989451 cname:{72c3a255-1c35-4a66-b75c-768a3c8f3557} Note that the payload type for Opus is 109. SDP Answer looks like this: v=0 o=company 2547 2548 IN IP4 209.236.99.123 s=company t=0 0 m=audio 40074 RTP/SAVPF 102 9 0 8 101 c=IN IP4 209.236.99.123 a=ice-lite a=ice-ufrag:hcE2w84B a=ice-pwd:iMc9m4dLuTf4ONhiFUsEwmzOzeQ5mVHE a=candidate:704553097 1 udp 2113937151 209.236.99.123 40074 typ host generation 0 a=fingerprint:sha-256 34:13:BB:CB:97:F6:EA:15:4D:FF:12:82:A2:45:BA:0B:1E:64:D7:E4:A9:A2:BE:22:DA:B5:BC:9C:2C:16:11:B0 a=rtpmap:102 opus/48000/2 a=ptime:20 a=rtpmap:9 G722/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 a=sendrecv a=rtcp-mux Note that the Opus payload type is 102. There is a fixes bug on non-matching payload types https://bugzilla.mozilla.org/show_bug.cgi?id=1221473 which if I'm reading it right seems to indicate it went into 41 and I'm testing with 47.0.1. Actual results: Opus audio sent from Firefox is Opus with payload type 102 and plays correctly at receiving end. Opus audio sent to Firefox is Opus with payload type 109 and plays all garbled. Note that this exact setup plays correctly bidirectionally with Chrome. Expected results: Opus should have played back cleanly. One thing I note is in the WebRTC log it seems like there is a completely different payload type set internally: 19236; Channel::Init() L16 (107/8000/1/128000) has been added to the RTP/RTCP receiver 19236; Channel::Init() L16 (108/16000/1/256000) has been added to the RTP/RTCP receiver 19236; Channel::Init() L16 (109/32000/1/512000) has been added to the RTP/RTCP receiver 19236; Channel::Init() L16 (111/8000/2/128000) has been added to the RTP/RTCP receiver 19236; Channel::Init() L16 (112/16000/2/256000) has been added to the RTP/RTCP receiver 19236; Channel::Init() L16 (113/32000/2/512000) has been added to the RTP/RTCP receiver 19236; Channel::Init() PCMU (0/8000/1/64000) has been added to the RTP/RTCP receiver 19236; Channel::SetSendCodec() 19236; Channel::Init() PCMA (8/8000/1/64000) has been added to the RTP/RTCP receiver 19236; Channel::Init() PCMU (110/8000/2/64000) has been added to the RTP/RTCP receiver 19236; Channel::Init() PCMA (118/8000/2/64000) has been added to the RTP/RTCP receiver 19236; Channel::Init() G722 (9/16000/1/64000) has been added to the RTP/RTCP receiver 19236; Channel::Init() G722 (119/16000/2/64000) has been added to the RTP/RTCP receiver 19236; Channel::Init() opus (120/48000/2/64000) has been added to the RTP/RTCP receiver 19236; Channel::Init() CN (13/8000/1/0) has been added to the RTP/RTCP receiver 19236; Channel::Init() CN (98/16000/1/0) has been added to the RTP/RTCP receiver 19236; Channel::Init() CN (99/32000/1/0) has been added to the RTP/RTCP receiver 19236; Channel::Init() telephone-event (106/8000/1/0) has been added to the RTP/RTCP receiver That seems to indicate the Opus payload type is expected to be 120 (not 109) that was signaled in the Offer. I'm wondering if Firefox is interpreting the data as L16 (indicated as payload type 109 in the receiver list)? Individual packet processing seems to go correctly: 16148; ExternalPlayoutGetData(speechData10ms=0x1f58eab8, samplingFreqHz=32000, current_delay_ms=0) 16148; Channel::NeededFrequency(id=2) 16148; ReceiveFrequency() 16148; PlayoutFrequency() 16148; UpdateToMix(mixList,rampOutList,mixParticipantList,3) 16148; Channel::GetAudioFrame(id=2) 16148; (neteq_impl.cc:163): GetAudio 16148; (decision_logic.cc:131): Buffers: 0 packets * 22 samples/packet + 94 samples in sync buffer = 94 16148; (neteq_impl.cc:699): GetDecision returned operation=2 and 0 packet(s) 16148; (neteq_impl.cc:812): Sync buffer (1 channel(s)): insert 0 samples, extract 320 samples 16148; (neteq_impl.cc:166): Produced 320 samples/channel for 1 channel(s) 16148; PlayoutFrequency() 16148; GetAdditionalAudio(additionalFramesList) 16148; UpdateMixedStatus(mixedParticipantsMap) 16148; MixFromList(mixedAudio, audioFrameList) 16148; MixAnonomouslyFromList(mixedAudio, audioFrameList) 16148; MixAnonomouslyFromList(mixedAudio, audioFrameList) 16148; UpdateVADPositiveParticipants(mixList) 16148; OutputMixer::NewMixedAudio(id=2, size=0) 16148; ClearAudioFrameList(audioFrameList) 16148; ClearAudioFrameList(audioFrameList) 16148; ClearAudioFrameList(audioFrameList) 16148; OutputMixer::GetMixedAudio(sample_rate_hz=32000, num_channels=1) but there's no indication (to me) about what codec was used for decode.
Component: Untriaged → WebRTC: Signaling
Product: Firefox → Core
Just to be clear: I'm assuming your actual bug report is that the audio is garbled. Correct?
Yes, correct. I was just trying to dig up as much info as I could, and found what looked to me like could be an issue.
Summary: WebRTC SDP offer payload type for Opus doesn't match internal WebRTC log → Opus payload type mis-match results in broken audio
Is that the *entire* SDP? (i.e. was video offered? Was it accepted or rejected in the answer?) If not, can you give the entire SDP? I'll note this because offers from Firefox will default to using 120 for VP8. What is generating the logs you're dumping? If video was offered, did the answer have the audio and video m= lines in the same order? The Answer has a small violation of spec in that you added a payload in the answer (101/telephone-event), though that shouldn't have any impact on this problem Can you save about:webrtc (there's a button to save it) and attach that here? Also, if possible, start firefox with the env variable NSPR_LOG_MODULES set to signaling:4 and NSPR_LOG_FILE set to a temp file, and upload that log after running the test. Thanks
Flags: needinfo?(mpietras)
Whiteboard: [Needinfo 2016/07/22]
Yes that is the entire SDP, video was not offered. I copied that from the developer console... I dumped the SDP to the debug console in the success callback of the peer-connection createOffer. I'm not sure how the 101/telephone-event is a violation of the spec... you mean WebRTC spec? The logs were done with Firefox environment variable set with signaling 5 which I think is more verbose: set MOZ_LOG=signaling:5,mtransport:5;webrtc_trace:65535;MediaManager:5,GetUserMedia:5 Unfortunately when I did environment variables for logging, did the about:webrtc, cleared history and log, started debug mode and AEC Logging, did my WebRTC call, Firefox faulted. It generated a small seemingly useless file GeckoChildCrash14048.extra with the contents: SystemMemoryUsePercentage=23 TotalVirtualMemory=4294836224 AvailableVirtualMemory=3835199488 TotalPageFile=39331602432 AvailablePageFile=31265021952 TotalPhysicalMemory=34231328768 AvailablePhysicalMemory=26217148416 I'll play with it some and see what more I can get.
Flags: needinfo?(mpietras)
So it looks like the AEC logging is what faulted it. I turned that off, I'm attaching WebRTC_1.log now.
(In reply to mpietras from comment #4) > I'm not sure how the 101/telephone-event is a violation of the spec... you > mean WebRTC spec? Just to address this separately: telephone-event is not violating the spec by itself. But the fact that you added a codec into the answer which was not offered by FF is a spec violation.
This looks to me like in fact the incoming packets with payload type 109 are treated as L16 packets: Channel::OnInitializeDecoder(id=131072, payloadType=109, payloadName=L16, frequency=32000, channels=1, rate=512000) (neteq_impl.cc:182): webrtc::NetEqImpl::RegisterPayloadType: static_cast<int>(rtp_payload_type)=109, codec=10
Status: UNCONFIRMED → NEW
backlog: --- → webrtc/webaudio+
Rank: 25
Ever confirmed: true
Priority: -- → P2
Whiteboard: [Needinfo 2016/07/22]
(In reply to Nils Ohlmeier [:drno] from comment #7) > (In reply to mpietras from comment #4) > > I'm not sure how the 101/telephone-event is a violation of the spec... you > > mean WebRTC spec? > > Just to address this separately: telephone-event is not violating the spec > by itself. But the fact that you added a codec into the answer which was not > offered by FF is a spec violation. Right, OK thanks, good catch.
Wondering if there's a known workaround for this issue or if we should continue to use just G.711? I'd much rather be using Opus of course...
Mass change P2->P3 to align with new Mozilla triage process.
Priority: P2 → P3

I am facing a similar issue with latest stable and nightly build. when payload type in local and remote SDP does not match, firefox is playing nothing.

SDP
Local SDP (Offer)

v=0
o=mozilla...THIS_IS_SDPARTA-67.0 702656407228142757 0 IN IP4 0.0.0.0
s=-
t=0 0
a=sendrecv
a=fingerprint:sha-256 A3:71:3B:36:68:DA:F7:E4:ED:83:1D:19:E4:60:98:07:8D:29:5C:09:72:FC:05:62:A9:E7:A9:90:29:7F:54:35
a=group:BUNDLE 0
a=ice-options:trickle
a=msid-semantic:WMS *
m=audio 63814 UDP/TLS/RTP/SAVPF 109 9 0 8 101
c=IN IP4 xxx.xxx.xxx.xxx
a=candidate:0 1 UDP 2122252543 xxx.xxx.xxx.xxx 63814 typ host
a=candidate:1 1 UDP 2122187007 xxx.xxx.xxx.xxx 49917 typ host
a=candidate:2 1 TCP 2105524479 xxx.xxx.xxx.xxx 9 typ host tcptype active
a=candidate:3 1 TCP 2105458943 xxx.xxx.xxx.xxx 9 typ host tcptype active
a=candidate:0 2 UDP 2122252542 xxx.xxx.xxx.xxx 50130 typ host
a=candidate:1 2 UDP 2122187006 xxx.xxx.xxx.xxx 56326 typ host
a=candidate:2 2 TCP 2105524478 xxx.xxx.xxx.xxx 9 typ host tcptype active
a=candidate:3 2 TCP 2105458942 xxx.xxx.xxx.xxx 9 typ host tcptype active
a=sendrecv
a=end-of-candidates
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2/recvonly urn:ietf:params:rtp-hdrext:csrc-audio-level
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=fmtp:109 maxplaybackrate=48000;stereo=1;useinbandfec=1
a=fmtp:101 0-15
a=ice-pwd:917fd51a0a7280cc59d1f10a34b45c96
a=ice-ufrag:d3488322
a=mid:0
a=msid:{6519e41c-ce45-4e4b-8791-d9ee860a8a18} {ad1f02e9-450b-6648-a19d-f8d83587092a}
a=rtcp:50130 IN IP4 xxx.xxx.xxx.xxx
a=rtcp-mux
a=rtpmap:109 opus/48000/2
a=rtpmap:9 G722/8000/1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000
a=setup:actpass
a=ssrc:2341447803 cname:{0969b691-9368-5849-9ed1-74cf0b2fddb9}

Remote SDP (Answer)

v=0
o=root 972696436 469494478 IN IP4 xxx.xxx.xxx.xxx
s=-
t=0 0
a=sendrecv
a=ice-lite
m=audio 7205 UDP/TLS/RTP/SAVPF 107 101
c=IN IP4 xxx.xxx.xxx.xxx
a=candidate:874625322 1 udp 2130706431 xxx.xxx.xxx.xxx 7205 typ host
a=sendrecv
a=fingerprint:sha-256 0A:B1:D8:D4:33:E9:0B:BB:2E:6D:08:E7:D9:77:D0:1D:C6:8C:62:9E:58:0A:7D:84:D4:7B:2D:79:48:47:EA:40
a=fmtp:107 maxplaybackrate=0;stereo=0;useinbandfec=1
a=fmtp:101 0-16
a=ice-pwd:MH2Ej7aZqnHtqmbsFoL1pp2H
a=ice-ufrag:pFpYkDmxFDza9nMU
a=maxptime:20
a=mid:0
a=ptime:20
a=rtcp-mux
a=rtpmap:107 opus/48000/2
a=rtpmap:101 telephone-event/8000
a=setup:active
a=ssrc:2575757703 cname:JSz91vC3l3UI1/et

RTP Stats:

inbound_rtp_audio_1

Local: 17:30:12 GMT+0800 (Hong Kong Standard Time) inbound-rtp SSRC: 1964845081 Received: 269 packets (34.77 Kb) Lost: 0 Jitter: 0
outbound_rtp_audio_0

Local: 17:30:12 GMT+0800 (Hong Kong Standard Time) outbound-rtp SSRC: 2341447803 Sent: 274 packets (24.93 Kb)
Assignee: nobody → docfaraday
Pushed by bcampen@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/e7003b05592a Part 0: Add some unit-tests related to payload type asymmetry. r=mjf https://hg.mozilla.org/integration/autoland/rev/f06d16709069 Part 1: Do not set recv tracks' payload types based on the remote SDP. Some related simplifications/fixes. r=mjf https://hg.mozilla.org/integration/autoland/rev/9fcab041c5d7 Part 2: Fix longstanding bug where rollback could wipe out the codecs in a transceiver, that was being hidden by a bug fixed in part 1. r=mjf

If we back this out of ESR 68, what are some examples of things that would break as a result of that?

Standards-compliant code doing mixed payload type negotiation would fail. Sounded like from the "regression" noted above that it's being resolved by WebEx removing the special-case for this bug. Why would there be desire to revert to non-compliant at this point?

Why would there be desire to revert to non-compliant at this point?

Because there are unupgraded WebEx servers in the wild and we're not sure how many will break.

When the next ESR comes out, people will be forced from 60 to 68 and might break.

Is there any way to support both behaviors?

It would require some non-trivial modifications, which would be risky for ESR.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: