1465255 - Receive-side media packet loss observed with hangouts

Status: RESOLVED WORKSFORME

(Core :: WebRTC, defect, P2)

Description

[Tim Haloun]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36

Steps to reproduce:

 - Join a hangouts meet conference with at least one remote participant.
 - Observe receive video packet loss in about:webrtc is nonzero
 - Observe in wireshark that inbound video stream's sequence numbers show lower loss than calculated in about:webrtc
 - Observe in pre-encrypted rtcp traffic receiver report NACK packets (recreated from rtplogger:5 -> text2pcap for example) for inbound video packets that wireshark indicates did make it to the local machine.


Actual results:

Packets were marked as missing that did make it across the network


Expected results:

In an unloaded system, in a static state once joined, with no network problems, packets shouldn't be dropped.  It would be great to know where in the pipeline this is happening.  If it was a queue overfilling or an srtp problem or what.

Comment 1

[u554753]

Per Bug 1431543, marking this as Core: WebRTC issue.

Comment 2

[Andreas Pehrson [:pehrsons]]

I can confirm that I see packet loss for received video between peers on my local network (in Sweden, via an (assumed) SFU in Mountain View). Very few audio packet were reported dropped.

Since this is in stats it's perhaps not critical, but if it screws with every service that is trying to rely on these stats we might want to bump priority. Making this a P2 for now.

In a three-way call with Firefox 62 peer A, Firefox 60 peer B (not sending video) and Chrome 68 peer C,
B reports:
- received 92894 video packets from A, lost 4544
- A having sent 80112 video packets
- received 8436 video packets from C, lost 566
- C having sent 8734 video packets

A reports:
- received 96613 video packets from C, lost 5842
- C having sent 102325 video packets

C reports:
- received 210824 videopackets from A, lost 0

For comparison, also,
A reports:
- received 21144 audio packets from C, lost 13

I haven't compared to packet traces but statistics already point towards a bug in the stats for received video packets.

Byron, Nico, could either of you dig into this to see if there's a root cause with us? I'm not sure whether we should aim at stats code or the network stack.

Comment 3

[Nico Grunbaum [:ng, @chew:mozilla.org]]

I'll take a look.

Comment 4

[Nils Ohlmeier [:drno]]

Just throwing out my ideas here what could cause this:
- One possibility would be something like bug 1435025 where Fx dropped all padded packets on the SRTP layer.
- Another possibility is internal queues, for example sockets queues or IPC queues between parent and child process dropping packets.

Comment 5

[Nico Grunbaum [:ng, @chew:mozilla.org]]

Though I was able to originally reproduce this, I have not been able to do so for a few days now. I have seen some scattered occasional packet loss reported, which seems to be bursty in nature (3 to 70) packets. I am investigating along the lines that Nils suggested above.