1251821 - [WebRtc] Packet loss when sending/receiving RTP stream (1080P) in windows 7

Status: RESOLVED FIXED

(Core :: WebRTC: Networking, defect, P1)

Description

[xpeng]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.10586

Steps to reproduce:

Description : we provided the video conference service using webrtc . 
Case 1:Send RTP stream（H264）
Start screen sharing with firefox in PC1 (windows7 32/64 bit), we found the outboundvideo packet loss on "about: webrtc" page. but there was no packet loss in wireshark which captured the packet in PC1 . 
Case 2:Receive Rtp Stream（H264/VP8）
Receive the video stream in firefox in PC1(windows7 32/64 bit), we found the inboundvideo packet loss on "about:webrtc" page . but there was no packet loss in wireshark which captured the packet in PC1 .

we also found:
1. There is no packet loss in windows 8/10 .
2. There is no packet loss in chrome.
3. If we modify the system's default register value, there will be no packet loss or the packet loss less than before. 
[HKEY_LOCAL_MACHINE \SYSTEM \CurrentControlSet \Services \Afd \Parameters]
 DefaultReceiveWindow = 16384
 DefaultSendWindow = 16384

I suppose it should be the socket buffer size issue in firefox .




Actual results:

Actual results:
Case 1 : packet loss when sending rtp stream
Case 2 : packet loss when receiving rtp stream


Expected results:

expected result:
Case 1 : packet loss should be the same as wireshark
Case 2 : packet loss should be the same as wireshark

Comment 1

[Loic]

Could you attach some network logs, please.

Comment 2

[Nils Ohlmeier [:drno]]

Great bug report. Thank you.
This confirms what I was suspecting happens in bug 1217677. I'll put a patch soon.

Comment 3

[Nils Ohlmeier [:drno]]

This try run https://treeherder.mozilla.org/#/jobs?repo=try&revision=12c2fbed0424
should result in a Windows binary which sets the receive buffer to 1MB (I can provide a direct link to download once the build has finished). It would be great if you could let me know if that fixes the issue for you.

Comment 4

[xpeng]

(In reply to Nils Ohlmeier [:drno] from comment #3)
> This try run
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=12c2fbed0424
> should result in a Windows binary which sets the receive buffer to 1MB (I
> can provide a direct link to download once the build has finished). It would
> be great if you could let me know if that fixes the issue for you.

yes , you could send the link to me . this modification which you'll provide was just for the receive buffer issue ?  
I suppose that it should be two issues :the send buffer & the receive buffer

Comment 5

[xpeng]

Attachment: packetloss.docx

the source code of firefox and the screen capture of result

Comment 6

[xpeng]

(In reply to Loic from comment #1)
> Could you attach some network logs, please.
I've submitted the source code of firefox which I modified  and the screen capture of result to Bugzilla.

Comment 7

[Nils Ohlmeier [:drno]]

(In reply to xpeng from comment #4)
> yes , you could send the link to me . this modification which you'll provide
> was just for the receive buffer issue ?  
> I suppose that it should be two issues :the send buffer & the receive buffer

I just uploaded a new patch in bug 1217677 and triggered new try builds. I'll add the link for downloading once the build has finished.
We are only aware of a problem on receiving side. So it would be great if you could test and verify if just increasing the receive buffer is enough. It would be easy to increase sends buffers as well, but we should only do it if it is really needed.

Comment 8

[Loic]

Attachment: packetloss.PDF

Comment 9

[Nils Ohlmeier [:drno]]

In this directory you can find the builds which contain the current fix:
  http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-cb2fcf460dc2247e70924fbcb00a2533c77b1683/try-win32/
If I'm not mistaken the simplest way to test is if you download the ZIP file:
  http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-cb2fcf460dc2247e70924fbcb00a2533c77b1683/try-win32/firefox-47.0a1.en-US.win32.zip
Because that one you simply unpack and you can start the Firefox Nightly from within the new folder without having to use the Windows installer.

It would help a lot if you could test it at let me know about the result.

Comment 10

[xpeng]

thx , I tested in 5 PCs (windows 7 32/64 bit) with your package, this is no packet loss now. it works well. Great. thank you . the send buffer issue would be fixed next time? if fixed it, you could send the package link to me for testing.  which version would be published with this modification?

Comment 11

[Loic]

(In reply to xpeng from comment #10)
> which version would be published with this modification?

Every new patch is shipped initially in Nightly (https://nightly.mozilla.org/) then some patches are backported in Aurora and Beta.

Comment 12

[Nils Ohlmeier [:drno]]

So the patch for bug 1217677 landed today in mozilla central. Which means the problem should be fixed in tomorrows Nightly build (2016-03-06) of 47.
The fix will "ride the trains" like we say at Mozilla, which according to the rapid release calendar https://wiki.mozilla.org/RapidRelease/Calendar means Firefox 47 will become the official release version in June 2016.

I'm going to close this as a duplicate of bug 1217677 as it should fix this problem. Feel free to comment in here if the problem shouldn't be fixed or you think this should be release sooner the Firefox version 47.

Comment 13

[xpeng]

(In reply to Nils Ohlmeier [:drno] from comment #12)
> So the patch for bug 1217677 landed today in mozilla central. Which means
> the problem should be fixed in tomorrows Nightly build (2016-03-06) of 47.
> The fix will "ride the trains" like we say at Mozilla, which according to
> the rapid release calendar https://wiki.mozilla.org/RapidRelease/Calendar
> means Firefox 47 will become the official release version in June 2016.
> 
> I'm going to close this as a duplicate of bug 1217677 as it should fix this
> problem. Feel free to comment in here if the problem shouldn't be fixed or
> you think this should be release sooner the Firefox version 47.
> 
> *** This bug has been marked as a duplicate of bug 1217677 ***

Case 2 has been fixed in the package which you provided , but the Case 1(packet loss when sending...) was still exist, it should be fixed I supposed that. the Case 1 should be also fixed in the Firefox version 47 ?

Comment 14

[Nils Ohlmeier [:drno]]

(In reply to xpeng from comment #13)
> Case 2 has been fixed in the package which you provided , but the Case
> 1(packet loss when sending...) was still exist, it should be fixed I
> supposed that. the Case 1 should be also fixed in the Firefox version 47 ?

Sorry my fault. I assumed that the sending problem was related or would get solved through fixing the receiving issue.

I just tried to reproduce the issue by attaching a Logitech C920 to my Windows 7 laptop. But in a test call I don't see any packet loss getting reported.

Are you referring to the "Dropped frames" value in about:webrtc? I see that going up, but that has nothing to do with network packet loss.

Could you please attach a copy of about:webrtc which shows the packet loss in sending you are seeing?

Comment 15

[xpeng]

RTP Stats
outbound_rtcp_video_0

Local: 10:10:13 GMT+0800 inboundrtp SSRC: 770155137 Received: 15995 packets (16932.21 Kb) Lost: 1901 Jitter: 0 RTT: 1 ms
inbound_rtp_video_0

Local: 10:10:15 GMT+0800 inboundrtp SSRC: 0
outbound_rtp_video_0

Encoder: Avg. bitrate: 1.16 Mbps (1.12 SD) Avg. framerate: 16.99 fps (6.55 SD) Dropped frames: 172642

Local: 10:10:15 GMT+0800 outboundrtp SSRC: 770155137 Sent: 17985 packets (19418.01 Kb)

Remote: 10:10:13 GMT+0800 inboundrtp SSRC: 770155137 Received: 15995 packets (16932.21 Kb) Lost: 1901 Jitter: 0 RTT: 1 ms

in the about:webrtc the feedback of packet loss(sending)  should be the result which received from the receive side through RTCP. 
I supposed that the "Dropped frames" has nothing to do with packet loss . Because the "Dropped frames" happened when encoding frames by the encoder module. the "packet loss(sending)" happened after that the frame had been encoded.

the process should be like this or not?
 
Send Side:             
 PC1--- start screen capturing --> capture the frame of screen -->send the frame to encoder module(OpenH264)
(drop some frames depend on it's logic)  --> packet the frame and deliver the packets to system --> system send the packet to Network Card --> Wireshark capture the packets --------Network---------> PC2 

Receive Side:
PC2 received the packets and send RTCP(jitter/packet loss (sending) and other information) to PC1 ----RTCP-->PC1(Received the RTCP packet and update the jitter/packet loss (sending) information to the about:webrtc page)  

 
the attachment which I provided proved that the package loss(sending) occurred between "packet the frame and deliver the packets to system " and "Wireshark capture the packets ".

Comment 16

[Nils Ohlmeier [:drno]]

Yes your process outline is correct.

One important thing to keep in mind though is that as you pointed out yourself the packet loss on the sending side represents what the sender actually received up in its RTP stack. So if
- packets get lost on the network or
- packets actually arrive at the machine but don't make it up to the RTP stack like in case of bug 1217677
you would see packet loss on the sending side in the stats.

Now I see in the word document you provided that you added logging for the sending side, but the screenshots show only the log for receiving side.

The one thing I have overlooked from your initial description is that this only happens on Win 7 when using screen sharing with H264, correct?
I'll check if I can repro exactly that.

I hope/assume that on the receiving side (PC2) is not a Firefox from before bug 1217677 got fixed, because that would obviously explain why PC2 reports packet loss back to PC1.

Comment 17

[Nils Ohlmeier [:drno]]

Looks like I'm able to reproduce this now.

Comment 18

[Nils Ohlmeier [:drno]]

To clarify: I see around 10% packet loss. And with increased send buffer this gets substantial less.

Comment 19

[Nils Ohlmeier [:drno]]

Attachment: MozReview Request: Bug 1251821: increase UDP socket send buffer on Win 7

Review commit: https://reviewboard.mozilla.org/r/45331/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/45331/

Comment 20

[Nils Ohlmeier [:drno]]

Comment on attachment 8739645 [details]
MozReview Request: Bug 1251821: increase UDP socket send buffer on Win 7

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/45331/diff/1-2/

Comment 21

[xpeng]

thx , which version would be published with this modification(increase send buffer on win7)?

Comment 22

[Nils Ohlmeier [:drno]]

The patch has not landed yet, as I'm trying to investigate the root cause for the problem.

I found out that the problem is actually not H264 specific, but also happens with VP8. And the reported packet loss is actually only a small fraction packet loos, but heavily packet sequencing errors.

Comment 23

[Nils Ohlmeier [:drno]]

So I tested this quite a bit more:

I am able to reproduce the packet loss when using a Win 7 machine (note: I'm using screen share of a full screen HD video on Youtube for testing purposes). In fact it is not really packet loss what I'm seeing, but instead wireshark on the sending machine (so it's not the networks fault) tells me that RTP video packets are leaving the machine in wrong order.
The packets in wrong order are not in the area of ~10%, but I'm assuming that the receiver throws away the whole I or P frame if a single RTP for a given frame arrives too late and then reports all of the packets as loss.
Now this packet re-ordering happens with VP8 and H264. But I don't see it happening on Win 10.

Comment 24

[xpeng]

you mean that there is no packet loss , the problem we saw caused by the out-of-order packet?

Comment 25

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Nils Ohlmeier [:drno] from comment #23)
> In fact it is not really packet loss what I'm seeing, but instead
> wireshark on the sending machine (so it's not the networks fault) tells me
> that RTP video packets are leaving the machine in wrong order.
> The packets in wrong order are not in the area of ~10%, but I'm assuming
> that the receiver throws away the whole I or P frame if a single RTP for a
> given frame arrives too late and then reports all of the packets as loss.
> Now this packet re-ordering happens with VP8 and H264. But I don't see it
> happening on Win 10.

Nils - can you share a wireshark capture with me?  You can sub-set it to a since frame or two's data.

Comment 26

[Nils Ohlmeier [:drno]]

Randell, just send an extract from one of the PCAP I have via email.

I'm currently reluctant to just land the "increase the send buffer" patch attached here, because I think it might be just covering up a real issue in NSPR. But we could land it and create a separate follow up bug for the root cause investigation (and the decide what to do with the send buffer size from this bug once we know what exactly is going on).

Comment 27

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Nils Ohlmeier [:drno] from comment #26)
> Randell, just send an extract from one of the PCAP I have via email.

So that sequence shows packets with a loss sequence, then a delay, an RTCP going the other direction, followed immediately by the missing packets, then the sequence moves on.

This is exactly the pattern of an RTCP NACK, which almost certainly this was.  So the packets were lost (probably dropped due to buffer overrun), NACKed, and retransmitted.  A larger buffer here makes total sense.

A higher level of logging would likely tell you there was a loss/NACK.

Comment 28

[Nils Ohlmeier [:drno]]

Comment on attachment 8739645 [details]
MozReview Request: Bug 1251821: increase UDP socket send buffer on Win 7

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/45331/diff/2-3/

Comment 29

[Nils Ohlmeier [:drno]]

Comment on attachment 8739645 [details]
MozReview Request: Bug 1251821: increase UDP socket send buffer on Win 7

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/45331/diff/3-4/

Comment 30

[Nils Ohlmeier [:drno]]

Comment on attachment 8739645 [details]
MozReview Request: Bug 1251821: increase UDP socket send buffer on Win 7

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/45331/diff/4-5/

Comment 31

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8739645 [details]
MozReview Request: Bug 1251821: increase UDP socket send buffer on Win 7

https://reviewboard.mozilla.org/r/45331/#review45349

Comment 32

[Josh Matthews [:jdm]]

Comment on attachment 8739645 [details]
MozReview Request: Bug 1251821: increase UDP socket send buffer on Win 7

https://reviewboard.mozilla.org/r/45331/#review45855

Comment 33

[Nils Ohlmeier [:drno]]

Verified manually that the try build binary fixes the HD video streaming issue (as expected).

Comment 34

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3cc3b76f4205

Comment 35

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/3cc3b76f4205