1237630 - Image freezes and no sound available after ~2 minutes call in https://appear.in/

Status: VERIFIED FIXED

(Core :: WebRTC, defect, P1)

Description

[Bogdan Maris, Desktop QA [Holiday+PTO 1-6 May]]

STR:
1. Start Firefox (we tested with latest Aurora on both devices PC and phone).
2. Visit https://appear.in/
3. Start a call between the two devices.

Expected: During the call there are no issues.

Actual results:
During a call between Asus P024 tablet (android version 5.0.2) and Ubuntu 14.04 x64 the image freezes and no sound for each guest (on tablet the person from Ubuntu is freezed and the other way around for desktop user). 

Same thing happens using a Nexus 5 with android 6.0.1 and Ubuntu 14.04 x64 and, Windows 7 x64; 

Notes:
- No issues were recorded while making a call between Nexus 5 with android 6.0.1 and Chromium (Ubuntu 14.04 x64)
- Will investigate further if this is a recent regression

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

I tried this a number of times with a Nexus 10 and Windows, both running Nightly.

Several times (but not all) I had Android->Windows video freeze; audio continued.   Once, after a long time with Android video frozen, video from Windows->android froze. Again, audio continued.

I've had only one other instance of video (only) freezing recently; during a talky.io call (from the same windows machine); the other person said my video froze after a few minutes.  I was runnging a second brwoser and the system was overloaded; I killed the second browser, reloaded the page and video and audio were fine for 45 minutes.

What we need is a log and maybe wireshark trace.  set NSPR_LOG_MODULES=signaling:4,webrtc_trace:65535 WEBRTC_TRACE_FILE=nspr NSPR_LOG_FILE=some temp file.   Try to record approximately when the freeze starts for correlating to the log timestamps.  I'll try this myself.

There might be a talky.io change, or there might be fallout from the webrtc.org 43 update. It might be a recent regression, though I can't think of any source for such a regression.   If so, we should be seeing it in Aurora/45 as well, and this report was initially against Aurora.  The initial report here was both audio and video failing, which I have not seen, so these may *not* be the same issue - though the logs needed are the same.

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

If it is the 43 update, it's likely some busted assumption in the 43 error recovery code

Comment 3

[Thomas Daede [:TD-Linux]]

I also see this happen - appear.in sessions last about 10 minutes max. It's happening with both Aurora and Nightly users (45 and 46). We've had to switch away from WebRTC meetings because of it.

I'll look at doing a wireshark trace.

I generally see all connections lost at the same time, with all of the connections timing out, though sometimes one user stays connected for a little while longer (it seemed to always be the only other Nightly / Aurora user)

Comment 4

[Maire Reavy [:mreavy]]

Thanks, Thomas, for chiming in. pkerr is taking the lead on fixing this, and I hope to have a fix within the next couple of days -- depending on how easy this is to reproduce. If we have trouble repro'ing, we'll definitely reach out to you -- and we'll ask you to test once we have a patch that we believe fixes the problem.

Comment 5

[Paul Kerr [:pkerr]]

I am able to reproduce between my Ubuntu machine and Android tablet via talky.io. Android side stops sending video. I am working on log collection and analysis.

Comment 7

[Paul Kerr [:pkerr]]

It appears that there is a new check being used in the video_sender: _codecDataBase.MatchesCurrentResolution() that begins to fail after the quality manager makes changes in the spatial resolution. This method did not exist in the VCMCodecDataBase class in our previous webrtc.org import. Based on a TODO left in the VideoSender::AddVideoFrame method, I believe I will need to add code to synchronize the changes in frame size of the I420VideoFrames being generated with the width and height recorded in the _codecDataBase.

Comment 8

[Paul Kerr [:pkerr]]

The change mentioned above was introduced into the webrtc.org codebase through issue https://webrtc-codereview.appspot.com/48379004. The same code exists in the branch48 repo.

Comment 9

[Paul Kerr [:pkerr]]

Attachment: Part 1: Video freeze from WebRTC sender

Fix for video freeze

Comment 10

[Paul Kerr [:pkerr]]

Attachment: Part 2: remove LOG statement generating a now defunct error condition

Log has changed in VCMCodecDataBase::SuportsRenderScheduling method and the existing error message is no longer valid.

Comment 11

[Paul Kerr [:pkerr]]

Comment on attachment 8710487 [details] [diff] [review]
Part 1: Video freeze from WebRTC sender

I have tested this under cpu stress on OS X, Linux and Android to the point where the LoadManager triggers a frame size change. This would have triggered the frame drops in the past but now the correct adaptation takes place and video remains stable.

Comment 12

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8710487 [details] [diff] [review]
Part 1: Video freeze from WebRTC sender

Review of attachment 8710487 [details] [diff] [review]:
-----------------------------------------------------------------

::: media/webrtc/trunk/webrtc/modules/video_coding/main/source/video_sender.cc
@@ -379,5 @@
> -  if (!_codecDataBase.MatchesCurrentResolution(videoFrame.width(),
> -                                               videoFrame.height())) {
> -    LOG(LS_ERROR) << "Incoming frame doesn't match set resolution. Dropping.";
> -    return VCM_PARAMETER_ERROR;
> -  }

Because upstream will continue to have this, I think I'd prefer to ifdef this code out for Mozilla.  This will make future merges a little easier

Comment 13

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b11b3a3c7c1b
https://hg.mozilla.org/integration/mozilla-inbound/rev/b756dd7e492a

Comment 14

[Paul Kerr [:pkerr]]

Attachment: Part 1: Video freeze from WebRTC sender

Incorporate review comment. Carry forward r+.

Comment 15

[Paul Kerr [:pkerr]]

Comment on attachment 8710644 [details] [diff] [review]
Part 1: Video freeze from WebRTC sender

Review of attachment 8710644 [details] [diff] [review]:
-----------------------------------------------------------------

Carry forward r=rjesup.

Comment 16

[Paul Kerr [:pkerr]]

Comment on attachment 8710644 [details] [diff] [review]
Part 1: Video freeze from WebRTC sender

Approval Request Comment
[Feature/regressing bug #]: 1237630
[User impact if declined]:
If the system load reaches the point that the LoadManager requests a change in the frame size the video encoder will begin to drop frames. The user's video will appear to freeze on the receiving end of the connection.
[Describe test coverage new/current, TreeHerder]:
There is no performance test currently that would trigger this condition. 
[Risks and why]: Low risk.
A new frame size verification test that was added in the newer webrtc.org release branch (43) is meant to handle a theoretical bug (see link in comment above) that may appear when using hardware codecs. Mozilla does not use hardware codecs outside of FxOS. The change made with this patch reverts the video encode path operation back to what has successfully be the model in previous releases of WebRTC from Mozilla: the codec is responsible for handling frame size changes. In addition, manual testing has been done on OS X, Linux and Android to first reliably trigger the video freeze and then, after applying the patch, preventing the freeze.
[String/UUID change made/needed]: none needed

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/b11b3a3c7c1b
https://hg.mozilla.org/mozilla-central/rev/b756dd7e492a

Comment 18

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8710644 [details] [diff] [review]
Part 1: Video freeze from WebRTC sender

Fix a freeze, taking it.

Comment 19

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/015db92e8e45
https://hg.mozilla.org/releases/mozilla-aurora/rev/e5961b5798f2

Comment 20

[Bogdan Maris, Desktop QA [Holiday+PTO 1-6 May]]

I did not reproduce this anymore on a appear.in call between Asus ZenPad 8 (Android 5.0.2) and Ubuntu 14.04 32-bit and Windows 7 64-bit using Firefox 46 beta 8.