1256679 - Muting/resuming video MediaStreamTrack causes delay

Status: RESOLVED DUPLICATE of bug 1208371

(Core :: WebRTC, defect, P1)

Description

[lminiero]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.18 Safari/537.36

Steps to reproduce:

For some time I've noticed a weird behaviour when using Firefox in my tests with my WebRTC gateway. Specifically, if I mute a video MediaStreamTrack in a web application, and then after some time unmute it again, it comes back delayed. It seems to happen more evidently when the bitrate of the video is low, while it's barely noticeable for higher bitrates for some reason, although it might or might not be related. 

It is easily reproduceable, if you want to check it yourself: 

1. open https://janus.conf.meetecho.com/echotest (simple gateway demo that just sends you back whatever you send to it) 
2. use the "bandwidth" control to cap it to 128kbps (as anticipated, it seems more evident at lower bitrates: that control forces REMB feedback at 128000) 
3. mute the local video track: echotest.webrtcStuff.myStream.getVideoTracks()[0].enabled=false 
4. wait some time and unmute it again: echotest.webrtcStuff.myStream.getVideoTracks()[0].enabled=true 
5. you'll see that video is now delayed, while audio is still ok.


Actual results:

After those steps, the video is delayed. The demo doesn't do any RTP sequence number/timestamp overwriting, so what you get back is exactly the same as what you sent, which makes me think something goes wrong in how timestamps are computed/written in the Firefox WebRTC stack after video transmission is resumed. If I got it right, the framerate in the video changes when you disable video (IIRC you basically send a black frame at a much lower framerate), which means something might be messed up in those transitions.


Expected results:

I would expect no delay at all, with the video resuming at the same delay it was before pausing the capture. I've seen the issue around for a few months, and it affected Chrome too up to some time ago, so I thought it was something we were doing wrong: anyway, I've tried Chrome 50 lately and the issue is gone there, video has no delay anymore, which is making me think it might be indeed a bug in Firefox.

Comment 1

[Maire Reavy [:mreavy]]

Hey Andreas -- IIRC this sounds like a bug you reported and worked on (that I thought we fixed).  Any insights on what's going on here?

Comment 2

[Andreas Pehrson [:pehrsons]]

I know jesup observed this some time ago, and I had a theory last week of what could be affecting this (I saw time buildup in the disabled-video-track path), but I have in no way confirmed it.

We'll have to dig deeper to see whether it's in our code (I doubt it's in the MediaStreamGraph itself, but it could be something in MediaPipeline or VideoConduit that causes ripple effects) or any underlying webrtc.org layers.

It might also get affected by my cloning patch queue so we should recheck after bug 1208371 lands.

Comment 3

[Andreas Pehrson [:pehrsons]]

Hard to perceive but I think I have this fixed with my async image format converter (which also does async black frame allocation) that I just wrote for bug 1208371.

Even harder to tell since we also go from a direct listener in MediaPipeline to being driven by the MediaStreamGraph. Could be covering something up.

Comment 4

[Andreas Pehrson [:pehrsons]]

Bug 1208371 landed. I see a significant improvement following your steps to reproduce. If you could also double check yourself I'd be grateful.

Comment 5

[lminiero]

Thanks for the update! Which Firefox version should I use for testing? I'm currently abroad for a conference, but I'll test and come back to you as soon as I get back to the office on Monday.

Comment 6

[Andreas Pehrson [:pehrsons]]

Nightly from today or later should be fine!
https://nightly.mozilla.org

Comment 7

[lminiero]

Just found some time to test: I used Nightly 48.0a1 (2016-04-12), and I can confirm it seems to work, thanks!