972097 - Cannot view video from Firefox 28 on opentok

Status: VERIFIED FIXED

(Core :: WebRTC, defect)

Description

[msander]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36

Steps to reproduce:

1. Open https://opentokrtc.com/OPENTOK-11391 in Firefox 28
2. Open the URL in Chrome
3. Open the URL in a new Firefox tab


Actual results:

None of the browsers can see video from Firefox 28 Beta browsers.

Also affects 29 Aurora
Does not affect 27.


Expected results:

Should be able to view video published from Firefox 28.

Comment 1

[Byron Campen [:bwc]]

I can replicate this scenario, here is what I can tell so far:

- There are separate PeerConnections for outgoing media and incoming media
- ICE has succeeded on 4 PeerConnections, all of which are apparently connected to a gateway of some sort
- Both PeerConnections that handle outgoing media are sending packets to the expected destinations for both audio and video, and have one component that settled on a server-reflexive/host pair.
- One PeerConnection that handles incoming media has established with two ICE components for each of audio and video. In both cases, one component has settled on a server-reflexive/host pair (which is receiving packets), and another has settled on a relayed/host pair (which is not receiving packets).
- The other PeerConnection that handles incoming media is similar, except no packets are being received on any succeeded candidate pair for audio or video. 
- After a long wait (basically, long enough for me to dredge through wireshark traces and correlate the traffic with the ICE stats), both tabs in Firefox are displaying the remote video.

Comment 2

[Byron Campen [:bwc]]

Some more weird stuff; the candidate pairs for incoming media seem a little strange:

Local candidate	Remote candidate	ICE State	Priority	Nominated	Selected
10.250.7.243:51176/host	72.5.167.175:54768/host	failed	9149908167411368000		
10.250.7.243:62381/host	72.5.167.175:60115/host	failed	9149908163116401000		
10.254.250.70:57878/host	72.5.167.175:54768/host	failed	9142308343040180000		
10.254.250.70:63259/host	72.5.167.175:60115/host	failed	9142308338745213000		
63.245.220.240:55337/peerreflexive	72.5.167.175:54768/host	succeeded	7996986662804521000		
63.245.220.240:55337/peerreflexive	72.5.167.175:60115/host	succeeded	7996986658509554000		
72.5.167.174:37282/relayed	72.5.167.175:54768/host	succeeded	431219664287170560	*	*
72.5.167.174:38719/relayed	72.5.167.175:60115/host	succeeded	431219659992203260		
72.5.167.174:39432/relayed	72.5.167.175:54768/host	succeeded	70931694097530880	*	
72.5.167.174:34049/relayed	72.5.167.175:60115/host	succeeded	70931689802563580	*	*


Note that the same remote port/ip is in both the host candidates and the relayed candidates. Also, the higher priority reflexive candidates have succeeded, but have not been nominated.

Comment 3

[Byron Campen [:bwc]]

Actually, I misread; the repeated address/ports are expected. The lack of nomination on higher priority succeeded pairs is still odd.

Comment 4

[Byron Campen [:bwc]]

Ok, managed to find the missing inbound traffic; it is coming in over TURN, because the higher priority succeeded pairs are never nominated. Let me look and see who is controlling.

Comment 5

[Byron Campen [:bwc]]

Hmm. I'm seeing check requests coming from the gateway with both an ICE-CONTROLLED attribute, and a USE-CANDIDATE attribute. That's a little strange. It does not seem to have harmed anything though.

Comment 6

[Byron Campen [:bwc]]

Ok, the reason the higher-priority succeeded pairs are never being nominated is because we never see a check request from the other end (it would be really nice to have a pcap from the gateway here). I will upload the pcap file from my end, and the stuff from about:webrtc.

Comment 7

[Byron Campen [:bwc]]

Attachment: pcap file

Comment 8

[Byron Campen [:bwc]]

Attachment: Copy/paste of about:webrtc

Comment 9

[msander]

I'll talk to one of our server engineers to see if we can get you some more information about what's happening from the gateway.

Comment 10

[Byron Campen [:bwc]]

It looks like we regressed on Nov 09 (regression range is f003c386c77a to 9e571ad29946). Here is some stuff we landed in that range:

changeset:   154074:50bf8a24a269
user:        Byron Campen [:bwc] <[e-mail]>
date:        Thu Nov 07 14:48:43 2013 -0800
description:
Bug 936031 - Test case for bug. r=abr


changeset:   154075:5627b707affe
user:        Byron Campen [:bwc] <[e-mail]>
date:        Thu Nov 07 15:03:06 2013 -0800
description:
Bug 936031 - Attempted fix. r=ehugg


changeset:   154076:7aca035355ae
user:        Randell Jesup <[e-mail]>
date:        Thu Nov 07 20:07:47 2013 -0500
description:
Bug 932112: Webrtc updated to 5041, pull made Mon Oct 28 12:17:00 EDT 2013 rs=jesup


changeset:   154077:bd8f1571937f
user:        Randell Jesup <[e-mail]>
date:        Thu Nov 07 20:07:47 2013 -0500
description:
Bug 932112:  Rollup of changes previously applied to media/webrtc/trunk/webrtc rs=jesup
* * *
* * *
Add AndroidAudioManager to the moz.build files.


changeset:   154078:5b8a157a6ed5
user:        Gian-Carlo Pascutto <[e-mail]>
date:        Thu Nov 07 20:07:47 2013 -0500
description:
Bug 932112:  Use the non-main-thread FindClass implementation r=blassey


changeset:   154079:32d82a576615
user:        Randell Jesup <[e-mail]>
date:        Thu Nov 07 20:07:47 2013 -0500
description:
Bug 932112: JB reflect for low-latency params r=mfinkle


changeset:   154081:cc7e3ef4bb80
user:        Gian-Carlo Pascutto <[e-mail]>
date:        Thu Nov 07 20:07:48 2013 -0500
description:
Bug 932112: Initialize both JNI and OpenSLES so fallback can work. r=jesup


changeset:   154082:4481461159f2
user:        Gian-Carlo Pascutto <[e-mail]>
date:        Thu Nov 07 20:07:48 2013 -0500
description:
Bug 932112:  Add a non-ARM MemoryBarrier function. r=glandium

Comment 11

[Byron Campen [:bwc]]

My suspicion is either the SDP fix in bug 936031, or the webrtc.org update.

Comment 12

[Byron Campen [:bwc]]

Attachment: Webrtc trace from Nov 09 nightly (OS X)

Command line:

NSPR_LOG_MODULES=signaling:6,mtransport:6,webrtc_trace:65535 WEBRTC_TRACE_FILE=$(pwd)/webrtc.log R_LOG_LEVEL=6 R_LOG_DESTINATION=stderr /Applications/FirefoxNightly.app/Contents/MacOS/firefox -P testing &> ffx.out

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/11/2013-11-09-03-02-06-mozilla-central/

Comment 13

[Byron Campen [:bwc]]

Attachment: stderr/stdout for test on Nov 09 nightly (OS X)

Command line:

NSPR_LOG_MODULES=signaling:6,mtransport:6,webrtc_trace:65535 WEBRTC_TRACE_FILE=$(pwd)/webrtc.log R_LOG_LEVEL=6 R_LOG_DESTINATION=stderr /Applications/FirefoxNightly.app/Contents/MacOS/firefox -P testing &> ffx.out

Comment 14

[Randell Jesup [:jesup] (needinfo me)]

Attachment: bug972097

Lost an "rtcp_" in the source merge from 3.43.  Caused RTCP not to be received in one-way PeerConnections

verbal r=padenot

Comment 15

[Randell Jesup [:jesup] (needinfo me)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/abe987db4368

We should uplift this once it's confirmed fixed

Comment 16

[Byron Campen [:bwc]]

Verified fixed.

Comment 17

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8376588 [details] [diff] [review]
bug972097

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 932112

User impact if declined: One-way video channels will fail to render video (sites like TokBox use this to do WebRTC conferences

Testing completed (on m-c, etc.): Fix verified by :bwc (and verified the error messages without the fix finger this as the problem).  Will be on m-c momentarily.

Risk to taking this patch (and alternatives if risky): Risk is nil; this merely corrects a mis-merge where a mod to change a boolean name didn't get applied (so it references the wrong boolean).  This is fine in two-way channels, but breaks one-way.  The fix has been taken upstream, but this import was from a stable branch made before the upstream landing.

String or IDL/UUID changes made by this patch: None.

Comment 18

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/abe987db4368

Comment 19

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/caa38669fa1d
https://hg.mozilla.org/releases/mozilla-beta/rev/4fef9a9dbdf3

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g28_v1_3/rev/4fef9a9dbdf3

Comment 21

[Petruta Horea [:phorea]]

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:28.0) Gecko/20100101 Firefox/28.0 (20140213172947)		
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:29.0) Gecko/20100101 Firefox/29.0 (20140219004002)		
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:30.0) Gecko/20100101 Firefox/30.0 (20140219030203)	

Verified as fixed on Firefox 28 beta 4, latest Aurora 29.0a2 and latest Nightly 30.0a1.