960856 - Cannot communicate audio in a P2P call from Taipei's Mozilla Guest network to a home network

Status: RESOLVED DUPLICATE of bug 950660

(Core :: WebRTC: Networking, defect)

Description

[Jason Smith [:jsmith]]

Build Environment 1:

Build Date - 1/12/2014 Buri 1.4
Network - Mozilla-G wifi network in Taipei

Build Environment 2:

Build Date - 1/16/2014 Buri 1.3
Network - Home wifi network in US

STR

1. Have BE 1 go to apprtc.appspot.com, accept permissions
2. Have BE 2 go to the apprtc URL seen in BE 1, accept permissions
3. Try talking back & forth vocally

Expected

Audio should be sent & received between each peer.

Actual

The call successfully establishes, but audio cannot be sent between each peer.

Note - working on getting logs. If this needs to be reproduced again, then have one of the Taipei developers do an audio call to Naoki Hirata over Taipei's Mozilla-G wifi network to Naoki's wifi network.

Comment 1

[Jason Smith [:jsmith]]

One correction - the Taipei's wifi network is Mozilla Guest, not Mozilla-G. I forgot we don't support connecting to Mozilla-G & Mozilla wifi networks on FxOS right now.

Comment 2

[u459114]

Steven, please check this problem first

Comment 3

[Jason Smith [:jsmith]]

Attachment: BE 1 Wireshark Trace

First example of a trace from BE 1

Comment 4

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: BE 2 Wireshark Trace

Here's the other side's Wireshark Trace log.

Comment 5

[Jason Smith [:jsmith]]

FWIW - Steven mentioned in person that audio communication works fine on Taipei Guest --> Taipei Guest. He's going to dive into the wireshark trace to analyze the bug more.

Comment 6

[StevenLee[:slee]]

(In reply to Jason Smith [:jsmith] from comment #5)
> FWIW - Steven mentioned in person that audio communication works fine on
> Taipei Guest --> Taipei Guest. He's going to dive into the wireshark trace
> to analyze the bug more.
I tried on desktop browser and device under Taipei Guest and worked fine. I will try both on devices. If there is any updates, I will post it. BTW, I checked the wireshark log and found that there are only stun packets.

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

Note from email this was only attempting audio as the 1.3 device doesn't support video.  When the taipei side was at a hotel, it works.  This implies it's related to the network characteristics of the Taipei-Guest network's firewall.

It looks like naoki's side got STUN responses.  I can't find any indication that the "callingside" trace (taipei) ever got *any* response to STUN requests.  Note that I might not see/understand TURN-TCP, though I don't see any TCP SYN packets after taipei starts sending STUN UDP.

nicer logs (R_LOG_LEVEL etc) would be more useful, plus logs from nspr for signaling:5,mtransport:5 would be good too.

Does this reproduce with one end (either one) being a desktop running 29?  Both ends being a desktop?  Both tests would be good first-level checks of the network before getting too deep into whether it's a B2G issue.  In either case, logs from the desktop(s) would help a lot in debugging this and a lot easier to get.

Note that Taipei Guest -> Taipei-Guest are very likely to work.

See https://wiki.mozilla.org/Media/WebRTC/Logging

Comment 8

[Jason Smith [:jsmith]]

(In reply to Randell Jesup [:jesup] from comment #7)
> Note from email this was only attempting audio as the 1.3 device doesn't
> support video.  When the taipei side was at a hotel, it works.  This implies
> it's related to the network characteristics of the Taipei-Guest network's
> firewall.

To clarify - the hotel testing was done in North Carolina in which each peer was on the same network. I haven't tested a call to Naoki from the Taipei hotel.

> 
> It looks like naoki's side got STUN responses.  I can't find any indication
> that the "callingside" trace (taipei) ever got *any* response to STUN
> requests.  Note that I might not see/understand TURN-TCP, though I don't see
> any TCP SYN packets after taipei starts sending STUN UDP.
> 
> nicer logs (R_LOG_LEVEL etc) would be more useful, plus logs from nspr for
> signaling:5,mtransport:5 would be good too.
> 
> Does this reproduce with one end (either one) being a desktop running 29? 
> Both ends being a desktop?  Both tests would be good first-level checks of
> the network before getting too deep into whether it's a B2G issue.  In
> either case, logs from the desktop(s) would help a lot in debugging this and
> a lot easier to get.

I'll work with Naoki on this next week to do desktop --> fxos in each direction.

I'll leave qawanted on here to see if this bug reproduces from doing a desktop to fxos call from Taipei Guest to the home network. If it reproduces there, then we'll work on getting the NSPR logs from the desktop machine.

Comment 9

[StevenLee[:slee]]

I tried that both fxos and desktop are under Taipei-Guest, the connection is fine. BTW, how can I use apprtc and construct audio only connection? My test is to force the media type to fake on fxos.
One more question, could we connect 2 peers between Mozilla-Guest and Mozilla?

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

(In reply to StevenLee[:slee] from comment #9)
> I tried that both fxos and desktop are under Taipei-Guest, the connection is
> fine. 

Right - testing from within the same firewall doesn't test any significant part of ICE or TURN, and with this sort of failure that's the first place to look (and if I read things correctly, all UDP STUN requests failed).  If TCP turn was working on both of those devices, it *should* have worked; I didn't see it try in the wireshark, but I may have missed it.

jsmith's test within the same hotel network may or may not have used ICE/TURN - some hotel networks like the one we had at the Sunnyvale work week block peer-2-peer connections within the firewall, even if they use the external IP (hairpinning).

> BTW, how can I use apprtc and construct audio only connection? My test
> is to force the media type to fake on fxos.

You have to deny access to video I think (for 1.4 or desktop).  It's not designed for audio-only calls, so it's painful.

> One more question, could we connect 2 peers between Mozilla-Guest and
> Mozilla?

In Mountain View?  I assume so, at least using TURN. I'm not there so I haven't done the test myself, but I'm certain it's been done.

Comment 11

[Jason Smith [:jsmith]]

Btw - one idea I was thinking that would help with debugging is to have Steven do an audio P2P call to Naoki sometime next week with Steven's device setup for whatever debug logging he needs. That would help figure out what's not working here.

Comment 12

[Eric Rescorla (:ekr)]

Important note: B2G does not currently have support for TURN TCP.
See https://bugzilla.mozilla.org/show_bug.cgi?id=950660
If you are behind a firewall that blocks UDP,
B2G will fail and desktop will work. 
(this is not a blocker since we shipped FF for a long time without
TURN TCP). Based on Randell's analysis that sounds like the problem.

Comment 13

[Jason Smith [:jsmith]]

(In reply to Eric Rescorla (:ekr) from comment #12)
> Important note: B2G does not currently have support for TURN TCP.
> See https://bugzilla.mozilla.org/show_bug.cgi?id=950660
> If you are behind a firewall that blocks UDP,
> B2G will fail and desktop will work. 
> (this is not a blocker since we shipped FF for a long time without
> TURN TCP). Based on Randell's analysis that sounds like the problem.

Just because we shipped desktop with the problem doesn't necessarily mean it's okay to ship FxOS with the problem. When desktop initially shipped, TURN wasn't supported in P2P. Now, TURN is supported & is actively used on the web in P2P apps, including on apprtc, which is a key demo application. We also know that TURN TCP landed in Gecko 28 in bug 906968, so this code path exists in FxOS 1.3. Additionally, we have at least two content partners looking to build two communication style apps on FxOS, so we have be conscious that P2P communication works extremely well to get partner buy in to build those apps. On that regard, I see that this problem is a content blocker for 1.3 with severe impact, given the fact that audio communication is failing & the fact this impacts a key demo application.

Note - one thing that would be interesting to look into is to figure out if the patch "turn off TURN TCP for gonk" isn't having negative impact to cause audio communication to fail here on FxOS.

Comment 14

[Eric Rescorla (:ekr)]

(In reply to Jason Smith [:jsmith] from comment #13)
> (In reply to Eric Rescorla (:ekr) from comment #12)
> > Important note: B2G does not currently have support for TURN TCP.
> > See https://bugzilla.mozilla.org/show_bug.cgi?id=950660
> > If you are behind a firewall that blocks UDP,
> > B2G will fail and desktop will work. 
> > (this is not a blocker since we shipped FF for a long time without
> > TURN TCP). Based on Randell's analysis that sounds like the problem.
> 
> Just because we shipped desktop with the problem doesn't necessarily mean
> it's okay to ship FxOS with the problem.

It's not a problem. It's a missing feature.


> When desktop initially shipped,
> TURN wasn't supported in P2P. Now, TURN is supported & is actively used on
> the web in P2P apps, including on apprtc, which is a key demo application.
> We also know that TURN TCP landed in Gecko 28 in bug 906968, so this code
> path exists in FxOS 1.3.

Not quite. Part of the code path exists but because of the E10S split,
we need to add *extra* code to make TURN TCP work and that code isn't complete
(it's in the bug I pointed at in c12).


> Additionally, we have at least two content partners
> looking to build two communication style apps on FxOS, so we have be
> conscious that P2P communication works extremely well to get partner buy in
> to build those apps. On that regard, I see that this problem is a content
> blocker for 1.3 with severe impact, given the fact that audio communication
> is failing & the fact this impacts a key demo application.

Well, audio communication is hardly going to work any better if we turn
it off entirely. Note that even with TURN TCP there are still going to
be network environments where WebRTC fails (because they block TURN)
and where yet more aggressive proxying is needed. Moreover, consider
that if we had simply slipped TURN TCP to FF 30 we wouldn't even
be having this conversation.


> Note - one thing that would be interesting to look into is to figure out if
> the patch "turn off TURN TCP for gonk" isn't having negative impact to cause
> audio communication to fail here on FxOS.

See above.

Comment 15

[Jason Smith [:jsmith]]

(In reply to Eric Rescorla (:ekr) from comment #14)
> > Additionally, we have at least two content partners
> > looking to build two communication style apps on FxOS, so we have be
> > conscious that P2P communication works extremely well to get partner buy in
> > to build those apps. On that regard, I see that this problem is a content
> > blocker for 1.3 with severe impact, given the fact that audio communication
> > is failing & the fact this impacts a key demo application.
> 
> Well, audio communication is hardly going to work any better if we turn
> it off entirely. Note that even with TURN TCP there are still going to
> be network environments where WebRTC fails (because they block TURN)
> and where yet more aggressive proxying is needed. Moreover, consider
> that if we had simply slipped TURN TCP to FF 30 we wouldn't even
> be having this conversation.

In that case - What we probably need to better understand here is to get a picture of the scope of wifi network environments that will be impacted by this bug that FxOS supports. 

Note - I know we support a more limited scope of wifi networks (e.g. we don't support WPA-Enterprise in FxOS) in comparison to Desktop.

I'll do some wifi tests across different networks that I have available to see if this reproduces in other situations with apprtc.

Comment 16

[Eric Rescorla (:ekr)]

Well, I can tell you the properties: any network that blocks UDP more or less
entirely. Rather than going full AppRTC, I would just take a copy of
the turn unittest program (media/mtransport/tests) and run it
in the target network environment. If it succeeds, you should be good
to go. Otherwise not. If there is an environment where turn_unittest works
on both sides and apprtc does not, that's most likely some other kind of
defect

Comment 17

[Jason Smith [:jsmith]]

Okay - what I think we can do here is evangelize to partners that their P2P apps using TURN will only work on networks that do not have UDP packets blocked. So I'll clear nomination here & add a dependency to bug 950660.

Comment 18

[Eric Rescorla (:ekr)]

SGTM