1278331 - SFO - Vidyo not working over guest WiFi network

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: AVOps: Vidyo, task)

Description

[Tony Recendez]

Vidyo desktop not working on the Guest Network in SFO. Works fine in MTV.

Gathering additional details to assist with troubleshooting.

Comment 1

[Dave Curado :dcurado]

If you can specify the source IP (the host on the guest network) and the destination IP address that host is trying to get to, on which port... that would be great.
Thank you.

Comment 2

[Mark Richards [:freshness]]

The K-Ingleside line's iMac is pulling the IP: 10.251.34.84 from the Mozilla Guest network

Comment 3

[Mark Richards [:freshness]]

N-Judah line is pulling the IP: 10.251.35.113 from the Mozilla Guest network

Comment 4

[Dave Curado :dcurado]

OK, great, Thanks!
Now, what destination IP and destination port are their clients trying to get to?
Thanks!

Comment 5

[Mark Richards [:freshness]]

J-Church line is pulling the IP: 10.251.34.164 from the Mozilla Guest network

Comment 6

[Dave Curado :dcurado]

Mark -- thank you, I've got enough source IP addresses now.

What I need now is/are destination IP and ports.

There have been no changes made to the office firewalls in some time.
The policies from the guest network out to the Internet, or to the corp zone, or to our internal VPN,
all match between mtv2 and sfo1.  (in fact, all our office firewalls now run the same configuration)

So, this is a bit of a puzzle.

Comment 7

[Mark Richards [:freshness]]

Totally understand, was just logging them individually because it's quite an arduous process retrieving the IP addresses from the iMacs.

What they're trying to hit is:
Portal: Name:	v.mozilla.com
Address: 63.245.215.252

Comment 8

[Dave Curado :dcurado]

Mark -- thank you, I've got enough source IP addresses now.

What I need now is/are destination IP and ports.

There have been no changes made to the office firewalls in some time.
The policies from the guest network out to the Internet, or to the corp zone, or to our internal VPN,
all match between mtv2 and sfo1.  (in fact, all our office firewalls now run the same configuration)

So, this is a bit of a puzzle.

Comment 9

[Dave Curado :dcurado]

Weird.
Can a host on the guest wifi ping v.mozilla.com?

some debugging:

from the firewall in sfo1, I can ping v.mozilla.com

[e-mail]> ping 63.245.215.252
PING 63.245.215.252 (63.245.215.252): 56 data bytes
64 bytes from 63.245.215.252: icmp_seq=0 ttl=60 time=21.413 ms
64 bytes from 63.245.215.252: icmp_seq=1 ttl=60 time=4.215 ms
64 bytes from 63.245.215.252: icmp_seq=2 ttl=60 time=4.464 ms
64 bytes from 63.245.215.252: icmp_seq=3 ttl=60 time=5.417 ms
^C
--- 63.245.215.252 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 4.215/8.877/21.413/7.251 ms


and if I trace the security policies, everything (except email) is
allowed out of the guest network...

[e-mail]> show route 63.245.215.252

inet.0: 101 destinations, 123 routes (100 active, 0 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 19w6d 10:20:56
                    > to 63.245.219.49 via reth0.1045 <-------
                    [BGP/170] 15w3d 22:29:58, MED 0, localpref 100
                      AS path: 3549 I
                    > to 159.63.23.37 via reth0.426
                    [Static/200] 19w6d 10:20:56
                    > to 159.63.23.37 via reth0.426

{primary:node0}
[e-mail]> show interfaces reth0.1045 | match zone
    Security: Zone: external <------------

{primary:node0}
[e-mail]> show security policies from-zone guest to-zone external
node0:
--------------------------------------------------------------------------
From zone: guest, To zone: external
  Policy: filter-smtp, State: enabled, Index: 26, Scope Policy: 0, Sequence number: 1
    Source addresses: guest-ww
    Destination addresses: any
    Applications: filter-smtp-app
    Action: deny
  Policy: filter-internal, State: enabled, Index: 27, Scope Policy: 0, Sequence number: 2
    Source addresses: any
    Destination addresses: mozilla-internal
    Applications: any
    Action: deny
  Policy: interwebs, State: enabled, Index: 28, Scope Policy: 0, Sequence number: 3
    Source addresses: any        <-------
    Destination addresses: any   <-------
    Applications: any            <-------
    Action: permit, log          <-------

Comment 10

[Mark Richards [:freshness]]

Yes, I'm able to ping v.mozilla.com from my laptop while on guest, while literally next to these conference rooms on the third floor.

Comment 11

[Mark Richards [:freshness]]

These particular machines were functioning without issue last week, and then on Monday they stopped working altogether on Mozilla guest.  Also, they are able to log in to Vidyo Desktop, just cannot complete any calls.

Comment 12

[Dave Curado :dcurado]

OK, great info, thank you.
Did anything change with v.mozilla in that time frame?
Not trying to point the finger of blame at AVops here, it's just that nothing
changed in SFO1 at all, and nothing changed on the core switches where v.mozilla.com connects.

Since you can ping it, we know reachability is fine.  There is some sort of filtering going
on somewhere.  

Do you happen to know which tcp/udp port(s) the clients use when sending packets to v.mozilla.com?

Thanks.

Comment 13

[Mark Richards [:freshness]]

Yes, UDP ports 50000-65535

Comment 14

[Dave Curado :dcurado]

From guest in sfo1 going out to v.mozilla.com, nothing is blocked.
(see comment 9)

From there packets go to border1.pao1, then border1.scl3 (no filtering there) and then
on to core1.scl3, where there is a packet filter, but packets to v.mozilla.com are explicitly allowed
through...

term vidyo {
    from {
        destination-prefix-list {
            vidyo-router;
            vidyo-portal; <---- this line includes v.mozilla.com
            vidyo-replay;
        }
    }
    then accept;
}

That said, the fact that we can ping from src to dest, but not make calls, certainly does *feel* like
a firewall/filter problem.  I'm just not seeing one.  (and nothing has changed in the past weeks/months
with this network gear)  I've pinged Guillermo to see if we can debug this together.

Comment 15

[Dave Curado :dcurado]

Guillermo has identified the problem.
The fix will require a CAB approval for a brief vidyo hit.

Comment 16

[Dave Curado :dcurado]

closing due to inactivity.
I believe Guillermo will open a service now case for the required CAB approval mentioned in
comment 15.

Comment 17

[Guillermo Huerta [Disabled MoCo account]]

Confirming this was a Vidyo Infrastructure configuration issue.

CHG0010646 has been approved to fix the config.

Comment 18

[Guillermo Huerta [Disabled MoCo account]]

Work completed