683829 - network slowness between sjc1 and scl1

Status: RESOLVED DUPLICATE of bug 688506

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Noah Meyerhans [:noahm]]

Just so there's a record of it:

While looking into some alerts related to the Mercurial mirror infrastructure I've been working on, I observed the following:

[root@dm-svn02 main]# ping -n -A -c 30 hg1.build.scl1.mozilla.com 
PING hg1.build.scl1.mozilla.com (10.12.51.200) 56(84) bytes of data.
...

--- hg1.build.scl1.mozilla.com ping statistics ---
30 packets transmitted, 30 received, 0% packet loss, time 45745ms
rtt min/avg/max/mdev = 129.399/1472.295/3621.313/939.301 ms, pipe 3, ipg/ewma 1577.427/1482.355 ms

This has been generally repeatable, however occasionally everything will suddenly speed up and seem much closer to normal:
[root@dm-svn02 main]# ping -n -A -c 30 hg1.build.scl1.mozilla.com 
PING hg1.build.scl1.mozilla.com (10.12.51.200) 56(84) bytes of data.
...

--- hg1.build.scl1.mozilla.com ping statistics ---
30 packets transmitted, 30 received, 0% packet loss, time 61ms
rtt min/avg/max/mdev = 1.906/2.084/2.451/0.126 ms, ipg/ewma 2.129/2.035 ms

However, it seems slow more often than not.  Unfortunately, I don't have any more concrete data.  This was observed at roughly 00:00 PDT on Thursday.  Pings above are from dm-svn02 (in MPT) to hg1.build.scl1.mozilla.com in scl1.  I haven't observed packet loss at any point, just long and variable RTTs.

Comment 1

[Aaron]

Can you do an mtr or traceroute so we can try to narrow down what is causing the slowness?  

There has been a lot of work done on the connection between sjc1 and scl1 lately to improve the speed of the link, but I'm not sure if its entirely complete.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

That connection is back to running over a VPN tunnel between SRX650's in sjc1 and scl1.  That's a connection known to have high latency under load, but low packet loss.  So I think this is a known state, and can probably be duped to one of the (several..) releng bugs on the topic.