568192 - 40% packet loss at MOZILLA-2.GigabitEthernet4-15.ar2.SNV2.gblx.net

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

This is from mtr between my laptop and the MPT VPN jumphost:
Host                                                 Loss%   Snt Drop    Avg
 1. 192.168.1.1                                        0.2%   894    2    1.3
 2. lo0.internet.ivpn.pe14.telstraclear.net            1.3%   894   12   57.0
 3. ???
 4. ie1-g-0-0-0.telstraclear.net                       1.7%   894   15   61.6
 5. ge-0-2-0-1.xcore1.acld.telstraclear.net            1.6%   894   14   61.5
 6. 203.167.233.10                                     1.5%   893   13   74.6
 7. i-14-1-0.wil-core02.bx.reach.com                   1.7%   893   15  208.0
 8. i-1-1.eqla01.bi.reach.com                          1.8%   893   16  209.1
 9. gblx-peer.eqla01.pr.reach.com                      1.7%   893   15  221.8
10. MOZILLA-2.GigabitEthernet4-15.ar2.SNV2.gblx.net   40.8%   893  364  209.2
11. v9.core2.sj.mozilla.com                           44.1%   893  394  198.1
12. ???

There's a big loss of packets between reach and MOZILLA-2.GigabitEthernet4-15.ar2.SNV2.gblx.net, which shows up as a very laggy shell on the jumphost. Load is very low on that box.

Comment 1

[Dave Miller [:justdave]]

our path to both ISC and Beijing are through gblx, and I've been getting paged frequently tonight about both of those locations being offline, however, I've not managed to catch an mtr trace that actually shows it yet.

Comment 2

[matthew zeier [:mrz]]

Reverse appears to go through level3...

Comment 3

[matthew zeier [:mrz]]

i've turned down globalcrossing peers.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

WFM, I'm routed via mzima.net now.

Comment 5

[matthew zeier [:mrz]]

This specific issue is resolved.  We'll work on troubleshooting GBLX elsewhere.