652156 - Bugzilla is slow from Australia since the move to PHX

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Dave Miller [:justdave]]

07:34:37 < glob> ping has increased from 200ms to 500/600ms
07:35:26 < glob> the hop between i-2-1.paix01.bi.reach.com and 206.111.12.173.ptr.us.xo.net being the reason
07:47:36 < glob> traces at http://bugzilla.pastebin.mozilla.org/1209923

Comment 1

[Dave Miller [:justdave]]

The mentioned traceroute, in case the pastebin expires:

old, now:

Tracing route to dyna-bugzilla.acelb.sj.mozilla.com [63.245.209.72]
over a maximum of 30 hops:

  1    <1 ms     1 ms    <1 ms  10.1.1.1
  2    15 ms    14 ms    14 ms  nexthop.wa.iinet.net.au [203.215.5.243]
  3    14 ms    14 ms    15 ms  te7-2.per-qv1-bdr2.iinet.net.au [203.215.4.38]
  4    72 ms    74 ms    75 ms  xe-3-3-0-0.syd-mas-core1.iinet.net.au [203.215.20.68]
  5   220 ms   220 ms   220 ms  34023.sydp06.cu.reach.com [134.159.160.57]
  6   221 ms   220 ms   220 ms  i-0-5-0-0.tlot-core01.bx.reach.com [202.84.140.102]
  7   224 ms   224 ms   224 ms  i-1-1.eqla01.bi.reach.com [202.84.251.194]
  8   258 ms   257 ms   261 ms  gblx-peer.eqla01.pr.reach.com [134.159.63.202]
  9   307 ms   251 ms   251 ms  MOZILLA.TenGigabitEthernet2-1.ar1.SNV2.gblx.net [64.210.30.186]
 10   249 ms   250 ms   249 ms  te-8-2.core2.sjc.mozilla.net [63.245.208.98]
 11   258 ms   257 ms   258 ms  dyna-bugzilla.acelb.sj.mozilla.com [63.245.209.72]

new, now:

Tracing route to bugzilla-mozilla-org.glb.mozilla.net [63.245.217.60]
over a maximum of 30 hops:

  1    <1 ms     1 ms     1 ms  10.1.1.1
  2    14 ms    14 ms    14 ms  nexthop.wa.iinet.net.au [203.215.5.243]
  3    14 ms    14 ms    14 ms  te7-2.per-qv1-bdr2.iinet.net.au [203.215.4.38]
  4    70 ms    72 ms    74 ms  xe-3-3-0-0.syd-mas-core1.iinet.net.au [203.215.20.68]
  5   231 ms   231 ms   230 ms  34023.sydp06.cu.reach.com [134.159.160.57]
  6   263 ms   288 ms   266 ms  i-0-0-2-0.paix-core01.bx.reach.com [202.84.140.106]
  7   223 ms   223 ms   222 ms  i-2-1.paix01.bi.reach.com [202.84.251.101]
  8   454 ms   451 ms   456 ms  206.111.12.173.ptr.us.xo.net [206.111.12.173]
  9   466 ms   450 ms   458 ms  te-3-4-0.rar3.sanjose-ca.us.xo.net [207.88.12.177]
 10   472 ms   469 ms   466 ms  te-3-0-0.rar3.la-ca.us.xo.net [207.88.12.62]
 11   464 ms   457 ms   464 ms  ae0d0.mcr1.phoenix-az.us.xo.net [216.156.0.130]
 12   477 ms   476 ms   467 ms  ae1d0.mcr2.phoenix-az.us.xo.net [216.156.1.66]
 13   481 ms   488 ms   488 ms  ip65-47-255-6.z255-47-65.customer.algx.net [65.47.255.6]
 14   486 ms   480 ms   473 ms  xe-1-0-1.core2.phx1.mozilla.net [63.245.216.26]
 15   488 ms   484 ms   477 ms  v-500.fw1.phx1.mozilla.net [63.245.216.33]
 16   488 ms   485 ms   484 ms  bugzilla.zlb.phx.mozilla.net [63.245.217.60]

new, 3 hours ago:

Tracing route to bugzilla-mozilla-org.glb.mozilla.net [63.245.217.60]
over a maximum of 30 hops:

  1     2 ms    <1 ms    <1 ms  10.1.1.1
  2    14 ms    14 ms    14 ms  nexthop.wa.iinet.net.au [203.215.5.243]
  3    14 ms    14 ms    14 ms  te7-2.per-qv1-bdr2.iinet.net.au [203.215.4.38]
  4    68 ms    67 ms    68 ms  xe-3-3-0-0.syd-mas-core1.iinet.net.au [203.215.20.68]
  5   221 ms   221 ms   220 ms  34023.sydp06.cu.reach.com [134.159.160.57]
  6   222 ms   222 ms   223 ms  i-0-0-2-0.paix-core01.bx.reach.com [202.84.140.106]
  7   223 ms   223 ms   222 ms  i-2-1.paix01.bi.reach.com [202.84.251.101]
  8   637 ms   636 ms   648 ms  206.111.12.173.ptr.us.xo.net [206.111.12.173]
  9   632 ms   627 ms   624 ms  te-3-4-0.rar3.sanjose-ca.us.xo.net [207.88.12.177]
 10   662 ms   653 ms   646 ms  te-3-0-0.rar3.la-ca.us.xo.net [207.88.12.62]
 11   651 ms   652 ms   659 ms  ae0d0.mcr1.phoenix-az.us.xo.net [216.156.0.130]
 12   664 ms   655 ms   651 ms  ae1d0.mcr2.phoenix-az.us.xo.net [216.156.1.66]
 13   643 ms   647 ms   658 ms  ip65-47-255-6.z255-47-65.customer.algx.net [65.47.255.6]
 14   655 ms   657 ms   652 ms  xe-1-0-1.core2.phx1.mozilla.net [63.245.216.26]
 15   668 ms   639 ms   647 ms  v-500.fw1.phx1.mozilla.net [63.245.216.33]
 16   657 ms   656 ms   640 ms  bugzilla.zlb.phx.mozilla.net [63.245.217.60]

Comment 2

[:glob ✱]

things have just significantly improved, with the hop between i-2-1.paix01.bi.reach.com and 206.111.12.173.ptr.us.xo.net dropping from 200ms to 400ms down to 0ms!


Tracing route to bugzilla-mozilla-org.glb.mozilla.net [63.245.217.60]
over a maximum of 30 hops:

  1    <1 ms     1 ms    <1 ms  10.1.1.1
  2    15 ms    14 ms    14 ms  nexthop.wa.iinet.net.au [203.215.5.243]
  3    14 ms    14 ms    14 ms  te7-2.per-qv1-bdr2.iinet.net.au [203.215.4.38]
  4    68 ms    69 ms    69 ms  xe-3-3-0-0.syd-mas-core1.iinet.net.au [203.215.20.68]
  5   220 ms   220 ms   220 ms  34023.sydp06.cu.reach.com [134.159.160.57]
  6   223 ms   222 ms   223 ms  i-0-0-2-0.paix-core01.bx.reach.com [202.84.140.106]
  7   222 ms   222 ms   223 ms  i-2-1.paix01.bi.reach.com [202.84.251.101]
  8   222 ms   222 ms   222 ms  206.111.12.173.ptr.us.xo.net [206.111.12.173]
  9   223 ms   225 ms   223 ms  te-3-4-0.rar3.sanjose-ca.us.xo.net [207.88.12.177]
 10   243 ms   242 ms   241 ms  te-3-0-0.rar3.la-ca.us.xo.net [207.88.12.62]
 11   241 ms   241 ms   241 ms  ae0d0.mcr1.phoenix-az.us.xo.net [216.156.0.130]
 12   240 ms   241 ms   240 ms  ae1d0.mcr2.phoenix-az.us.xo.net [216.156.1.66]
 13   252 ms   241 ms   241 ms  ip65-47-255-6.z255-47-65.customer.algx.net [65.47.255.6]
 14   241 ms   241 ms   241 ms  xe-1-0-1.core2.phx1.mozilla.net [63.245.216.26]
 15   241 ms   241 ms   242 ms  v-500.fw1.phx1.mozilla.net [63.245.216.33]
 16   241 ms   241 ms   241 ms  bugzilla.zlb.phx.mozilla.net [63.245.217.60]

Comment 3

[:glob ✱]

i'll mark this as fixed :)

Comment 4

[:glob ✱]

it's back to being slow, however i guess there isn't much mozilla can do about routing outside its network, so i'll leave this bug closed.

  7   223 ms   225 ms   223 ms  i-2-1.paix01.bi.reach.com [202.84.251.101]
  8   460 ms   463 ms   467 ms  206.111.12.173.ptr.us.xo.net [206.111.12.173]