856390 - alk1 dns/network issues

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Michael Burns [:mburns]]

Have had some lag issues/dns pages with admin1{a,b}.akl1.

[[e-mail]1 ~]# dig @localhost mozilla.com
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.4 <<>> @localhost mozilla.com
; (2 servers found)
;; global options: +cmd
;; connection timed out; no servers could be reached

[[e-mail]1 ~]# service named status
version: 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.4
CPUs found: 4
worker threads: 4
number of zones: 766
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 3
query logging is OFF
recursive clients: 1/0/1000
tcp clients: 0/100
server is up and running
named (pid  9837) is running...

[[e-mail]1 ~]# tail -f /var/named/chroot/var/log/named/general
31-Mar-2013 06:22:54.499 xfer-in: error: transfer of 'mozilla.pt/IN/offices-in' from 81.92.212.12#53: failed to connect: timed out
31-Mar-2013 06:22:54.499 xfer-in: info: transfer of 'mozilla.pt/IN/offices-in' from 81.92.212.12#53: Transfer completed: 0 messages, 0 records, 0 bytes, 21.000 secs (0 bytes/sec)

Comment 1

[Michael Burns [:mburns]]

<smokeping>	 Smokeping: Offices.AKL1.v-mozilla-com [from admin1b.private.akl1.mozilla.com] is WARNING (alert: warn-loss20p, loss: 0%, 100%, 100%, 100%, rtt: 140ms, U, U, U)
<smokeping>	 Smokeping: Offices.AKL1.Bugzilla [from admin1b.private.akl1.mozilla.com] is OK (alert: warn-loss20p, loss: 100%, 100%, 100%, 100%, rtt: U, U, U, U)

Sat 23:41:38 PDT [517] admin1b.private.akl1.mozilla.com:Recursive DNS is OK: DNS OK: 0.384 seconds response time. google.com returns 74.125.237.64,74.125.237.65,74.125.237.66,74.125.237.67,74.125.237.68,74.125.237.69,74.125.237.70,74.125.237.71,74.125.237.72,74.125.237.73,74.125.237.78

Sat 23:42:18 PDT [518] admin1a.private.akl1.mozilla.com:Recursive DNS is OK: DNS OK: 0.381 seconds response time. google.com returns 74.125.237.32,74.125.237.33,74.125.237.34,74.125.237.35,74.125.237.36,74.125.237.37,74.125.237.38,74.125.237.39,74.125.237.40,74.125.237.41,74.125.237.46

Comment 2

[Michael Burns [:mburns]]

The network connectivity seems to be flapping.


[[e-mail]1 ~]# ping 74.125.224.206
PING 74.125.224.206 (74.125.224.206) 56(84) bytes of data.
^Cp
--- 74.125.224.206 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2359ms

[[e-mail]1 ~]# dig google.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.4 <<>> google.com
;; global options: +cmd
;; connection timed out; no servers could be reached

<nagios-scl3> Sun 03:56:09 PDT [578] ringring.mv.mozilla.com:Asterisk Trunk - SIP/aukland_sip is CRITICAL: SIP/aukland_sip UNREACHABLE

Comment 3

[Michael Burns [:mburns]]

[[e-mail]1 ~]# ping google.com
PING google.com (74.125.237.34) 56(84) bytes of data.
64 bytes from syd01s05-in-f2.1e100.net (74.125.237.34): icmp_seq=1 ttl=56 time=46.3 ms
64 bytes from syd01s05-in-f2.1e100.net (74.125.237.34): icmp_seq=2 ttl=56 time=46.2 ms
...

Comment 4

[Michael Burns [:mburns]]

This flapped over the weekend, but has recovered and stayed online. Closing.