623100 - mozcom-cdn.mozilla.net isn't resolving

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Mozilla.com is broken because mozcom-cdn.mozilla.net isn't resolving, at least internally:

C:\Users\stephend>tracert mozcom-cdn.mozilla.net
Unable to resolve target system name mozcom-cdn.mozilla.net.

C:\Users\stephend>nslookup mozcom-cdn.mozilla.net
Server:  mv-ns01.mv.mozilla.com
Address:  10.250.0.21

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to mv-ns01.mv.mozilla.com timed-out

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

This is just internal, from what I understand.

Comment 2

[Phong Tran [:phong]]

this looks to be an internal problem only, but still needs to be fixed.

Comment 3

[Ravi Pina [:ravi]]

[ravi@mvadm01 ~]$ dig @10.250.0.21 mozcom-cdn.mozilla.net any

; <<>> DiG 9.7.0-RedHat-9.7.0-1.rhel5 <<>> @10.250.0.21 mozcom-cdn.mozilla.net any
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45527
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 0

;; QUESTION SECTION:
;mozcom-cdn.mozilla.net.                IN      ANY

;; ANSWER SECTION:
mozcom-cdn.mozilla.net. 182     IN      CNAME   mozcom-cdn.3crowd.mozilla.net.

;; AUTHORITY SECTION:
mozilla.net.            286     IN      NS      ns1.mozilla.org.
mozilla.net.            286     IN      NS      ns2.mozilla.org.

;; Query time: 1 msec
;; SERVER: 10.250.0.21#53(10.250.0.21)
;; WHEN: Tue Jan  4 17:28:54 2011
;; MSG SIZE  rcvd: 119

Comment 4

[Ravi Pina [:ravi]]

This appears to be the record that is not working from the office:

[ravi@mvadm01 ~]$ dig @10.250.0.21 mozcom-cdn.3crowd.mozilla.net any

; <<>> DiG 9.7.0-RedHat-9.7.0-1.rhel5 <<>> @10.250.0.21 mozcom-cdn.3crowd.mozilla.net any
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Comment 5

[Ravi Pina [:ravi]]

So the issue appears to be only queries from SJC1 and any site that goes through it.  PHX1, for example, is working fine.

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

Turns out developer.mozilla.org is down, as well.  Bumping up a bit.

Comment 7

[Phong Tran [:phong]]

That is a separate problem that we have a bug for.  bug 599870.

the problem with the cdn is on 3crowd's side.

Comment 8

[Ravi Pina [:ravi]]

3crowed rolled back some changes and DNS resolves as expected:

[ravi@mvadm01 ~]$ dig @r1.3crowd.com mozcom-cdn.3crowd.mozilla.net any

; <<>> DiG 9.7.0-RedHat-9.7.0-1.rhel5 <<>> @r1.3crowd.com mozcom-cdn.3crowd.mozilla.net any
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34677
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;mozcom-cdn.3crowd.mozilla.net. IN      ANY

;; ANSWER SECTION:
mozcom-cdn.3crowd.mozilla.net. 1 IN     CNAME   n2.panthercdn.com.

;; Query time: 3 msec
;; SERVER: 74.115.30.30#53(74.115.30.30)
;; WHEN: Tue Jan  4 19:44:32 2011
;; MSG SIZE  rcvd: 78

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

stephen-donners-macbook-pro:~ stephendonner$ dig mozcom-cdn.mozilla.net

; <<>> DiG 9.6.0-APPLE-P2 <<>> mozcom-cdn.mozilla.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21332
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 2, ADDITIONAL: 0

;; QUESTION SECTION:
;mozcom-cdn.mozilla.net.		IN	A

;; ANSWER SECTION:
mozcom-cdn.mozilla.net.	56	IN	CNAME	mozcom-cdn.3crowd.mozilla.net.
mozcom-cdn.3crowd.mozilla.net. 1 IN	CNAME	n2.panthercdn.com.
n2.panthercdn.com.	20	IN	A	66.114.48.20
n2.panthercdn.com.	20	IN	A	66.114.48.11

;; AUTHORITY SECTION:
panthercdn.com.		112769	IN	NS	ns1.panthercdn.com.
panthercdn.com.		112769	IN	NS	ns2.panthercdn.com.

;; Query time: 13 msec
;; SERVER: 10.250.0.21#53(10.250.0.21)
;; WHEN: Wed Jan  5 10:08:48 2011
;; MSG SIZE  rcvd: 171

Verified FIXED.