Closed Bug 442960 Opened 16 years ago Closed 16 years ago

Several Mozilla sites inaccessible (server not found)

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Nukeador, Assigned: mrz)

References

Details

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; es-ES; rv:1.9) Gecko/2008061015 Firefox/3.0
Build Identifier: 

We are receiving several report from Spanish users (http://www.mozilla-hispano.org/foro/viewtopic.php?f=2&t=1661) having problems to access http://www.mozilla.com/ http://www.mozilla-europe.org/ http://www.mozilla.org/ and http://addons.mozilla.org/es-ES/firefox/

They get a server not found error. I'm having these problems too using different DNS servers (80.58.61.250 from Telefonica España and 208.67.222.222 from OpenDNS).

I don't know where is the problem exactly.

Reproducible: Always
Assignee: nobody → server-ops
Severity: major → critical
Component: www.mozilla.org → Server Operations
OS: Linux → All
QA Contact: www-mozilla-org → justin
Hardware: PC → All
Summary: Serveral Mozilla sites inaccessible (server not found) → Several Mozilla sites inaccessible (server not found)
sgarrity mentioned this as well
pointing dnstracer at the Telefonica España nameserver gives me the following:

> 80.58.61.250 (80.58.61.250) Got answer 
> 80.58.61.250 (80.58.61.250)      www.mozilla.com -> www-mozilla-com.geo.mozilla.com
> 80.58.61.250 (80.58.61.250)      www-mozilla-com.geo.mozilla.com -> www-mozilla-com.glb.mozilla.com
> Tracing to www-mozilla-com.glb.mozilla.com via 80.58.61.250, timeout 15 seconds
> 80.58.61.250 (80.58.61.250) 
>  |\___ ns01.nslb.sj.mozilla.com [glb.mozilla.com] (63.245.209.101) Got authoritative answer 
>  |\___ ns01.anycast.mozilla.com [glb.mozilla.com] (63.245.223.10) Got authoritative answer 
>   \___ ns01.nllb.nl.mozilla.com [glb.mozilla.com] (63.245.213.101) Got authoritative answer 
>
> ns01.nllb.nl.mozilla.com (63.245.213.101) www-mozilla-com.glb.mozilla.com -> 63.245.213.13
> ns01.anycast.mozilla.com (63.245.223.10) www-mozilla-com.glb.mozilla.com -> 63.245.213.13
> ns01.nslb.sj.mozilla.com (63.245.209.101) www-mozilla-com.glb.mozilla.com -> 63.245.213.13
Summary: Several Mozilla sites inaccessible (server not found) → Serveral Mozilla sites inaccessible (server not found)
Summary: Serveral Mozilla sites inaccessible (server not found) → Several Mozilla sites inaccessible (server not found)
> macbook:~ dave$ dnstracer -s80.58.61.250 -o -4 www.mozilla-europe.org
> Tracing to www.mozilla-europe.org via 80.58.61.250, timeout 15 seconds
> 80.58.61.250 (80.58.61.250) Got answer 
>
> 80.58.61.250 (80.58.61.250)             www.mozilla-europe.org -> moz-euro.glb.mozilla.com
> macbook:~ dave$ dnstracer -s80.58.61.250 -o -4 moz-euro.glb.mozilla.com
> Tracing to moz-euro.glb.mozilla.com via 80.58.61.250, timeout 15 seconds
> 80.58.61.250 (80.58.61.250) 
>  |\___ ns01.nllb.nl.mozilla.com [glb.mozilla.com] (63.245.213.101) Got authoritative answer 
>  |\___ ns01.nslb.sj.mozilla.com [glb.mozilla.com] (63.245.209.101) Got authoritative answer 
>   \___ ns01.anycast.mozilla.com [glb.mozilla.com] (63.245.223.10) Got authoritative answer 
>
> ns01.anycast.mozilla.com (63.245.223.10) moz-euro.glb.mozilla.com -> 63.245.213.33
> ns01.nslb.sj.mozilla.com (63.245.209.101) moz-euro.glb.mozilla.com -> 63.245.213.33
> ns01.nllb.nl.mozilla.com (63.245.213.101) moz-euro.glb.mozilla.com -> 63.245.213.33

And we have a winner:

> macbook:~ dave$ dnstracer -s208.67.222.222 -o -4 www.mozilla-europe.org
> Tracing to www.mozilla-europe.org via 208.67.222.222, timeout 15 seconds
> 208.67.222.222 (208.67.222.222) 
>
> macbook:~ dave$ 

OpenDNS is returning no response.
Even having Telefonica España nameserver as primary one, I'm unable to access the sites.
I was wondering about that first response I got, I had to re-query with the CNAME it returned to get the IP address.  Usually the nameserver will follow the chain for you and include it all in the response...
Hmm, interesting.  When I query Telefonica España with dig, host, or dnstracer, I get the CNAME from www.mozilla-europe.org to moz-euro.glb.mozilla.com and that's it.

When I query them for moz-euro.glb.mozilla.com, dnstracer tells me it got a referral to Mozilla's nameservers, while host and dig both tell me they got an NXDOMAIN response.
Er, to clarify, dig says SERVFAIL, host says NXDOMAIN
SERVFAIL is a response I would actually expect if I queried a nameserver that an ISP had closed for customer use only.  Anyone using Telefonica España have Linux or Mac OS X and can try some of those tools and paste the output?
Yes, tell me what to do.
dig @80.58.61.250 www.mozilla-europe.org

dig @80.58.61.250 moz-euro.glb.mozilla.com

dnstracer -s80.58.61.250 -o -4 www.mozilla-europe.org

dnstracer -s80.58.61.250 -o -4 moz-euro.glb.mozilla.com
Assignee: server-ops → justdave
nuke@nuke-laptop:~$ dig @80.58.61.250 www.mozilla-europe.org

; <<>> DiG 9.4.2 <<>> @80.58.61.250 www.mozilla-europe.org
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 32700
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.mozilla-europe.org.                IN      A

;; ANSWER SECTION:
www.mozilla-europe.org. 31      IN      CNAME   moz-euro.glb.mozilla.com.

;; Query time: 71 msec
;; SERVER: 80.58.61.250#53(80.58.61.250)
;; WHEN: Tue Jul  1 20:02:42 2008
;; MSG SIZE  rcvd: 78

nuke@nuke-laptop:~$ dig @80.58.61.250 moz-euro.glb.mozilla.com

; <<>> DiG 9.4.2 <<>> @80.58.61.250 moz-euro.glb.mozilla.com
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 52658
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;moz-euro.glb.mozilla.com.	IN	A

;; Query time: 70 msec
;; SERVER: 80.58.61.250#53(80.58.61.250)
;; WHEN: Tue Jul  1 20:02:59 2008
;; MSG SIZE  rcvd: 42

nuke@nuke-laptop:~$ dnstracer -s80.58.61.250 -o -4 www.mozilla-europe.org

Tracing to www.mozilla-europe.org[a] via 80.58.61.250, maximum of 3 retries
80.58.61.250 (80.58.61.250) Got answer [received type is cname] 
 |\___ ns01.nslb.sj.mozilla.com [glb.mozilla.com] (63.245.209.101) Refers backwards 
 |\___ ns01.anycast.mozilla.com [glb.mozilla.com] (63.245.223.10) Refers backwards 
  \___ ns01.nllb.nl.mozilla.com [glb.mozilla.com] (63.245.213.101) Refers backwards 

80.58.61.250 (80.58.61.250)             www.mozilla-europe.org -> moz-euro.glb.mozilla.com

nuke@nuke-laptop:~$ dnstracer -s80.58.61.250 -o -4 moz-euro.glb.mozilla.com

Tracing to moz-euro.glb.mozilla.com[a] via 80.58.61.250, maximum of 3 retries
80.58.61.250 (80.58.61.250) 
 |\___ ns01.nslb.sj.mozilla.com [glb.mozilla.com] (63.245.209.101) Got authoritative answer 
 |\___ ns01.anycast.mozilla.com [glb.mozilla.com] (63.245.223.10) Got authoritative answer 
  \___ ns01.nllb.nl.mozilla.com [glb.mozilla.com] (63.245.213.101) Got authoritative answer 

ns01.nllb.nl.mozilla.com (63.245.213.101) moz-euro.glb.mozilla.com -> 63.245.213.33
ns01.anycast.mozilla.com (63.245.223.10) moz-euro.glb.mozilla.com -> 63.245.213.33
ns01.nslb.sj.mozilla.com (63.245.209.101) moz-euro.glb.mozilla.com -> 63.245.209.63

I'm not able to get anything mozilla.com or mozilla.org either.

In case it's helpful: 

$ dig @24.222.0.94 www.mozilla-europe.org

; <<>> DiG 9.5.0rc1 <<>> @24.222.0.94 www.mozilla-europe.org
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 54111
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.mozilla-europe.org.		IN	A

;; ANSWER SECTION:
www.mozilla-europe.org.	60	IN	CNAME	moz-euro.glb.mozilla.com.

;; Query time: 113 msec
;; SERVER: 24.222.0.94#53(24.222.0.94)
;; WHEN: Tue Jul  1 15:25:31 2008
;; MSG SIZE  rcvd: 78



$ dig @24.222.0.94 moz-euro.glb.mozilla.com

; <<>> DiG 9.5.0rc1 <<>> @24.222.0.94 moz-euro.glb.mozilla.com
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 33878
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;moz-euro.glb.mozilla.com.	IN	A

;; Query time: 33 msec
;; SERVER: 24.222.0.94#53(24.222.0.94)
;; WHEN: Tue Jul  1 15:28:17 2008
;; MSG SIZE  rcvd: 42



$ dnstracer -s24.222.0.94 -o -4 www.mozilla-europe.org
Tracing to www.mozilla-europe.org[a] via 24.222.0.94, maximum of 3 retries
24.222.0.94 (24.222.0.94) 
 |\___ ns2.mozilla.org [mozilla-europe.org] (140.211.166.194) Got authoritative answer [received type is cname] 
  \___ ns1.mozilla.org [mozilla-europe.org] (63.245.208.161) Got authoritative answer [received type is cname] 

ns1.mozilla.org (63.245.208.161)        www.mozilla-europe.org -> moz-euro.glb.mozilla.com
ns2.mozilla.org (140.211.166.194)       www.mozilla-europe.org -> moz-euro.glb.mozilla.com


$ dnstracer -s24.222.0.94 -o -4 moz-euro.glb.mozilla.com
Tracing to moz-euro.glb.mozilla.com[a] via 24.222.0.94, maximum of 3 retries
24.222.0.94 (24.222.0.94) 
 |\___ ns01.nllb.nl.mozilla.com [glb.mozilla.com] (63.245.213.101) Got authoritative answer 
 |\___ ns01.nslb.sj.mozilla.com [glb.mozilla.com] (63.245.209.101) Got authoritative answer 
  \___ ns01.anycast.mozilla.com [glb.mozilla.com] (63.245.223.10) Got authoritative answer 

ns01.anycast.mozilla.com (63.245.223.10) moz-euro.glb.mozilla.com -> 63.245.209.63
ns01.nslb.sj.mozilla.com (63.245.209.101) moz-euro.glb.mozilla.com -> 63.245.209.63
ns01.nllb.nl.mozilla.com (63.245.213.101) moz-euro.glb.mozilla.com -> 63.245.209.63
Status: UNCONFIRMED → NEW
Ever confirmed: true
OK, so the people talking about this on IRC say it's been broken for about 2 days.  The timing on that coincides with an OS upgrade on our Netscalers, which are what serve the DNS for the glb.mozilla.com zone.  Not to say that's the problem, but it's highly suspicious.  Recursive DNS servers running BIND and TinyDNS are both known to properly resolve these domains still.  Anyone happen to know what DNS software Telstra and Telefónica España are using?
Just for everyone's edification, our office has received at least 10 emails today from users in Japan who can't see our sites.  It seems limited to certain ISPs (here in Japan, OCN - the largest consumer ISP - and DION users can't see our sites.)

I have checked DNS servers of OCN out at Japan, and they do not
find out "www.mozilla,org". I use IP addresses from a below url.

http://www.ocn.ne.jp/info/tech/netset/

The primary server and the secondary can't resolve "www.mozilla.org"
at this point.Many people complain about the issue to Mozilla Japan,
but unfortunately, we can do nothing to it
So we hope to fix the problem ASAP at MC IT team.


**** Primary server*****
======= Strarting log messages ========
C:\Users\shigeru>nslookup - 202.234.232.6
既定のサーバー: nv-td501.ocn.ad.jp
Address: 202.234.232.6:53

> > www.mozilla.org
サーバー: nv-td501.ocn.ad.jp
Address: 202.234.232.6:53

*** nv-td501.ocn.ad.jp が www.mozilla.org を見つけられません: Server failed

> > wwww.mozilla.jp
サーバー: nv-td501.ocn.ad.jp
Address: 202.234.232.6:53

*** nv-td501.ocn.ad.jp が wwww.mozilla.jp を見つけられません:
Non-existent domain
===== End of the messages =========


**** Secondary server*****

======= Strarting log messages ========
C:\Users\shigeru>nslookup - 221.113.139.250
既定のサーバー: nv-kd501.ocn.ad.jp
Address: 221.113.139.250:53

> > www.mozilla.org
サーバー: nv-kd501.ocn.ad.jp
Address: 221.113.139.250:53

*** nv-kd501.ocn.ad.jp が www.mozilla.org を見つけられません: Server failed

===== End of the messages =========



I got lost off the CC on the Citrix case on this so I have no idea what the status is (there's apparently been action on it but I don't know what).  mrz's been the one in contact with Citrix.
Assignee: justdave → mrz
I've disabled the gslb CNAMEs in geodns and added a A RR for both www.mozilla.com and www.mozilla.org.  

Commented out the gslb CNAME for AMO and added San Jose's A RR. 

Citrix fell off the map lastnight debugging.
mozilla.org and mozilla.com are now unavailable from France
Mozilla Addons, Mozilla.org, Mozilla.com and Mozilla Europe are working now from Spain.
Himorin confirms on IRC working from Japan now also.
sites are still unaccessible from paris
A few of the major ISPs (apparently in an attempt to keep their servers fast?) appear to ignore the TTLs and cache lookups for a longer period of time.  They should start coming back online as your ISP's DNS servers start expiring their cache.  If you happen to know someone at your ISP with the power to kick it, wouldn't hurt to ask them.
mozilla-europe.org is still down for Spanish users.
Possible fix coming...
(In reply to comment #25)
> mozilla-europe.org is still down for Spanish users.
> 

Can you try this again?  Citrix provided what they believe is the fix but I don't have an easy way to test.
(In reply to comment #27)
> 
> Can you try this again?  Citrix provided what they believe is the fix but I
> don't have an easy way to test.
> 

Yes, now Mozilla Europe is working here, thanks.


Great!

I've moved www.mozilla.com, www.mozilla.org and addons.mozilla.org back into GSLB.  
this is fixed.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.