628472 - 3crowd resources inaccessible from some locations

Status: RESOLVED INVALID

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Pascal Chevrel:pascalc]

Mozilla.com has no css nor videos, all the content on http://mozcom-cdn.mozilla.net is not loaded.

http://www.downforeveryoneorjustme.com/mozcom-cdn.mozilla.net
It's not just you! http://mozcom-cdn.mozilla.net looks down from here.

http://isthatsitedown.com/check!mozcom-cdn.mozilla.net
Check for : mozcom-cdn.mozilla.net
Iam really sorry , mozcom-cdn.mozilla.net is down , try after a while pleases

Comment 1

[Ben Kero [:bkero]]

We're trying to replicate this.

Comment 2

[Ravi Pina [:ravi]]

I am not able to reproduce this from the Paris office.

Can you do `dig mozcom-cdn.mozilla.net` please?

Comment 3

[Pascal Chevrel:pascalc]

dig mozcom-cdn.mozilla.net

; <<>> DiG 9.7.1-P2 <<>> mozcom-cdn.mozilla.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 20728
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;mozcom-cdn.mozilla.net.		IN	A

;; ANSWER SECTION:
mozcom-cdn.mozilla.net.	70	IN	CNAME	mozcom-cdn.3crowd.mozilla.net.

;; Query time: 971 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Tue Jan 25 02:54:35 2011
;; MSG SIZE  rcvd: 72



Note that Ehsan who saw this bug earlier is probably in Canada, so not it's not just from Paris

Comment 4

[Ravi Pina [:ravi]]

Are you at home?  Who is your ISP?  Can you do a `dig mozcom-cdn.3crowd.mozilla.net any`?

Comment 5

[Pascal Chevrel:pascalc]

I am at home, my ISP is SFR (second biggest ISP after Orange so if it's ISP related, it may affect a large chunk of the population)

dig mozcom-cdn.3crowd.mozilla.net any

; <<>> DiG 9.7.1-P2 <<>> mozcom-cdn.3crowd.mozilla.net any
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52386
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;mozcom-cdn.3crowd.mozilla.net.	IN	ANY

;; ANSWER SECTION:
mozcom-cdn.3crowd.mozilla.net. 176 IN	NS	r1.3crowd.com.

;; ADDITIONAL SECTION:
r1.3crowd.com.		890	IN	A	74.115.30.30

;; Query time: 40 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Tue Jan 25 03:05:14 2011
;; MSG SIZE  rcvd: 90

Comment 7

[Ravi Pina [:ravi]]

Spoke with engineer at 3Crowd about the issue and relayed data to help with his troubleshooting.

Updated will be made as I get them.

Comment 8

[Pascal Chevrel:pascalc]

thanks

Comment 9

[Ravi Pina [:ravi]]

A triggering event occurred that sent 3crowd destined traffic from some locations to an alternative 3crowd site (Asia).  There appeared to be unrelated issues to 3crowd on the path to this alternate site that was causing the resources to be inaccessible.  The routes have been dropped to circumvent this bad path so things should be available again.

Please confirm things are reachable.

Comment 10

[Pascal Chevrel:pascalc]

This is now working for me.

Comment 11

[Ravi Pina [:ravi]]

Excellent.

Comment 12

[Pascal Chevrel:pascalc]

reopened, down again, this time from the paris office

Comment 13

[Pascal Chevrel:pascalc]

dig mozcom-cdn.mozilla.net

; <<>> DiG 9.7.1-P2 <<>> mozcom-cdn.mozilla.net
;; global options: +cmd
;; connection timed out; no servers could be reached

Comment 14

[Pascal Chevrel:pascalc]

and now it is working again. I am not sure what I should do about that because this happens too often and is hard to reproduce...

Comment 15

[Ravi Pina [:ravi]]

The problem you are resenting is unrelated to the problem of the original description.  For example in Comment 13 "connection timed out; no servers could be reached" suggests more an issue between your client and your configured DNS and not 3crowd.

In the future please open a new bug providing the output of `dig mozcom-cdn.3crowd.mozilla.net any`, your location, your configured DNS, and time of the query that will help.