Closed Bug 647254 Opened 14 years ago Closed 14 years ago

Comcast DNSSEC nameservers not resolving irc.mozilla.org

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
minor

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: abillings, Assigned: fox2mike)

Details

Starting in the afternoon of March 31, I can no longer resolve irc.mozilla.org (and possibly some other, but not all, mozilla.org addresses) using comcast's DNS. In order to get onto IRC, I must turn on VPN to the MV network and then I can resolve things. That switches my DNS over to Mozilla's and things work fine. This has been noticed on my home comcast network but no the Internet cafe two blocks away. I've debugged it (and had the same results) on two different computers on my comcast network.
Confirmed it fails from a node in Pleasanton.
Comcast is giving me: * 75.75.75.75 * 75.75.75.76 as nameservers.
jabba says this works in Milpitas but he doesn't know his nameservers.
Group: infra
I suspected DNSSEC, but this is not the case : http://dnsviz.net/d/irc.mozilla.org/dnssec/ Of course, we added our standby key to the signatures (automatically), since the old one is now over 6 months old. This has affected nothing. I can't see why comcast isn't resolving stuff. If we can rule out our end, I'll poke some folks at Comcast and see what's up.
Al, what are the nameserver addresses you are getting from comcast?
http://www.dnssec.comcast.net/ says 75.75.75.75 and 75.75.76.76 are DNSSEC validating resolvers FWIW.
I'm using 75.75.75.75 and .76, per what my router is telling me.
(In reply to comment #7) > I'm using 75.75.75.75 and .76, per what my router is telling me. What boxes do you have at your disposal? linux? mac? can you do a dig @75.75.75.75 irc.mozilla.org and repeat the same for 76 too? Also, can you do a dig +dnssec @75.75.75.75 irc.mozilla.org and paste these results?
mrz-mb:~ mrz$ dig @75.75.75.75 irc.mozilla.org ; <<>> DiG 9.6.0-APPLE-P2 <<>> @75.75.75.75 irc.mozilla.org ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 43829 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;irc.mozilla.org. IN A ;; Query time: 3121 msec ;; SERVER: 75.75.75.75#53(75.75.75.75) ;; WHEN: Fri Apr 1 21:19:10 2011 ;; MSG SIZE rcvd: 33 mrz-mb:~ mrz$ dig @75.75.75.76 irc.mozilla.org ; <<>> DiG 9.6.0-APPLE-P2 <<>> @75.75.75.76 irc.mozilla.org ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached
fwiw it is 75.75.76.76 and 75.75.75.75 and they have recently moved all of their nameservers to auth DNSSEC: AirMax:Sites $ dig irc.mozilla.org @75.75.76.76 ; <<>> DiG 9.6.0-APPLE-P2 <<>> irc.mozilla.org @75.75.76.76 ;; global options: +cmd ;; connection timed out; no servers could be reached AirMax:Sites $ dig irc.mozilla.org @75.75.75.75 ; <<>> DiG 9.6.0-APPLE-P2 <<>> irc.mozilla.org @75.75.75.75 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 16476 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;irc.mozilla.org. IN A ;; Query time: 3778 msec ;; SERVER: 75.75.75.75#53(75.75.75.75) ;; WHEN: Fri Apr 1 22:27:22 2011 ;; MSG SIZE rcvd: 33 NON irc.mozilla.org works on both: AirMax:Sites $ dig www.mozilla.org @75.75.75.75 ; <<>> DiG 9.6.0-APPLE-P2 <<>> www.mozilla.org @75.75.75.75 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36906 ;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.mozilla.org. IN A ;; ANSWER SECTION: www.mozilla.org. 38 IN CNAME www-mozilla-org.geo.mozilla.com. www-mozilla-org.geo.mozilla.com. 3320 IN CNAME www-mozilla-org.glb.mozilla.net. www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.217.21 ;; Query time: 37 msec ;; SERVER: 75.75.75.75#53(75.75.75.75) ;; WHEN: Fri Apr 1 22:27:46 2011 ;; MSG SIZE rcvd: 203 AirMax:Sites $ dig www.mozilla.org @75.75.76.76 ; <<>> DiG 9.6.0-APPLE-P2 <<>> www.mozilla.org @75.75.76.76 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61990 ;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.mozilla.org. IN A ;; ANSWER SECTION: www.mozilla.org. 29 IN CNAME www-mozilla-org.geo.mozilla.com. www-mozilla-org.geo.mozilla.com. 3494 IN CNAME www-mozilla-org.glb.mozilla.net. www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.209.11 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.209.11 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.209.11 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.209.11 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.209.11 ;; Query time: 44 msec ;; SERVER: 75.75.76.76#53(75.75.76.76) ;; WHEN: Fri Apr 1 22:27:52 2011 ;; MSG SIZE rcvd: 203 so it is specific to the irc host.
Billy, Can you re-run those queries with the +dnssec flag? thanks!
(In reply to comment #10) And more importantly it works with +dnssec flag, but I'm fairly certain it didn't a few hours ago... ravi-mozilla-mbp:~ ravi$ dig +dnssec www.mozilla.org @75.75.75.75 ; <<>> DiG 9.6.0-APPLE-P2 <<>> +dnssec www.mozilla.org @75.75.75.75 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32103 ;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 4000 ;; QUESTION SECTION: ;www.mozilla.org. IN A ;; ANSWER SECTION: www.mozilla.org. 53 IN CNAME www-mozilla-org.geo.mozilla.com. www.mozilla.org. 53 IN RRSIG CNAME 7 3 60 20110429150652 20110330151615 17852 mozilla.org. n5mzCliw5bDwcuX64cbrCNGaSM5QPghAj0h3esHokNpDLZvNe7lBsbOT lnj9UhMQkYKzZq+Yvd3WI3tmzsEeGXCFq7VDMP7bvamY8mM/Xl1w4jVd oaLIUF698epJB8vQWrvIIAGsSvSZlUEpXYMAtDAfBOj7Wy0+1c6UrZ8i ze4= www.mozilla.org. 53 IN RRSIG CNAME 7 3 60 20110429151032 20110330151615 62897 mozilla.org. x4Q1TwbPWsj63LFOt4VYxQzVPWkIx4mYrzbhHrYjA6pXLFh9UXEfNS7z kKKxmMKYhbWwXYBtwRMThd75WCraLf9ClV/at4k8pL/LyMZpeB6iP91d 8TjGM1gfOSc/6oMhxr9T3LE0QRyRp6Paqg3TVhoK90SEm6NYqWSX5ORU E8A= www-mozilla-org.geo.mozilla.com. 3576 IN CNAME www-mozilla-org.glb.mozilla.net. www-mozilla-org.glb.mozilla.net. 6 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 6 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 6 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 6 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 6 IN A 63.245.217.21 ;; Query time: 25 msec ;; SERVER: 75.75.75.75#53(75.75.75.75) ;; WHEN: Fri Apr 1 22:43:14 2011 ;; MSG SIZE rcvd: 556 ravi-mozilla-mbp:~ ravi$ dig +dnssec www.mozilla.org @75.75.76.76 ; <<>> DiG 9.6.0-APPLE-P2 <<>> +dnssec www.mozilla.org @75.75.76.76 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16399 ;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 4000 ;; QUESTION SECTION: ;www.mozilla.org. IN A ;; ANSWER SECTION: www.mozilla.org. 60 IN CNAME www-mozilla-org.geo.mozilla.com. www.mozilla.org. 60 IN RRSIG CNAME 7 3 60 20110429150652 20110330151615 17852 mozilla.org. n5mzCliw5bDwcuX64cbrCNGaSM5QPghAj0h3esHokNpDLZvNe7lBsbOT lnj9UhMQkYKzZq+Yvd3WI3tmzsEeGXCFq7VDMP7bvamY8mM/Xl1w4jVd oaLIUF698epJB8vQWrvIIAGsSvSZlUEpXYMAtDAfBOj7Wy0+1c6UrZ8i ze4= www.mozilla.org. 60 IN RRSIG CNAME 7 3 60 20110429151032 20110330151615 62897 mozilla.org. x4Q1TwbPWsj63LFOt4VYxQzVPWkIx4mYrzbhHrYjA6pXLFh9UXEfNS7z kKKxmMKYhbWwXYBtwRMThd75WCraLf9ClV/at4k8pL/LyMZpeB6iP91d 8TjGM1gfOSc/6oMhxr9T3LE0QRyRp6Paqg3TVhoK90SEm6NYqWSX5ORU E8A= www-mozilla-org.geo.mozilla.com. 2557 IN CNAME www-mozilla-org.glb.mozilla.net. www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.217.21 www-mozilla-org.glb.mozilla.net. 30 IN A 63.245.217.21 ;; Query time: 41 msec ;; SERVER: 75.75.76.76#53(75.75.76.76) ;; WHEN: Fri Apr 1 22:43:28 2011 ;; MSG SIZE rcvd: 556
Same result as Ravi, and unchanged for irc.mozilla w/ the dnssec flag. Billy
Billy, can you check for irc.mozilla.org? I overlooked that you only did www in your update. ravi-mozilla-mbp:~ ravi$ dig +dnssec irc.mozilla.org @75.75.76.76 ; <<>> DiG 9.6.0-APPLE-P2 <<>> +dnssec irc.mozilla.org @75.75.76.76 ;; global options: +cmd ;; connection timed out; no servers could be reached ravi-mozilla-mbp:~ ravi$ dig +dnssec irc.mozilla.org @75.75.75.75 ;; reply from unexpected source: 68.87.76.185#53, expected 75.75.75.75#53 ; <<>> DiG 9.6.0-APPLE-P2 <<>> +dnssec irc.mozilla.org @75.75.75.75 ;; global options: +cmd ;; connection timed out; no servers could be reached
AirMax:Sites maxinux$ dig irc.mozilla.org @75.75.75.75 +dnssec ; <<>> DiG 9.6.0-APPLE-P2 <<>> irc.mozilla.org @75.75.75.75 +dnssec ;; global options: +cmd ;; connection timed out; no servers could be reached AirMax:Sites maxinux$ dig irc.mozilla.org @75.75.76.76 +dnssec ; <<>> DiG 9.6.0-APPLE-P2 <<>> irc.mozilla.org @75.75.76.76 +dnssec ;; global options: +cmd ;; connection timed out; no servers could be reached
In Milpitas, the name servers handed to me from Comcast are 68.87.76.182 and 68.87.78.134. Looks like the 75.75.75.75 and .76 nameservers could be to blame? jabba@JabbaBook ~> dig +dnssec irc.mozilla.org @75.75.75.75 ; <<>> DiG 9.6.0-APPLE-P2 <<>> +dnssec irc.mozilla.org @75.75.75.75 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 6711 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 4000 ;; QUESTION SECTION: ;irc.mozilla.org. IN A ;; Query time: 3648 msec ;; SERVER: 75.75.75.75#53(75.75.75.75) ;; WHEN: Sat Apr 2 06:03:19 2011 ;; MSG SIZE rcvd: 44 jabba@JabbaBook ~> dig +dnssec irc.mozilla.org @68.87.76.182 ; <<>> DiG 9.6.0-APPLE-P2 <<>> +dnssec irc.mozilla.org @68.87.76.182 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15140 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 4000 ;; QUESTION SECTION: ;irc.mozilla.org. IN A ;; ANSWER SECTION: irc.mozilla.org. 60 IN A 63.245.208.159 ;; Query time: 28 msec ;; SERVER: 68.87.76.182#53(68.87.76.182) ;; WHEN: Sat Apr 2 06:03:57 2011 ;; MSG SIZE rcvd: 60 jabba@JabbaBook ~> dig +dnssec irc.mozilla.org @68.87.78.134 10 ↵ ; <<>> DiG 9.6.0-APPLE-P2 <<>> +dnssec irc.mozilla.org @68.87.78.134 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34425 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 4000 ;; QUESTION SECTION: ;irc.mozilla.org. IN A ;; ANSWER SECTION: irc.mozilla.org. 60 IN A 63.245.208.159 ;; Query time: 98 msec ;; SERVER: 68.87.78.134#53(68.87.78.134) ;; WHEN: Sat Apr 2 06:04:15 2011 ;; MSG SIZE rcvd: 60 jabba@JabbaBook ~>
Yeah. only 75.75.75.75 and 75.75.76.76 are DNSSEC enabled, which means jabba...your +dnssec for your nameservers do nothing extra... I'd be happy to email the person I know at Comcast when I get back and ask them to take a look. Unfortunately, my business card cache is back home, so this will have to wait till the 12th.
Assignee: server-ops → shyam
Summary: Comcast DNS not resolving irc.mozilla.org → Comcast DNSSEC nameservers not resolving irc.mozilla.org
Comcast says they have completed the DNSSEC rollout to all name servers. Dns.comcast.net: We are now migrating all customers to DNSSEC-validating servers. This will happen automatically via DHCP updates between October 2010 and March 2011. Learn more at our DNSSEC Information Center or by watching this short video. Somehow I do not think that is to blame; I am a business customer, I will give support a call and ask them to start looking into it as well.
Comcast Business support ticket opened, waiting for T2 response via phone. My ticket number : CR241337101; but the tech I was speaking to was unable to load this ticket (only allowed to visit comcast sites supposedly). I recommend people tweet this to @ComcastCares also
Interesting, per http://dns.comcast.net they have a cached entry for irc.mozilla.org but are not returning it? Comcast DNS Cache Query This feature allows you to query Comcast's caching servers to validate DNS data. 75.75.75.75 (75.75.75.75) 63.245.208.159
Great, ldap.mozilla.org is affected by this as well.
Not having access to the zone to know entities I see one commonality between irc and ldap; neither rely on glb.mozilla.net ; so I suspect all records that return an A direct from mozilla.org will be affected.
A few days ago the #comodogate hacker posted the secret key for addon.mozilla.org's cert. If the mozilla.org cert was regenerated and if that's the cert used by dnssec, then your dnssec files would need to be re-signed with the new key, right? I've been using dnssec at comcast without incident since I signed up last September so the fact that it stopped working now makes me suspect this is related to the addon cert issue.
Except that addon.mozilla.org (and addons.mozilla.org for good measure) works: AirMax:Sites maxinux$ dig addon.mozilla.org @75.75.75.75 +short redirect.glb.mozilla.com. 63.245.209.24 AirMax:Sites maxinux$ dig addons.mozilla.org @75.75.75.75 +short amo.glb.mozilla.net. 63.245.217.40 63.245.217.40 63.245.217.40
oh look, irc is working now. yay!
Confirmed: AirMax:Sites maxinux$ dig ldap.mozilla.org @75.75.75.75 +short pm-ns01.mozilla.org. 63.245.208.172 AirMax:Sites maxinux$ dig irc.mozilla.org @75.75.75.75 +short 63.245.208.159
The key that the Comodogate attacker posted was the bogus one that the attacker created. None of our real keys ever got compromised. (And we do use a different key for signing the DNSSEC zones). Probably related: https://www.isc.org/announcement/bind-9-dnssec-validation-fails-new-ds-record Notice the date given in that advisory for when servers would start failing if they didn't restart. March 31.
(In reply to comment #23) The cert in question was only for addons.mozilla.org and is different from our DNSSEC signing keys. The incident with Comodo was an isolated incident.
(In reply to comment #27) The DURZ for COM finishing on 3/31 would not affect ORG. No word back on my ticket or tweets to comcastcares and comcaststeve.
(In reply to comment #29) > The DURZ for COM finishing on 3/31 would not affect ORG. No word back on my > ticket or tweets to comcastcares and comcaststeve. Yeah, I wouldn't have thought so, either, but the timing is mighty coincidental.
I believe this to be resolved though I don't think we got a reason for cause. Marking as such.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
I'd still be happy to try and find out :)
I never got a call back from Comcast, you could call and refer to my ticket for info perhaps
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.