662077 - DNS timeouts resolving mozilla.org names using Time Warner Road Runner DNS servers

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Bill Gianopoulos [:WG9s]]

DNS queries for the mozilla.com zone using the Time Warner DNS servers for my ISP connection often take over 9 seconds to return a reply.  The DNS query response timeout for Windows is 8 seconds.  This results in a lot of DNS lookup failures.  I also see DNS timeouts under Linux.

The DNS servers I am using are:

209.18.47.61
209.18.47.62

I suspect this is a DNSSEC issue as the Sandia Labs DNSviz tool yields the following three DNSSEC errors:

RRSIG mozilla.org/A by mozilla.org/DNSKEY alg 7, key 63025: The RRSIG was made by a revoked key.

RRSIG mozilla.org/MX by mozilla.org/DNSKEY alg 7, key 63025: The RRSIG was made by a revoked key.

RRSIG mozilla.org/SOA by mozilla.org/DNSKEY alg 7, key 63025: The RRSIG was made by a revoked key.

Comment 1

[Bill Gianopoulos [:WG9s]]

Ooops-

Forgot to include the URL for the tool.

http://dnsviz.net/d/mozilla.org/dnssec/

Comment 2

[Bill Gianopoulos [:WG9s]]

Looking at the graphical output, rather than just the errors, it appears there are 2 RRSIGs for each of these one with a good key and one with the bad key, so perhaps just deleting the bad ones will make things work better.

Comment 3

[Bill Gianopoulos [:WG9s]]

Further testing makes me wonder if this really is DNSSEC related or not.  Using the same ISP connection, if, on my Linux system, I run a local DNS server everything resolves very quickly.  This is all using the stock DNS server that comes with fedora 14 which should support DNSSEC, so I am now confused.  The only real difference is that the Time Warner DNS servers are on the 209.18.47.0/24 subnet whereas my IP address is on the 74.65.155..0/24 subnet.  I have no idea what difference that should make here though.

Comment 4

[Bill Gianopoulos [:WG9s]]

Well you can change this to normal, but it prevents me from being able to provide my builds on weekends.

Comment 5

[Dumitru Gherman [:dumitru]]

Bill,
Can you try please other public name servers while we investigate the cause of the timeouts? Looks indeed that those two nameservers don't resolve properly, so I think it could be an issue on Time Warner's end. Also it would be interesting to see if other domains have this issue.
I have tried the Google name servers (8.8.8.8 and 8.8.4.4) and they resolve our domains with no problems.

Comment 6

[Ravi Pina [:ravi]]

I went ahead and removed the signing keys in question from the chain.

http://dnsviz.net/d/mozilla.org/dnssec/ last updated 2011-06-04 23:21:20 UTC no longer shows errors.

Comment 7

[Bill Gianopoulos [:WG9s]]

Thanks for the tip on using the Google public DNS servers.  I feel stupid for not knowing that those existed.  I reconfigured my router to return 8.8.8.8 and 8.8.4.4 as the dns servers in response to dhcp requests and that has fixed my issue form my summer weekend home, which also fixes the fact that I objected to the severity being reduced.  My main reason for submitting it for major, was that if was actually actively working on fixing a bug (as I sometimes do) then the if I had no real workaround, blocker would have been my idea of the severity.

Your removal of the bad signing keys did not alter the symptoms at all, so I guess this never was a DNSSEC issue.

However, there is still an issue that is not just a Time Warner issue.

Is it OK if I email you some further data relating to this issue that has to do with my work network (which I am not authorized to post in a publicly viewable bug).

Comment 8

[Ravi Pina [:ravi]]

Feel free to mail [e-mail].

Comment 9

[Shyam Mani [:fox2mike]]

(In reply to comment #6)
> I went ahead and removed the signing keys in question from the chain.
> 
> http://dnsviz.net/d/mozilla.org/dnssec/ last updated 2011-06-04 23:21:20 UTC
> no longer shows errors.

Sigh. I was testing key revocation and those keys were supposed to be gone in a day. DNSSEC was pointing out that it was signed with a revoked key.

Comment 10

[Shyam Mani [:fox2mike]]

(In reply to comment #2)
> Looking at the graphical output, rather than just the errors, it appears
> there are 2 RRSIGs for each of these one with a good key and one with the
> bad key, so perhaps just deleting the bad ones will make things work better.

If you had looked at that output carefully and bothered to read the legend,  you'd have seen those red circles meant that key had the revocation bits set, which was totally intentional in this case. Also, had you bothered to check with http://dnssec-debugger.verisignlabs.com/ you'd have seen that there was no issue with dnssec verification itself.

Comment 11

[Shyam Mani [:fox2mike]]

Also, unless 209.18.47.61 and 62 are DNSSEC validating resolvers (and if they were, you would have gotten NO response for mozilla.org IF it was actually failing validation) there is no way to say that dnssec was causing your DNS issues.

(In reply to comment #7)
> Is it OK if I email you some further data relating to this issue that has to
> do with my work network (which I am not authorized to post in a publicly
> viewable bug).

You could make the bug not public (or we could, if you so request it).

Comment 12

[Bill Gianopoulos [:WG9s]]

(In reply to comment #11)
> Also, unless 209.18.47.61 and 62 are DNSSEC validating resolvers (and if
> they were, you would have gotten NO response for mozilla.org IF it was
> actually failing validation) there is no way to say that dnssec was causing
> your DNS issues.
> 
> (In reply to comment #7)
> > Is it OK if I email you some further data relating to this issue that has to
> > do with my work network (which I am not authorized to post in a publicly
> > viewable bug).
> 
> You could make the bug not public (or we could, if you so request it).

I am sorry, I thought I had already mentioned this here.

I filed bug 662289 on the issue I was seeing from work, which I eventually realized was DNSSEC related sort of.  The DNS relies for a DNSSEC signed zone are longer and the Cisco firewall maximum message length in the inspect rule needed to be raised.

I suspect the Time Warner issue is similar since the symptoms were identical.  I will try to contact the appropriate people at Time Warner/Road Runner to report thei issue and solution.