770275 - Possible DNSSEC issue with wildcard domains like *.bugzilla.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations :: Infrastructure: Other, task)

Description

[(dormant account)]

Attachment: screenshot

I've been suffering from this for 3 weeks on multiple browsers, operating systems, networks on my laptop. If I try again, occasionally the patches show. Frequency of failure goes from rare to failing 90% of the time. Not sure what affects it

Comment 1

[Shyam Mani [:fox2mike]]

What are your DNS resolvers? Which OS is this?

Comment 2

[Shyam Mani [:fox2mike]]

Assuming the bugzilla field above is correct, Mac...I'll need to know what your DNS resolvers are to dig deeper.

Comment 3

[:glob ✱]

(In reply to Shyam Mani [:fox2mike] from comment #2)
> Assuming the bugzilla field above is correct

fwiw works for me.
the frame's url is https://bug769191.bugzilla.mozilla.org/attachment.cgi?id=637423

Comment 5

[(dormant account)]

(In reply to Shyam Mani [:fox2mike] from comment #1)
> What are your DNS resolvers? Which OS is this?

This happens on mac and windows 7. What's a dns resolver?

Comment 6

[(dormant account)]

Byron this does not seem to be related to my specific internet provider. I was traveling and was getting the exact same problem on hotel and conference internet. I'm using comcast at home.

Comment 7

[:Gavin Sharp [email: [e-mail]]]

Problems three weeks ago could have been bug 765670. There were definitely intermittent issues with bugzilla DNS before then. Since then I haven't heard any reports aside from the ones here, though.

That it's affecting more than just Taras suggests it could be an intermittent/isolated issue with Dynect or something?

Comment 9

[Shyam Mani [:fox2mike]]

(In reply to Taras Glek (:taras) from comment #5)
> (In reply to Shyam Mani [:fox2mike] from comment #1)
> > What are your DNS resolvers? Which OS is this?
> 
> This happens on mac and windows 7. What's a dns resolver?

Open a terminal on mac and run cat /etc/resolv.conf and paste the output here?

Comment 10

[(dormant account)]

My LAN uses a local dnsmasq resolver that forwards to 75.75.75.75, 75.75.76.76(ie comcast)

Comment 11

[Shyam Mani [:fox2mike]]

Dude, this is awesome information. Epic stuff :)

I'll look into this asap, I'm on pto most of this week, but this is serious enough for me to debug and fix. CC'ing some people for information.

The reason I'm so excited about this, is that's comcast's DNSSEC enabled DNS server..and this is an issue on our end with DNSSEC and wildcard domains.

Comment 12

[Shyam Mani [:fox2mike]]

Another thing that I just thought of...this may be happening because of our delegation of these domains to dynect. I'll give it some more thought.

Comment 13

[Corey Shields [:cshields]]

(In reply to Shyam Mani [:fox2mike] from comment #12)
> Another thing that I just thought of...this may be happening because of our
> delegation of these domains to dynect. I'll give it some more thought.

https://bugzilla.mozilla.org/show_bug.cgi?id=771861#c6 indicates this was an end-user problem?

Comment 14

[Mark Finkle (:mfinkle) (use needinfo?)]

My comcast dns nameservers are also: 75.75.76.76, 75.75.75.75

I am also getting a "server not found" when trying to view any attachment.

Comment 15

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I've also had some issues with viewing attachments since moving to San Jose a few weeks ago and switching to Comcast's internet service.

My nameservers are also 75.75.75.75 and 75.75.76.76

Comment 16

[rfw2nd]

I'm on Comcast at home.  My nameservers are 75.75.75.75, 75.75.76.76

While I get the "server not found" message on my PC, when I access it through my phone (which is on 4G), I am able to successfully view the attachment.

Comment 17

[:glob ✱]

you can use http://dns.comcast.net/dig-tool.php to query comcast's dns.

using a bugzilla wildcard hostname results in failure most of the time (it worked once out of twelve attempts for me).

the error message when it fails is "Invalid request sent.".

the success response returned:

wild.bugzilla.dynect.mozilla.net.
bugzilla-wild.zlb.phx.mozilla.net.
63.245.217.61

resolving wild.bugzilla.dynect.mozilla.net always works.


testing with other domains that appear to use dns wildcards (such as *.blogspot.com) doesn't show result in failure.

Comment 18

[Jared Wein [:jaws] (please needinfo? me)]

H

Comment 19

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Byron Jones ‹:glob› from comment #17)
> you can use http://dns.comcast.net/dig-tool.php to query comcast's dns.
> 
> using a bugzilla wildcard hostname results in failure most of the time (it
> worked once out of twelve attempts for me).

Anyway to get this to work? I am having a tough time using bugzilla with this happening.

> resolving wild.bugzilla.dynect.mozilla.net always works.

This still fails for me:
https://bug773535.bugzilla.dynect.mozilla.net/attachment.cgi?id=641720

Comment 20

[:glob ✱]

(In reply to Mark Finkle (:mfinkle) from comment #19)
> Anyway to get this to work? I am having a tough time using bugzilla with
> this happening.

the only work around i'm aware of is using different dns servers.

Comment 21

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Byron Jones ‹:glob› from comment #20)
> (In reply to Mark Finkle (:mfinkle) from comment #19)
> > Anyway to get this to work? I am having a tough time using bugzilla with
> > this happening.
> 
> the only work around i'm aware of is using different dns servers.

Yep and I switch to Google's DNS servers. Things are working now.

Comment 22

[Shyam Mani [:fox2mike]]

I'm back and will need sometime to confirm that the issue isn't on our end. I'll keep the bug updated.

Comment 23

[Jake Maul [:jakem]]

Just for the record, this also seems to happen with *.etherpad.mozilla.org (had a report of this today, and confirmed it with the tool in comment 17). This is different in that it does not go through mozilla.net or Dynect or 3crowd... it is a simple wildcard A record in mozilla.org.

Comment 24

[Shyam Mani [:fox2mike]]

This seems to be an issue on Comcast's end. We're working with them to see if we can pinpoint and resolve the issue.

Comment 25

[Jared Wein [:jaws] (please needinfo? me)]

And also for *.pastebin.mozilla.org

Comment 26

[Jesse Ruderman]

I tried switching to Google's DNS.  That fixed Bugzilla attachments, but broke resolving pvtbuilds2.dmz.scl3.mozilla.com when connected to MPT VPN with Tunnelblick.

Comment 27

[Shyam Mani [:fox2mike]]

To elaborate some more, the issue is with Comcast's DNS servers and the way they treat NSEC3 responses. I'm still waiting for a response from them.

Comment 28

[Shyam Mani [:fox2mike]]

Comcast says this is resolved now. If someone on Comcast can check and verify that the following query returns an IP :

dig +dnssec @75.75.75.75 bug770275.bugzilla.mozilla.org

We're good to go.

Comment 29

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

$ dig +dnssec @75.75.75.75 bug770275.bugzilla.mozilla.org

; <<>> DiG 9.8.1-P1 <<>> +dnssec @75.75.75.75 bug770275.bugzilla.mozilla.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54709
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4000
;; QUESTION SECTION:
;bug770275.bugzilla.mozilla.org.	IN	A

;; ANSWER SECTION:
bug770275.bugzilla.mozilla.org.	25 IN	CNAME	wild.bugzilla.dynect.mozilla.net.
bug770275.bugzilla.mozilla.org.	25 IN	RRSIG	CNAME 7 3 60 20120816195310 20120717200111 63920 mozilla.org. M5CU+laA/rD90Z/07LiM14WKf6owxX4ZKg/hmDjqjJxSxAe7MaVc4PVi l+fR4O/GLCkkPWKN6gH+7i653EJTAMmavn+MYm9rjlPtvWZ93LIBMXpO mdcD8ymfINu1v2gKkQMd0RWGJeY5IRqsoB031rgI3rpcMUptMVD8K1P4 kV0=
wild.bugzilla.dynect.mozilla.net. 900 IN CNAME	bugzilla-wild.zlb.phx.mozilla.net.
bugzilla-wild.zlb.phx.mozilla.net. 300 IN A	63.245.217.61

;; Query time: 85 msec
;; SERVER: 75.75.75.75#53(75.75.75.75)
;; WHEN: Sat Jul 21 15:25:15 2012
;; MSG SIZE  rcvd: 328