30917 - implement DNS caching and request cancelation

Status: VERIFIED FIXED

(Core :: Networking, defect, P2)

Description

[Warren Harris]

We need to implement DNS caching and request cancelation. The code for this 
exists on the DNS_CANCEL_BRANCH which Gordon will land after beta.

Comment 1

[Warren Harris]

=> M15

Comment 2

[Warren Harris]

Moving what's not done for M15 to M16.

Comment 3

[leger]

Putting on [nsbeta2+][5/16] radar.  This is a feature MUST complete work by 
05/16 or we may pull this feature for PR2.

Comment 4

[gordon]

The DNS_CANCEL_BRANCH has been landed (for awhile now).  We just need to tune the 
cache size.

Comment 5

[leger]

Is this fixed now?  tever what are the latest test results.

Comment 6

[gordon]

The nsbeta2 work is complete.  The bug is being left open as a reminder that we 
need to do performance tuning.

Comment 7

[leger]

Putting on [nsbeta3+] radar for PR3 work.

Comment 8

[Mike]

M16 has been out for a while now, these bugs target milestones need to be 
updated.

Comment 9

[leger]

Removing [nsbeta3+], will re-eval later for beta3.

Comment 10

[Gagan]

2 separate issues, but neither for beta3.

Comment 11

[Warren Harris]

This bug was fixed months ago, wasn't it? If there's performance tuning that 
can be done here, please file a separate bug.

Comment 12

[Alex Bischoff]

It's been a while since any activity on this bug -- has DNS caching been
implemented?

Comment 13

[gordon]

I'll take a look at this in the .9 or .91 timeframe.  It should be a relatively 
small change.

Comment 14

[gordon]

*** Bug 76610 has been marked as a duplicate of this bug. ***

Comment 15

[Gagan]

*** Bug 74153 has been marked as a duplicate of this bug. ***

Comment 16

[Phil Peterson]

Just to recap what we discussed in the performance meeting, we talked about
having a JS pref (not user visible) which controls the lifetime of a DNS cache
entry. Since 4.x cached DNS entries for 30 minutes, and lots of people hated
that, the pref will have a short default value, like 5 or 10 minutes. 

Customers who use proxy servers may want to jack up this value in their own
builds, since the proxy server is the only address they ever resolve.

Comment 17

[Bob Lord]

Does this lifetime refer to the amount of time the entry lives in the cache, or
the amount of time since it was last used?

This matter seems to hit me a little harder at home than at work since I lost my
DSL connection and spend several hours per night dialed-up via a modem.  It's a
little frustrating when I click a link on a web page which takes me to another
page on the same site, and I'm blocked for a couple of seconds waiting for the
DNS resolution to take place. 
   

Comment 18

[gordon]

I think the expiration time needs to be the life of the DNS entry in the cache, 
not an offset from the last use.  Otherwise, a DNS entry could last indefinitely, 
which is exactly what the folks who run servers have complained about.  We will 
use a preference however, so embedders can arbitrarily crank up the time as high 
as they want to optimize for their special uses.  My recollection from the 
newsgroup debates we had a couple years ago is that keeping DNS entries around 
for about 5 minutes wouldn't aggravate them too much, but might drastically 
reduce the latency for typical users.

Comment 19

[[e-mail]]

qa to me.

Any reason we wouldn't just use the TTL value? Then the hostmaster can handle 
the details (proxy server records could have a long ttl so they stick longer, 
etc.)

Comment 20

[benc]

(qa to me should have been done as [e-mail]...)

Comment 21

[Dan Mosedale (:dmosedale, :dmose)]

Many platforms provide no API at all to getting the TTL information.  We'd have
to write our own XP resolver with such an API or borrow one from somewhere. 
This is perhaps the right thing to do, though, but a bunch more work.

Comment 22

[[e-mail]]

Yuk. Nevermind what I said above.

Comment 23

[Matthias Versen [:Matti]]

*** Bug 75134 has been marked as a duplicate of this bug. ***

Comment 24

[R.K.Aa.]

*** Bug 81469 has been marked as a duplicate of this bug. ***

Comment 25

[benc]

+mostfreq, so we can catch dupes faster

Comment 26

[Marek Z. Jeziorek]

per PDT triage moving to 0.9.2

Comment 27

[gordon]

Attachment: Patch that works with Mac. Needs building and testing on Unix and Windows.

Comment 28

[gordon]

I have nsDnsService.[h|cpp] on a branch named DNS_BRANCH, and I've fixed some 
build errors on Linux and Windows.

Linux and Windows still need a bit of debugging.

Comment 29

[gordon]

Bug fixes have been checked into the DNS_BRANCH, and Linux and Windows seems to 
work.  If anybody would like to build and test this change, I'd be very 
interested in the results.

Comment 30

[Gagan]

gordon please attach the diffs (from the trunk) here... a lot of reviewers don't
always have access to the branch. Thanks.

Comment 31

[gordon]

Attachment: Linux and Windows now work. Test feedback welcome.

Comment 32

[Tomi Leppikangas]

I tested with patch on linux. Looked dns trafic with tcpdump and seems
that names only resolved once.

One problem with patch was that mozilla sometimes hangs on shutdown, but
that seems to be fixed on branch.

Comment 33

[gordon]

Attachment: Thanks for testing. Here's the latest patch from the DNS_BRANCH.

Comment 34

[Deven Corzine]

I'm seeing horrible DNS resolving problems in Mozilla, and it's starting to make
the browser unusable for the first time since I switched to Mozilla as my
primary browser in November 2000.  It's not the DNS system, because Netscape 4
will load pages right away while Mozilla is stuck resolving, or worse, claims
that a hostname is unknown.  I have seen this happen: (1) try to load a URL in
Mozilla -- error occurs that the hostname is unknown, (2) load the same URL in
Netscape 4 -- comes right up, fast, no problem, (3) try to load the same URL
again in Mozilla -- same error occurs.

I am also frequently getting error messages claiming unknown hosts like
"ad.doubleclick.net" and other advertising servers.  While I don't mind missing
banner ads, I do mind having to dismiss the dialogs (sometimes many on a single
page, actually), and again Netscape 4 can always resolve those hostnames just
fine.  I don't know what's going on here, but the DNS performance and
reliability is absolutely pathetic.

This branch code, is it an XP resolver that will honor TTL values or is it just
an internal cache for results from the system resolver?  (It seems like both are
needed, ideally...)

If an XP resolver is needed, I'd be interested in working on that if I could
find the time.  (That's a VERY big "if".)  However, I have no idea how to
approach the problem, being mostly unfamiliar with Mozilla internals.  I know
I've got the skills that I could implement a resolver, but I have no clue how it
would be properly integrated with Mozilla.  I would be inclined to make it run
in a single thread, processing requests asynchronously, though perhaps with a
synchronous API call available for other threads to block on a result.

If this is an area that hasn't been addressed, could someone give me some
pointers on how to proceed, and how the current code works?  I'm really NOT
making any promises here, but if nobody's working on it, it wouldn't hurt for me
to take a stab at it...

Comment 35

[Darin Fisher]

sr=darin for the current DNS_BRANCH:

===================================================================
File: nsDnsService.h   	Status: Up-to-date

   Working revision:	1.27.28.4
   Repository revision:	1.27.28.4	/cvsroot/mozilla/netwerk/dns/src/nsDnsService.h,v
   Sticky Tag:		DNS_BRANCH (branch: 1.27.28)
   Sticky Date:		(none)
   Sticky Options:	(none)

===================================================================
File: nsDnsService.cpp 	Status: Up-to-date

   Working revision:	1.79.24.12
   Repository revision:	1.79.24.12	/cvsroot/mozilla/netwerk/dns/src/nsDnsService.cpp,v
   Sticky Tag:		DNS_BRANCH (branch: 1.79.24)
   Sticky Date:		(none)
   Sticky Options:	(none)

Comment 36

[Asa Dotzler [:asa]]

a= [e-mail] for checkin to the trunk.
(on behalf of drivers)

Comment 37

[Darin Fisher]

stop the presses!!!

my optimized winnt build is hanging at startup with this patch.  i see the
ui thread getting stuck here (shown running TestHttp):

NTDLL! 77f6829b()
KERNEL32! 77f04f41()
NSPR4! 3001a669()
NSPR4! 3001a865()
NECKO! 01794af2()
NECKO! 01794dc8()
XPCOM! 10039fa8()
XPCOM! 1003cef4()
XPCOM! 10043ed3()
XPCOM! 1003fb88()
XPCOM! 10040143()
NECKO! 017847f7()
NECKO! 01784f00()
XPCOM! 10039fa8()
XPCOM! 1003cef4()
XPCOM! 10043ed3()
XPCOM! 1003fb88()
XPCOM! 10040143()
XPC3250! 01c20bfa()
XPCOM! 10050204()
XPC3250! 01c2d851()
XPC3250! 01c33b3d()
JS3250! 00c883d2()
JS3250! 00c9351e()
JS3250! 00c88b68()
JS3250! 00c6659c()
JSLOADER! 01473b2f()
JSLOADER! 014731f5()
JSLOADER! 01472c3d()
JSLOADER! 014729c9()
JSLOADER! 01472412()
JSLOADER! 01472293()
XPCOM! 1003ebcd()
XPCOM! 10013d04()
PLDS4! 00231d42()
XPCOM! 10013ccd()
XPCOM! 10004067()
XPCOM! 1003ea74()
XPCOM! 1003e3c4()
XPCOM! 100441ab()
TESTHTTP! 0040176e()
TESTHTTP! 004017bc()
TESTHTTP! 00402693()
KERNEL32! 77f1ba06()

i suspect that this is the DNS service getting hung up.

gordon: i also needed to add the following patch to nsDnsService.cpp to get
it to compile on win32:

Index: nsDnsService.cpp
===================================================================
RCS file: /cvsroot/mozilla/netwerk/dns/src/nsDnsService.cpp,v
retrieving revision 1.79.24.12
retrieving revision 1.79.24.13
diff -u -r1.79.24.12 -r1.79.24.13
--- nsDnsService.cpp	2001/06/06 21:49:51	1.79.24.12
+++ nsDnsService.cpp	2001/06/06 23:44:44	1.79.24.13
@@ -1279,7 +1279,7 @@
         DispatchMessage(&msg);
     }
 
-    return rv;
+    return NS_OK;
 }
 
 #endif

i've gone ahead and checked in this patch on the DNS_BRANCH.

Comment 38

[Darin Fisher]

problems solved.

Comment 39

[Mike Kaply [:mkaply]]

You guys suck. Big time.

This code that was checked in on the trunk doesn't even compile on 
BeOS and OS/2.

You could have at least let someone know that this was coming in.

Comment 40

[gordon]

Mike, your fix for OS/2 and BEOS should allow the DNS service to work, but we 
still need to add code to evict lookups after they've been stored in the dns 
cache.  I've create bug 84420 to track that, which I can fix tomorrow after 
getting the necessary reviews.

Marking this bug FIXED.

Comment 41

[LinuxLover]

Using nightly build 08 Jun 2001.  Works like charm. 

But for google (http://www.google.com) it resolves again and again!  Why?  b/c
google has some sort of round robin dns?  

A linke 'host -a www.google.com' results
Trying "www.google.com."
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27022
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 4

;; QUESTION SECTION:
;www.google.com.                        IN      ANY

;; ANSWER SECTION:
www.google.com.         226     IN      A       216.239.37.100

;; AUTHORITY SECTION:
GOOGLE.COM.             159117  IN      NS      NS2.GOOGLE.COM.
GOOGLE.COM.             159117  IN      NS      NS1.GOOGLE.COM.
GOOGLE.COM.             159117  IN      NS      NS3.GOOGLE.COM.
GOOGLE.COM.             159117  IN      NS      NS4.GOOGLE.COM.

;; ADDITIONAL SECTION:
NS2.GOOGLE.COM.         65882   IN      A       216.239.34.10
NS1.GOOGLE.COM.         35890   IN      A       216.239.32.10
NS3.GOOGLE.COM.         45104   IN      A       216.239.36.10
NS4.GOOGLE.COM.         28997   IN      A       216.239.38.10

Received 194 bytes from 216.128.2.250#53 in 757 ms


any ideas?

Comment 42

[benc]

LinuxLover:

If this happens on Mozilla 0.9.1 or Netscape 6.1b1, please open a new bug and
we'll go there.

I'll be marking this verified once I get done verifying through the backlog and
write a good test case.

Comment 43

[gordon]

Ben:  we didn't have DNS caching in 0.9.1 or the beta.

Comment 44

[benc]

gordon: whatever :) The main point was to send the google problem to another
bug, unless you think it should stay here. I want to avoid a lot of bug morphing.

Comment 45

[Sivakiran Tummala]

 looked at the tcpdump i see the dns request send every time

Comment 46

[gordon]

What platform are you testing?

Comment 47

[Austin Donnelly]

Attachment: tcpdump of over-zealous DNS activity

Comment 48

[Austin Donnelly]

My previous comment attachment shows much superfluous DNS activity,
in this case merely caused by loading http://news.bbc.co.uk/
hornet.cl.cam.ac.uk is the client machine running Mozilla beta 0.9.3
             (Build ID 2001080104)
wwwcache.cam.ac.uk is my local web cache
resolv0.cl.cam.ac.uk is my local DNS server
We badly need DNS caching.

Austin

Comment 49

[Austin Donnelly]

This bug only seems to manifest itself when using a web cache.
A friend running 0.9.3 with Direct Connection to Internet selected in
the Advanced/Proxies preferences page doesn't see this behaviour.
I do, and I have "Automatic proxy configuration" selected, set to
  http://www.cl.cam.ac.uk/proxy.configThis config file reads as follows:
function FindProxyForURL(url, host)
{
   if(  shExpMatch( url, "*cgi*" ) ||
        shExpMatch( url, "snews:*" ) ||
        shExpMatch( url, "https:*" ) ||
        dnsDomainIs( host, "pgp.net") ||
        dnsDomainIs( host, "cam") ||
        dnsDomainIs( host, "ac") ||
        dnsDomainIs( host, "ac.uk" ) ||
        dnsDomainIs( host, "ja.net" ) ||
        dnsDomainIs( host, "www.avantek.co.uk" ) ||
        dnsDomainIs( host, "www.elsevier.nl" ) ||
        dnsDomainIs( host, "localhost" ) ||
        isPlainHostName( host ) ||
        isInNet( host, "131.111.0.0", "255.255.0.0") ||
        isInNet( host, "128.232.0.0", "255.255.0.0") ||
        isInNet( host, "129.169.0.0", "255.255.0.0") ||
        isInNet( host, "192.168.100.1", "255.255.255.0") ||
        isInNet( host, "127.0.0.1",   "255.255.255.255") )
        return "DIRECT";
   else return  "PROXY wwwcache.cam.ac.uk:8080; " +
                "DIRECT";
}

Comment 50

[gordon]

Okay, could you create a new bug describing the problem you're having with auto
proxy config, and close this bug again.  Please include the platform you're
testing on.  That will make it easier for us to try duplicating your results.

Thanks.

Comment 51

[Austin Donnelly]

This bug can be closed.  I've opened a new bug report about the
interaction with automatic proxy config, Bug#94079.

Austin

Comment 52

[timeless]

restoring fixed per [e-mail] 2001-06-06 21:41

Comment 53

[Vladimir Ermakov]

verifying per coments