Last Comment Bug 417689 - firefox 3b3: very slow dns lookups
: firefox 3b3: very slow dns lookups
Status: RESOLVED INCOMPLETE
: perf
Product: Core
Classification: Components
Component: Networking (show other bugs)
: Trunk
: All Mac OS X
: -- major with 6 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-02-15 01:50 PST by Lucas Madar
Modified: 2016-02-01 14:32 PST (History)
22 users (show)
vseerror: needinfo? (david)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments

Description Lucas Madar 2008-02-15 01:50:48 PST
User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3

3b3 spends a lot longer 'looking up' servers than 3b2 or Safari do. This is not a local DNS issue; it occurs even when running a caching DNS server and using only cached domains.

This makes sites that load data from a lot of different places almost unusable. 10 seconds to load www.cnn.com vs 3 in Safari.

Reproducible: Always

Steps to Reproduce:
Load a website you haven't just visited or reload one you haven't refreshed in 3 minutes or so.



Marked this as a major bug because it is a major hit to usability.
Comment 1 Jo Hermans 2008-02-15 02:10:16 PST
Probably related to bug 408881 (re-enabling of IPv6 for Mac).

Can you please check this, by going to about:config, and set network.dns.disableIPv6 to true (create the entry if necessary) ? You'll probably have to restart Firefox too.
Comment 2 Lucas Madar 2008-02-15 21:23:39 PST
(In reply to comment #1)

> Can you please check this, by going to about:config, and set
> network.dns.disableIPv6 to true (create the entry if necessary) ? You'll
> probably have to restart Firefox too.

This appears to fix the problem. I do have ipv6 turned off for my main internet interface (configure ipv6 for tcp/ip under Airport prefs is set to 'off'). However, I have additional virtual interfaces (parallels and a vpn) that utilize IPv6.
Comment 3 Jo Hermans 2008-02-16 03:20:12 PST
I think that you're using a DNS server which is behind the VPN interface (your company ?), and the Mac OS X tried to to AAAA lookups first, because it knows it's an IPv6 capable interface. At least, that how I think the OS works, to provide adequate backwards compatibility. Then it fell back to A-lookups to the same DNS server, or to a DNS server on your main interface (the ISP ?).

Maybe you can mark that VPN interface as IPv4 only, if you think that the DNS server on that network is having problems with IPv6.
Comment 4 Florian von Kurnatowski 2008-05-26 12:51:22 PDT
same experience here - seen the same effect on two different networks (company and my home DSL), and disabling IPv6 resolved it.

I believe, however, that some fix should be done before release, as this was behaving without problems in FF2 in the same networks (makes sense because it probably didn't have the IPv6 DNS support) and I'd assume that it'll affect many users for whom this is beyond their capabilities. It will make the product look slow as compared to Safari on all Mac and possibly other systems.
Comment 5 Mike Beltzner [:beltzner, not reading bugmail] 2008-05-26 13:22:45 PDT
So, a few thoughts:

 - we'll definitely relnote this (need to be using ipv6 and unroutable)
 - we probably want to turn the pref to be off by default
 - we can do that in a stability and support release
 - we should monitor to see how widespread a problem this is
Comment 6 Alex Newman 2008-06-09 05:34:23 PDT
The problem also manisfests with FF3rc2 on my OS X 10.5.3 G5 PPC box: setting network.dns.disableIPv6 to true does make this version actually usable. Network (home) is via cable, but is using IPv4, not using IPv6.

FF3rc2 actually appears to be a bit faster on OS X 10.4.11 with network.dns.disableIPv6 set to "false" default.
Comment 7 David Luyer 2008-06-11 15:49:45 PDT
(In reply to comment #5)
>  - we probably want to turn the pref to be off by default

This would be unfortunate.  The pref was off by default on OS X (only)
in FF2, and the original reason for this is long resolved.  FF3 will be
out for a long time and IPv6 is only going to continue to gain in
prominence during that time (as the number of sites where multilevel NAT
breaks AJAX etc when attempting to use IPv4 will only increase).

The question we should be focussing on in this case is why FF3rc3 is slower
than Safari.  Safari does IPv6 lookups in the same scenarios that FF3rc3
does IPv6 lookups.  Why is Safari faster?  What is it doing better?  Is
it ordering its DNS queries better?

What is the exact URL typed for the poor behavior?

Is Firefox scanning a (long?) DNS searchlist with AAAA before falling back to A?
Comment 8 Jo Hermans 2008-06-11 16:12:39 PDT
According to <http://hunter.pairsite.com/blogs/blog20080603.html>, it might be fixed in 10.5.3. Or is that only for SRV lookups ?
Comment 9 Wayne Mery (:wsmwk, NI for questions) 2009-01-31 08:59:05 PST
do you still see this?
I thought I also read this is gone.
dupe of bug 417242?
Comment 10 David Favor 2010-01-02 09:25:35 PST
Problem still exists on Mac 10.6.2 + Firefox 3.5.6 with or without ipv6 off.
Comment 11 David Favor 2010-01-02 09:26:12 PST
This applies to all sites, even when using local caching DNS.
Comment 12 John 2012-10-18 13:51:27 PDT
I have a windows XP and I just installed ipv6. I started to experience very slow response time from website with firefox. 

I did a packet capture and there are many requests for AAAA for many ocsp and CRL verification. The problem is they don't answer to AAAA yet. I don't know why the firefox features for ocsp request any times something that don't exist...

I put a packet capture in <a href='http://www.heypasteit.com/clip/0J7W'>this paste</a>.

The sequence start at 20:01:47.950204 and it takes 20 seconds to finally find out there is nothing there...

For google it's faster but, still it freezes the browser for 5 seconds. I don't think that disabling ipv6 is a good solution as I want ipv6 and I will not switched it off. 

If I can help more in anyway I will very happy to help.

John@wedebugyou.com
<a href='http://www.wedebugyou.com'>www.wedebugyou.com</a>
Comment 13 Henrik Skupin (:whimboo) 2012-10-18 23:32:44 PDT
(In reply to John from comment #12)
> I have a windows XP and I just installed ipv6. I started to experience very
> slow response time from website with firefox. 
[..]
> 
> The sequence start at 20:01:47.950204 and it takes 20 seconds to finally
> find out there is nothing there...

Josh, do we have bugs around which cover improvements for DNS lookups via IPv6?
Comment 14 Josh Aas 2012-10-19 01:49:38 PDT
(In reply to Henrik Skupin (:whimboo) from comment #13)

> Josh, do we have bugs around which cover improvements for DNS lookups via
> IPv6?

Not that I know of. Maybe Steve knows.
Comment 15 Jo Hermans 2012-10-19 02:06:09 PDT
(In reply to John from comment #12)
> I have a windows XP and I just installed ipv6. I started to experience very
> slow response time from website with firefox. 


According to bug 388836 (is it still accurate ?), we're not using IPv6 for OCSP.
Comment 16 John 2012-10-19 07:38:55 PDT
Here is how to reproduce.

You need an OSX with ipv6 enabled. Also note that it's not platform dependant but it affects Mac, Windows and Linux.

By default ocsp is enabled after an install. So verify that it's enable. Then browse https://www.google.com . It will be long. Then go to https://www.hotmail.com and you will see that it's also long.

If you check in the pastebin I posted before, http://www.heypasteit.com/clip/0J7W 
you can check at frame 20:01:47.950204

It's querying ocsp for 20 seconds and the browser is frozen during that time.

If you go to about:config and change the value of network.dns.ipv4OnlyDomains and you put the values in this list http://www.wedebugyou.com/ipv4domains.txt

After that the browser is fast to browse google, hotmail, ebay, etc in https

Jean
Comment 17 Steve Workman [:sworkman] (INACTIVE) 2012-10-19 11:35:04 PDT
(In reply to Josh Aas (Mozilla Corporation) from comment #14)
> (In reply to Henrik Skupin (:whimboo) from comment #13)
> 
> > Josh, do we have bugs around which cover improvements for DNS lookups via
> > IPv6?
> 
> Not that I know of. Maybe Steve knows.

None of my cc'd or assigned DNS bugs are related directly to IPV6. So, I did a bugzilla search :)
Bug 680932 suggests that falling back to IPv4 is not working right.
Bug 697415 is to do with IPv6 failing using Teredo tunnels.
Bug 653953 relates to IPv6 failing with SSL.
...

John, from comment 16 it seems like you only see this with OCSP - is that correct, or do you have other IPv6 connection issues? Trying to determine if it's OCSP specific or a more general issue that you just happen to observe with OCSP.

In comment 12 you say Google is faster - do you mean Chrome or a google site? Trying to determine if it's FF or something on your network. Do you see it on multiple networks? You said you see it on multiple OSes, so it's probably not your machine.

From the log you refer to in comment 16, it looks like the DNS resolver tries AAAA queries first, then fails and and eventually falls back to IPv4. So, it may be related to bug 680932.
Comment 18 John 2012-10-20 01:01:46 PDT
Sorry I am not very familiar with submitting bugs.

I will try to submit more details. :)

I checked the Bug 680932 and Bug 653953 and I can't 100% confirm that what I experience are link but, I am positive that they might also be link.

So I don't think my network as connectivity or networking issue. I checked everything and with Internet explorer in my network everything is fine. (sorry to mention IE but, I tested with it to eliminate hypothesis.)

So to give more details about my network to better understand, I have a dual-stack network. I use ipv4 and ipv6 with a tunnel-broker.

I can confirm that when I browse a website with firefox that uses https then it's very slow and sometimes it feels like it just don't open.

For the rest of the example I will use https://www.hotmail.com as a testing case. I could chose also any other https website like https://www.google.com and they all suffer the same issue.

So I am not sure what do you mean with only OCSP specific website. Is an https website always using OCSP? So if yes, then yes I observed extreme slowness with all HTTPS website when I have ipv6 and ipv4 enabled.

In comment 12 I said google is faster I meant https://www.google.com but, let's continue with https://www.hotmail.com since the capture I included in comment 12 is about this https website.

Since, I don't have multiple ipv6 enabled networks it's hard for me to test on another one. I have a friend who have one too, I will check with him if he experiences the same issues. I can't guarantee that he will reply soon as he lives on another continent.

This behaviour with firefox in my network it happens with firefox 16.0.1 in Windows XP SP3 and I also tested with firefox 16.0 under Mac OS X 10.6.8

I don't have any linux with me at the moment but, if it's necessary in the next days I could install one and run the same tests.

Now, if I completely disable OCSP in the options, all the internet browsing is as fast as with ipv4. As a human I can't notice a difference. 

I don't think it's good to disable OCSP. Is it a problem?

So I reenabled OCSP and I tried to understand what is wrong and how to temporarily "patch" the bug.

Now here are the very details troubleshooting and "patch" I applied.

We need to refer to http://www.heypasteit.com/clip/0J7W that I pasted in comment 12. The first packet 20:01:35.243087 is my first dns queries to go to https://www.hotmail.com

At this moment, I see the browser doing stuffs, it starts to go to hotmail and it seems to load something...

Then at 20:01:47.950204 it does a ipv6 dns queris for AAAA record of evintl-ocsp.verisign.com

Then 1 second later it retries with another dns server, then it retries again 1 second later with my first dns server, then it retries I think 7 times, then it tries a second ocsp server that, I guess, is hard-coded somewhere in mozilla.

Finally, around 20 seconds later, it figures out that ocsp is not ipv6 reachable and then try over a ipv4 dns queries with a A record.

Then https://www.hotmail.com loads.

Now why is ocsp querying so many times? I don't know.

What I did to bring back normal speed to my firefox browser in Windows and MAC, is to force firefox to query over ipv4 some non-working ipv6 domain. 

- In firefox browse to the address about:config.
- Search for the key network.dns.ipv4OnlyDomains and double-click on it.
- Then you need to put some servers to force a dns query over ipv4. Put all the servers from this up to date list  http://www.wedebugyou.com/ipv4domains.txt .
- Completely close all the browsers and reopen a new one.

I will update the lists if I found more ocsp non working over ipv6.

As soon as I do that, firefox browsing with ipv6 for https website is fast again. 

Now it seems that Bug 680932 and Bug 653953 could potentially be link to this behaviour. Should I also post all of this troubleshooting there too?

Anyway I remain available for more questions or troubleshooting.

thanks

Jean
Comment 19 John 2012-10-20 01:42:16 PDT
Wow, now I remove all the "patch" I applied and I can't reproduce the problem.

Now it seems that the OCSP provider added a better ipv6 to ipv4 name resolution.

See this port 53 tcpdump now. http://www.heypasteit.com/clip/0JBO

Check line 10:37:03.946087 and then it returns a CNAME and a A record to ocsp.verisign.net., A 199.7.57.72

I am sure it was not doing that before. I tried many times and over 2 days.

Maybe it was a temporary issue with my provider. I am sorry for this confusion. I guess you can disregard all of my comments as I can't reproduce this issue.

Sorry again.
Comment 20 Steve Workman [:sworkman] (INACTIVE) 2012-10-22 13:16:34 PDT
Glad to hear it worked out for you :) And thanks for following up with the extra details.
Comment 21 Wayne Mery (:wsmwk, NI for questions) 2014-12-21 06:10:28 PST
(In reply to John from comment #19)
> I can't reproduce this issue.

Where does this leave the original bug report, in light of the fact that no other bug reports are explicitly marked as related?
Comment 22 Brian Vanderburg 2015-01-16 14:19:39 PST
I am also experiencing this bug.

System: Debian Wheezy 7.7 64 bit
CPU: 6 core 3.4 GHz
Memory: 12 GB
Version: 35 64 bit (from mozilla downloads)
DNS: Running a local caching DNS server.
 
I am also experiencing similar problems.

When running Firefox 64 bit, DNS lookups frequently take forever (5+ seconds) and so do some "connecting" stages (but sending,waiting,receiving is usually quick).  I've tested this out in an empty profile as well as my main profile.  I've done all the tricks I could find online, disable dns ipv6, dns prefecthing, increase DNS cache entry counts, etc. Nothing fixed the problem, except using the 32 bit version of Firefox instead.  When I use the exact same version of Firefox but 32 bit instead of 64 bit, under the exact same profile, it is lightning fast.  No waiting for "Looking up blah.com" or anything.  So, what is different in the 64 bit version than the 32 bit version that causes extremely slow lookups?
Comment 23 psnizek 2015-04-19 23:59:20 PDT
I am experiencing this slow down too.

System: Ubuntu 14.04
CPU: AMD Phenom II x4
Memory: 16GB
Version: 37.0.1 64-Bit (Ubuntu package)
DNS: Running a local caching DNS server
Network: NFS network, with /home folders (including browser cache) hosted on fileserver (1 Gigabit Ethernet)

Vivaldi and Chrome are both snappier than Firefox when browsing to various sites.
Firefox seems to comparably take an eternity when resolving a domain name ("waiting for xyz site").
Interestingly after a while (after trying connect to 5-10 sites) it seems to get better.
Then close all firefox instances, wait a few minutes and start firefox again. And again dns lookup is slow for the first 1 to 3 sites.
Chrome and Vivaldi have no such issues.
Comment 24 siruguri 2015-05-13 12:07:01 PDT
+1 - I noticed this problem the last few weeks, found this bug report, disabled ipv6 in about:config, and now things are okay again. So at least I found the earlier workaround to work, but I thought I'd +1 the bug.
Comment 25 fisons 2015-06-02 15:02:54 PDT
Same problem here.

System: Xubuntu 14.04 (64bit)
CPU: Intel i7-4770K
RAM: 8GB

Super slow dns lookups render firefox absolutely unusable for me (waiting +30s for a single page is a no-go).
Same page in current Chrome: 1s

Disabled ipv6 did not help.
Comment 26 psnizek 2015-07-23 06:25:57 PDT
Update:
I notice some significant improvement with Firefox 39 regarding DNS lookup. The issue seems to be  fixed, but slow page-loading in general is not entirely resolved yet. Some pages (usually sub-pages in my case) take a long time, with the wheel spinning counter-clock-wise (or flipping back and forth between clockwise and counter-clockwise). When a sublink is clicked before the page is entirely loaded, it appears to me Firefox is prioritizing to finish loading elements that are linked to the current page (i.e. advertisements or stats) from other, slow responding servers, instead of aborting the whole loading process and immediately start loading the sub-page. This can cause a delay of 10-30 or even more seconds as opposed to webkit or Blink based browsers which load the same subpage immediately.
Comment 27 psnizek 2015-08-03 00:23:25 PDT
Correction regarding my update from 2015-07-23:
There's no improvement unfortunately. The reason for a better performance was that my 2nd DNS server wasn't working correctly due to corrupted data. This is fixed now and firefox is again super slow as always.
Comment 28 psnizek 2015-08-09 09:14:59 PDT
the following workaround helped to improve the problem significantly:
1. open firefox
2. type in about:config in the address bar
3. right click on the properties and select New > Integer in the context menu
4. Enter network.dnsCacheExpiration as the preference name and 0 as the integer value

more details: http://ccm.net/faq/555-disabling-the-dns-cache-in-mozilla-firefox
Comment 29 Daniel Stenberg [:bagder] 2015-08-10 15:10:34 PDT
(In reply to psnizek from comment #28)
> the following workaround helped to improve the problem significantly:

1 - The fact that _disabling_ the DNS cache helps speed things up for you is certainly an indication that the actual DNS resolves are not at all slow since you then rather introduce more resolves and yet you say that runs faster for you.

2 - In the problematic case when you're using the DNS cache with its default settings (or have you altered it in any other way?), does the problem then appear with all sites or just specific ones? Are the problematic sites resolving to IPv4 or IPv6 addresses?

is this still version 37?
Comment 30 psnizek 2015-08-11 03:21:38 PDT
Hi Daniel,

1 - There is a significant improvement, even though sometimes I experience some waiting time of 10-20 seconds (connecting to server) on an initial access to a website. However, subsequent connections now usually take anything from 1-3 seconds. This is definitely an improvement that happened after I have disabled the DNS caching in firefox. All in all I still perceive firefox to be quite a bit slower than his peers. Yes, our DNS is OK in terms of speed.

2 - I am testing following URLs: spiegel.de, zeit.de, autobild.de, automotorundsport.de, heise.de. The built-in firefox dns caching hasn't been played with at all, with the exception that I switched it off recently.

3 - version 39.0.3 on Ubuntu 14.04.2 LTS it is.

I like to mention also, that my /home folders are on an NFS share (1gbits/s ethernet) and not on a local disk. Firefox is writing/reading all data (including cache) into/from a folder within this structure, which is probably not the best idea - but chrome and all his derivatives on the opposite side handle this perfectly well). I noticed that firefox particularly doesn't like this solution as it tends to grey out every once in a while (probably depending on network traffic or busy fileserver IO stack).

I hope I was able to provide some useful information.

Regards
Comment 31 Daniel Stenberg [:bagder] 2015-08-11 05:58:08 PDT
psnizek,

I don't doubt that you experience a performance problem, but your description really doesn't make it sound like it is due to "very slow dns lookups" and thus this is not the right bug entry for it. I would urge you find another one that more closely matches your symptoms or perhaps create a new one if there isn't one such already. Feel free to cc me on it.

Then, it would be really cool if you could run a single-shot test with an installation that isn't on the NFS share just to see if that changes anything so that we can rule out the risk that this little detail actually has anything to do with your problem?

I use Firefox on 64bit Debian (which should be quite similar to Ubuntu) all the time and I've done so for a long time through a range of Firefox versions - I've never experienced these problems you mention. I also retried both version 38 and Nightly 42 right now on several of the suggested domains you have problems with, without any noticeable degradation in speed.

Would it be possible to do a wireshark recording of what happens over the network when you visit one of these sites and get such a slow response? I assume you get these problems on ordinary HTTP sites as well, and then such a trace would be the most beneficial (to have us rule out TLS related things).

We would also greatly appreciate if you would consider doing a log with Firefox (https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging) to allow us to even closer see what timings and things that lead to this madness.
Comment 32 psnizek 2015-08-11 14:08:03 PDT
Hi Daniel,

I gave it a second thought after reading your message. Indeed you seem to be right.
I don't know if I really should create a "new bug" entry, because of I think Firefox works as designed - it scribbles everything to disk creating large numbers of disk I/O operations. So I don't view it as a bug in the classic sense. And I also agree this doesn't belong into this "Slow DNS"-related forum. I therefore apologize that I still do my final post in here.

It appears all is caused by the NFS3 share that is simply overwhelmed by the number of I/O operations coming from firefox (checked disk stack on the fs and the raid was constantly busy when using firefox as browser). Our raid has a (very!) weak random I/O performance, especially for small blocks. This would also explain, why firefox requires so much time to terminate itself when exiting the app. It's because of it is busy to finish up with queued disk I/O operations. 

A second indicator is that I failed to reproduce the firefox issue on an other machine that is connected to the same network, but using its own local drive instead of the nfs share (Elementary OS Luna, Firefox 39).

I see two solutions:
1) use local hard drive as target for all firefox disk I/O operations 
2) attach an SSD drive as write-back caching device to the LVG by harnessing dm-cache. This does away with the NFS bottleneck related to random I/O operations, but won't solve the issue of possibly eating a hefty portion of the ethernet bandwith.

We decided to do 2) anyway because of other reasons, but I need to investigate how to configure firefox  to use a user-defined folder for all disk I/O operations.

I can't send you wireshark logs. But I did a firefox log. If you're interested I can send you the file (90MB)! Let me know where I should upload it please (ideally not to a public place).

best regards
peter
Comment 33 Patrick McManus [:mcmanus] 2016-02-01 14:32:43 PST
reopen with new information

Note You need to log in before you can comment on or make changes to this bug.