Beginning on October 25th, 2016, Persona will no longer be an option for authentication on BMO. For more details see Persona Deprecated.
Last Comment Bug 580099 - Prefetch DNS for hosts needed during startup
: Prefetch DNS for hosts needed during startup
Status: RESOLVED DUPLICATE of bug 881804
: perf, privacy
Product: Core
Classification: Components
Component: Networking (show other bugs)
: unspecified
: All All
: -- normal with 1 vote (vote)
: ---
Assigned To: Steve Workman [:sworkman] (INACTIVE)
: Patrick McManus [:mcmanus]
Depends on: leveldb
Blocks: dns_perf
  Show dependency treegraph
Reported: 2010-07-19 16:20 PDT by Jason Duell [:jduell] (needinfo me)
Modified: 2015-01-18 03:22 PST (History)
11 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---

Prefetch DNS for hosts needed during startup.1 (17.98 KB, patch)
2010-12-14 09:03 PST, Patrick McManus [:mcmanus]
no flags Details | Diff | Splinter Review
Prefetch DNS for hosts needed during startup.2 (18.96 KB, patch)
2010-12-15 10:20 PST, Patrick McManus [:mcmanus]
no flags Details | Diff | Splinter Review

Description Jason Duell [:jduell] (needinfo me) 2010-07-19 16:20:59 PDT
Chromium keeps a list of hostnames that are likely to be needed at startup (open tabs, etc.), and then begins resolving them very early during program startup.  They're claiming this can speed up page loads by 250-500 (depends on DNS resolution time, obviously):

We should so something similar.  We'll need to keep a list of hostnames that are in use.  And we'll need to figure out how early we can fire off the resolve requests.
Comment 1 Carlos Alén Silva 2010-08-02 11:50:03 PDT
Shouldn't Platform be All All?
Comment 2 Patrick McManus [:mcmanus] 2010-12-14 09:03:48 PST
Created attachment 497511 [details] [diff] [review]
Prefetch DNS for hosts needed during startup.1

# HG changeset patch
# Parent 210237f7d62607a0122d5e5fe6dc4ac14981aecb
bug 580099 Prefetch DNS for hosts needed during startup

Upon shutting down record the hostnames for each window/tab as well as
the hostnames necessary for any imgs, scripts, or links (but not
anchors) on those pages.

Upon startup, as soon as the profile dir is identified, use a separate
thread to load all of those in and feed them to the DNS prefetch

Do not store hostnames if private browsing is enabled or if is not 3 (i.e. we are not restoring pages at


the storage is done with js and xul on shutdown, the dns resolution is in C++ to make it as early as possible. I think this is the first bit of xul/js I've submitted in a patch, so it will require quality reviewing - but it works and is therefore a useful stake in the ground.

Network speeds are going to have everything to do with how useful this is but in my experience several hundred names were all loaded into the firefox DNS cache by the time the restore-page logic needed them.. I only saw a few that were still in 'pending' instead of full hits -but even those are going to see a benefit of having started earlier.
Comment 3 Brian Smith (:briansmith, :bsmith, use NEEDINFO?) 2010-12-14 14:22:43 PST
I would think that the ordering of the prefetching would be important. For example, it would be better to resolve the DNS in this order:

1. the active tab's page's domain 
2. the domains of CSS, images, and (non-lazily-loaded)
   javascript on active tab's page.
3. The domains of each inactive tabs' pages.
4. The domains of all the items in the inactive tabs' pages.
Comment 4 Patrick McManus [:mcmanus] 2010-12-14 14:43:21 PST
(In reply to comment #3)
> I would think that the ordering of the prefetching would be important. For
> example, it would be better to resolve the DNS in this order:

we can do that.. I was considering adding a metric for base vs included, but active vs not is a good addition too.
Comment 5 Patrick McManus [:mcmanus] 2010-12-15 10:20:37 PST
Created attachment 497811 [details] [diff] [review]
Prefetch DNS for hosts needed during startup.2

implement load order using scheme in comment 3 - slightly refined so that images are ranked behind css/js
Comment 6 Brian Smith (:briansmith, :bsmith, use NEEDINFO?) 2010-12-17 12:36:11 PST
AFAICT, this design is based on the following assumptions that aren't obviously valid:

1. We will not be able to get use any resources out of the cache without validating them, it will be too slow to get them out of the cache, and/or there are no negative consequences of doing unnecessary DNS lookups.

2. DNS will not be used as a tracking mechanism and/or we are not concerned about tracking/privacy as far as DNS lookups go.

3. The DNS prefetching for the items in inactive tabs does not negatively impact the (perceived) loading time of the active tab. (Of course, I am also assuming that it especially important to show the active tab's content ASAP.)

4. We will attempt to load all the items on all pages. (What about ad blockers and similar things?)

5. The extra work at startup does not have a significant negative impact on other parts of startup. (The team that did the research about browser performance perception said that users care more about the (perceived) startup time of the browser than they did about page loading performance--even though improvements to page loading performance saves them more time in aggregate.)

If a page is cached and the cached representation is fresh then we might not even need to do a DNS lookup for that domain. The same applies to (fresh) cached resources loaded from that page. In this case the only way we are going to speed up the loading of the page is by getting the page out of the cache sooner. But, AFAICT, this design will, if anything, make us get the page out of the cache (slightly) later.
Comment 7 Brian Smith (:briansmith, :bsmith, use NEEDINFO?) 2010-12-29 10:41:12 PST
Added "See also" link to relevant Chromium issue. Basically, doing too many DNS queries at once has severely counter-productive performance implications in some configurations. (IIRC, I have one of those 2wire routers at home that I can test with; they are very common because AT&T deploys them to their home internet customers.)

We should also look for at prefetching DNS for services we contact during startup that affect the startup page loading time (e.g. the safebrowsing/url-classifier server) if and only if data from those servers would actually be needed to show the active tab.
Comment 8 Patrick McManus [:mcmanus] 2010-12-29 12:53:42 PST
(In reply to comment #7)
> Added "See also" link to relevant Chromium issue. Basically, doing too many DNS
> queries at once has severely counter-productive performance implications in
> some configurations. 

if we think that's a common enough scenario that's an argument for some kind of different throttle and scheduler for the dns system. It doesn't really have much to do with startup.

I certainly acknowledge the the chromium problem report where firefox is mentioned once in a while, but just 60 days ago I asked around both mozilla and BIND circles if anyone had heard of problems with firefox prefetching in the field and I got 0 responses. It seems likely to be linked to AAAA lookups which can be likely to timeout - some more careful thinking is probably useful there. So we should be cautious about throwing the baby out with the bathwater when making changes.
Comment 9 Brian Smith (:briansmith, :bsmith, use NEEDINFO?) 2010-12-29 15:29:10 PST
If we end up implementing captive portal checking and/or a HTTP pipelining pretest by contacting a known server, then we probably need to do those checks early and thus prefetching DNS for those servers might also be a win.
Comment 10 Jason Duell [:jduell] (needinfo me) 2011-08-15 10:55:44 PDT
Seems like we could leverage whatever solution we come up with for bug 580104 (which seems higher priority too) to solve this--just store an entry for a bogus 'startup' host at shutdown that will return all DNS hostnames needed at startup.
Comment 11 Jason Duell [:jduell] (needinfo me) 2011-08-17 14:05:34 PDT
And it looks like that will be leveldb...
Comment 12 Jason Duell [:jduell] (needinfo me) 2012-01-04 14:50:25 PST
Comment on attachment 497811 [details] [diff] [review]
Prefetch DNS for hosts needed during startup.2

Review of attachment 497811 [details] [diff] [review]:


We should split this patch into two parts, and have someone more qualified than me review the xul/JS parts.

Also, I assume we'd rather wait for levelDB and store it using that instead of SQLite?  I could be persuaded to land it with SQLite if 1) leveldb is going to be a while, and 2) it's easy to remove any existing SQLite tables that we use here once we port to levelDB.

::: browser/base/content/hotDNS.js
@@ +68,5 @@
> +            }
> +        }
> +    },
> +    
> +    process: function(browser, baseMetric) {

s/process/some better name/   maybe "saveHostnamesFromTab"?

@@ +87,5 @@
> +                                 d.getElementsByTagName('script'), 'src',
> +                                 30 + baseMetric);
> +            this.processElements(browser,
> +                                 d.getElementsByTagName('link'), 'href',
> +                                 30 + baseMetric);

Lower number == higher prefetch priority, right?  Do we want to resolve DNS for href links before img's?  I would have thought we'd want img to be higher priority.

@@ +112,5 @@
> +        catch (e) {
> +            return;
> +        }
> +
> +        HotDNSHostHash = new Array();

I got curious about how you're using a Array as an associative array (ie storing with hostnames instead of ints as the keys), and came across  

which says "you shouldn't use an array as an associative array. You can use plain objects instead, although doing so comes with its own caveats." In bug 609941 some thunderbird code got changed to also make sure that the keys never conflict with base prototype properties: I'm guessing this isn't an issue if you use hostnames, though.  But this should probably be "HotDNSHostHas = {}" IIUC.   This is just more proof we need someone who actually knows JS to review this part :)
Comment 13 Patrick McManus [:mcmanus] 2012-01-04 14:53:06 PST
I don't have much interest in this patch anymore. Will pass it on to steve to see if it can help him - if not he can close it.
Comment 14 Steve Workman [:sworkman] (INACTIVE) 2013-06-24 14:26:00 PDT
Duping this to bug 881804, as the work on predictive connections is being discussed there.

*** This bug has been marked as a duplicate of bug 881804 ***

Note You need to log in before you can comment on or make changes to this bug.