Bugzilla

Comment 10

•

9 years ago

I asked this on the dns-operations@ list, but nobody answered:

> In any case, it would be helpful to the release mgmt team if they had a better idea of the
> problems this is causing and how critical of an issue it is, in order to better prioritize
> it (once a true cause has been found). Also, any ideas on timeline as to when this started
> would help a lot.

So, if any DNS provider/operator wants to comment on that, that would be very useful.

As for the timeline bit, sounds like it started around February 25, so likely tied to Firefox 36 release.

Comment 11

•

9 years ago

(In reply to Reed Loden [:reed] from comment #10)
<snip>

... and Kelcey just replied in the previous comment.

winfried.angele

Comment 12

•

9 years ago

It burdens DNS recursors and authoritatives unnecessarily. Especially if they request TCP for ANY queries (tc).

Kelsey Cummings

Comment 13

•

9 years ago

(In reply to Reed Loden [:reed] from comment #10) 
> So, if any DNS provider/operator wants to comment on that, that would be
> very useful.

I don't know if I'd call it critical but did catch our attention well enough that we needed to track down why we saw a sudden large unexplained increase in ANY requests.  A large national provider might have a stronger opinion about the effects of this behavior.

Assignee

Comment 14

•

9 years ago

Hi all - thanks for all the comments! I appreciate that seeing something new on your networks can be alarming; hopefully the following can explain our approach and mitigate your concerns.

Here’s some background: The goal was to get accurate TTLs for DNS records to improve accuracy for hostname-to-IP resolution. This was added in Bug 820391, and enabled in Bug 1084645. It is only enabled for Windows currently.

We had to use this approach of requesting A/AAAA then ANY because of the particular nuances of hostname resolution APIs:
— getaddrinfo, the POSIX cross-platform API doesn’t give access to TTL by itself. Previously on Windows, and currently on other platforms, we provided a fixed expiration time for the record because we can’t get the accurate TTL from the record itself.
— DnsQuery on Windows does provide access to the TTL, but it doesn’t provide an easy way to do an UNSPEC request - this means that not only could we not get A and AAAA at the same time, but we would miss out on NetBT resolutions. And the ordering of the addresses in the response would be different - this can be configured differently on different networks and machines and Windows takes care of that complexity in getaddrinfo. — Basically, DnsQuery on it’s own could have caused a ton of compatibility issues.

So, our solution was to call getaddrinfo first to get correct hostname resolutions, then perform an async-but-after call to DnsQuery using ANY to get the TTL information. Because it’s async, we’re not waiting on the client side for the response - if it doesn’t come, Firefox will use the fixed TTL we use on other platforms and it will run fine. We also apply a higher priority to the getaddrinfo requests; since they’re purpose is to get the list of resolved addresses, connections from Firefox should not be blocked by the secondary DnsQuery which is just about getting TTL for A and AAAA records.

In addition, in order to mitigate some of the issues brought up in this bug, we used several flags with our DnsQuery call:

DNS_QUERY_NO_NETBT — although we want NetBT results in the first call to getaddrinfo, we’re not as concerned about getting TTL for those, so we don’t want those requests on the wire.
DNS_QUERY_NO_MULTICAST — don’t do a multicast request on the network; we only want to query the servers.
DNS_QUERY_ACCEPT_TRUNCATED_RESPONSE — truncated responses are ok - this should avoid retries over TCP if we get a truncated response.

Hopefully that answers most of your concerns.

Of course, if anyone is seeing a real connection issue to do with this, the details of that would be most appreciated. Our goal has been to improve the connection accuracy of Firefox, not degrade it.

(Also, check out the pref “network.dns.get-ttl” in about:config - you can disable the TTL/ANY request by setting that to false).

Masatoshi Kimura [:emk]

Comment 15

•

9 years ago

Attached patch 1093983_use_aaaa_query — Details — Splinter Review

This is basically backout of bug 825501.

Comment 16

•

9 years ago

Andrew Sullivan (Director of Architecture at Dyn) posted the following to dns-operations@, so I'm reposting it here, as he does not have an account. Also viewable at https://lists.dns-oarc.net/pipermail/dns-operations/2015-February/012896.html.

> Hi,
> 
> I can't speak for recursive operators.  I looked for about 10 seconds
> into this on Dyn's authoritative data today, and there was a giant
> spike in ANY queries visible on them.  Note that this could be a
> coincidence -- I didn't check.  The increase is days and not weeks
> old.
> 
> I've read the bugzilla ticket, though, and it's not clear that the
> implementers understand what ANY does.  I don't have a login, but the
> ticket says that this list is being monitored so I hope this helps.
> ANY is _not_ a way of getting multiple records at once, and there's
> good reason to suppose (given the ANY queries I see above) that it's
> not actually happening in the order Steve Workman appears to think it
> is.
> 
> According to reporter causeless, what one sees is an A query followed
> by an ANY query.  On any properly-working system, the ANY query will
> provide exactly and only the response for the A query just resolved.
> ANY is not really an RRTTYPE, but rather a QTYPE that says, "Gimme
> what you have."  That this is resulting in cache abuse suggests that
> what's _actually_ happening is that the A or AAAA and the ANY are
> being issued in an unpredictable order.
> 
> In particular, this bit of Workman's response appears to suggest a
> misunderstanding of how DNS works:
> 
> 
> >    We had to use this approach of requesting A/AAAA then ANY because
> >    of the particular nuances of hostname resolution APIs: —
> >    getaddrinfo, the POSIX cross-platform API doesn’t give access to
> >    TTL by itself. Previously on Windows, and currently on other
> >    platforms, we provided a fixed expiration time for the record
> >    because we can’t get the accurate TTL from the record itself.  —
> >    DnsQuery on Windows does provide access to the TTL, but it doesn’t
> >    provide an easy way to do an UNSPEC request - this means that not
> >    only could we not get A and AAAA at the same time, but we would
> >    miss out on NetBT resolutions. And the ordering of the addresses
> >    in the response would be different - this can be configured
> >    differently on different networks and machines and Windows takes
> >    care of that complexity in getaddrinfo. — Basically, DnsQuery on
> >    it’s own could have caused a ton of compatibility issues.
> 
> I'm aware of the crappy no-TTL stuff in getaddrinfo.  (I suggest that
> to fix this for real, adopt the getdns api from
> https://getdnsapi.net.)  It is not possible, under POSIX or any other
> DNS query method, to get A and AAAA "at the same time": these are two
> different queries.  That's required by the DNS protocol because of
> caches.  It might well be that the DnsQuery interface (I haven't
> investigated it) ought to provide that factility, but according to the
> above complaint it doesn't.  ANY won't help you here: if there's
> anything in the cache, then ANY will return that and nothing else.
> 
> Also, this seems to suggest that there's an order to DNS query
> responses.  This is a mistake.  DNS provides unordered sets of data.
> In fact, against many modern resolvers, repeated queries for the same
> name, class, and type will return the results in different orders; but
> this is actually just to fake naive client implementations into using
> random records, since most of them just pick the first one.
> 
> In general, ANY is useful for troubleshooting but should never be used
> for regular operation.  Its output is unpredictable given the effects
> of caches.  It can return enormous result sets, and some authoritative
> servers have taken to refusing ANY queries because of the frequency
> with which such queries show up in amplification attacks.
> 
> The approach described in this ticket is broken.  I'm happy to talk
> with people about how to solve the problem they're trying to solve if
> that would help, but I suspect it's going to be harder than
> implementers think.  (This is a special case of the DNS Rule, which is
> "It's harder than it ought to be." :-/ )

Florian Weimer

Comment 17

•

9 years ago

(In reply to winfried.angele from comment #12)
> It burdens DNS recursors and authoritatives unnecessarily. Especially if
> they request TCP for ANY queries (tc).

There might be some confusion about this because ANY is not always implemented as “give me what's in your cache”.  Some caching resolvers send a query to upstream servers even if they already have cached data at the QNAME.

Assignee

Comment 18

•

9 years ago

Thanks for reposting Reed. Certainly open to learning more... Some thoughts inline.

(In reply to Reed Loden [:reed] from comment #16)
> Andrew Sullivan (Director of Architecture at Dyn) posted the following to
> dns-operations@, so I'm reposting it here, as he does not have an account.
> Also viewable at
> https://lists.dns-oarc.net/pipermail/dns-operations/2015-February/012896.
> html.
> 
> > Hi,
> > 
> > I can't speak for recursive operators.  I looked for about 10 seconds
> > into this on Dyn's authoritative data today, and there was a giant
> > spike in ANY queries visible on them.  Note that this could be a
> > coincidence -- I didn't check.  The increase is days and not weeks
> > old.

If we're sending an ANY for every A/AAAA request, then I can imagine that spike coming from Firefox.

> > I've read the bugzilla ticket, though, and it's not clear that the
> > implementers understand what ANY does.  I don't have a login, but the
> > ticket says that this list is being monitored so I hope this helps.
> > ANY is _not_ a way of getting multiple records at once, and there's
> > good reason to suppose (given the ANY queries I see above) that it's
> > not actually happening in the order Steve Workman appears to think it
> > is.
> > 
> > According to reporter causeless, what one sees is an A query followed
> > by an ANY query.  On any properly-working system, the ANY query will
> > provide exactly and only the response for the A query just resolved.
> > ANY is not really an RRTTYPE, but rather a QTYPE that says, "Gimme
> > what you have."  That this is resulting in cache abuse suggests that
> > what's _actually_ happening is that the A or AAAA and the ANY are
> > being issued in an unpredictable order.

I'd like to know more about what is meant by cache abuse here. "Gimme what you have" is what we were aiming for.

> > In particular, this bit of Workman's response appears to suggest a
> > misunderstanding of how DNS works:
> > 
> > 
> > >    We had to use this approach of requesting A/AAAA then ANY because
> > >    of the particular nuances of hostname resolution APIs: —
> > >    getaddrinfo, the POSIX cross-platform API doesn’t give access to
> > >    TTL by itself. Previously on Windows, and currently on other
> > >    platforms, we provided a fixed expiration time for the record
> > >    because we can’t get the accurate TTL from the record itself.  —
> > >    DnsQuery on Windows does provide access to the TTL, but it doesn’t
> > >    provide an easy way to do an UNSPEC request - this means that not
> > >    only could we not get A and AAAA at the same time, but we would
> > >    miss out on NetBT resolutions. And the ordering of the addresses
> > >    in the response would be different - this can be configured
> > >    differently on different networks and machines and Windows takes
> > >    care of that complexity in getaddrinfo. — Basically, DnsQuery on
> > >    it’s own could have caused a ton of compatibility issues.
> > 
> > I'm aware of the crappy no-TTL stuff in getaddrinfo.  (I suggest that
> > to fix this for real, adopt the getdns api from
> > https://getdnsapi.net.)

Adopting a different API is a bigger issue, unless we want to adopt a stub resolver. And that has it's own issues, the biggest of which we identified as aligning the configuration with windows for it's ordering and windows naming. Getting TTL in this way was our attempt to get the best parts from the two OS APIs available to us, getaddrinfo and DnsQuery.

> > It is not possible, under POSIX or any other
> > DNS query method, to get A and AAAA "at the same time": these are two
> > different queries.  That's required by the DNS protocol because of
> > caches.  It might well be that the DnsQuery interface (I haven't
> > investigated it) ought to provide that factility, but according to the
> > above complaint it doesn't.  ANY won't help you here: if there's
> > anything in the cache, then ANY will return that and nothing else.

Ah, let me clarify here. I didn't meant to suggest that A and AAAA were being requested in the same DNS request on the wire. I meant that we can get them from getaddrinfo in the same request. Yes, on the wire it's two separate requests.

> > Also, this seems to suggest that there's an order to DNS query
> > responses.  This is a mistake.  DNS provides unordered sets of data.
> > In fact, against many modern resolvers, repeated queries for the same
> > name, class, and type will return the results in different orders; but
> > this is actually just to fake naive client implementations into using
> > random records, since most of them just pick the first one.

Actually, this ordering issue is not about the servers, but about the OS. The results as presented to Firefox from getaddrinfo on Windows are ordered according to its configuration: [1] is an example thread in which someone has queried this. On some networks, Windows naming is preferred to regular DNS - we were attempting not to break that.

> > In general, ANY is useful for troubleshooting but should never be used
> > for regular operation.  Its output is unpredictable given the effects
> > of caches.  It can return enormous result sets, and some authoritative
> > servers have taken to refusing ANY queries because of the frequency
> > with which such queries show up in amplification attacks.
> > 
> > The approach described in this ticket is broken.  I'm happy to talk
> > with people about how to solve the problem they're trying to solve if
> > that would help, but I suspect it's going to be harder than
> > implementers think.  (This is a special case of the DNS Rule, which is
> > "It's harder than it ought to be." :-/ )

No disagreements there! Definitely not easy :)

[1] http://www.tech-archive.net/Archive/Windows/microsoft.public.windows.server.dns/2009-11/msg00159.html

Comment 19

•

9 years ago

Hi All - thanks for your comments here. I need to make a few points

1] Let's stipulate that the new firefox release does use ANY queries for the first time, so that's why measurements of ANY are up. That is clear.

2] Please understand that the intent of the algorithm is to both increase the correctness of our cache (first order objective) and increase the cache hit rate (decreasing overall dns traffic to the resolver) by making a dns query with a ttl available while still making addressing decisions based on getaddrinfo to ensure legacy compatibility.

3] its certainly plausible that ANY traffic is up dramatically but traffic overall is not significantly up (thanks to better cache effects). I'm actually willing to tolerate some increase in the name of correctness, but certainly the intention is not to increase things hugely. That being said, I haven't seen anything in this bug or in that mailing list traffic that suggests overall load is a problem. What I see is that specifically the ANY metric has changed (see #1) and that's just part of the picture.

4] I do think we can probably move away from ANY and achieve the same results - and we should. I want to understand if there is an exigent impact here though in a way that such a change shouldn't follow the normal development rollout pattern. I haven't really seen evidence of that other than the comment about amplification, which is indeed interesting and food for thought.

5] Discussions about correctness around ANY aren't compelling from a exigency pov - we're basically matching a record from ANY against the record from getaddrinfo() to find the ttl. If we can't find one we use the fallback constant that we've always used (and would use if we disabled the feature). But as I said in #4 - it seems this can be done more efficiently and more reliably with a tweak to the algorithm. (see 7)

6] getdnsapi is a good API - unfortunately it is not yet a suitable implementation afaict - I'm keeping my eye on it. The github documentation says it doesn't support windows, it does not appear to support non-dns naming and host configured precedence for that (windows names, yp, whatever else creates exact compatibility with getaddrinfo which would be a hard requirement), and the issue of code size vs the value it provides would need to evaluated.

7] with all of that being said, I think the firefox goals can be met while tweaking the current algorithm to lookup A/AAAA instead of ANY in some manner. I think the change is a little more complicated to execute than that but doable. Steve will lead it and I'll make a judgment on it as module owner.

8] What I need to understand in the short term, for the sake of this bug, is if there is an exigent reason the current behavior needs to stop immediately, or if this is the kind of change that we can roll out through the normal change process (~12 weeks). So far I would think it could take the normal path.

Florian Weimer

Comment 20

•

9 years ago

(In reply to Patrick McManus [:mcmanus] from comment #19)
> 2] Please understand that the intent of the algorithm is to both increase
> the correctness of our cache (first order objective) and increase the cache
> hit rate (decreasing overall dns traffic to the resolver) by making a dns
> query with a ttl available while still making addressing decisions based on
> getaddrinfo to ensure legacy compatibility.

Minor nit: If you delay resolver queries in the browser based on the TTL, you decrease the resolver cache rate because you will send fewer queries that can be answered from cache.

> 3] its certainly plausible that ANY traffic is up dramatically but traffic
> overall is not significantly up (thanks to better cache effects).

Any TTL-based reduction will only have an impact on the query rate at the caching resolver.  The caching resolver already acts as a filter, so that authorative servers do not see a difference.  In contrast, the ANY query is forwarded upstream in some cases (see comment 17).  This means that a claim about overall reduction is dubious.  And from a caching resolver perspective, cache hits which do not require TCP fallback are extremely cheap, compared to cache misses (whether they use TCP or not).

Petr Špaček

Comment 21

•

9 years ago

This whole thing is about getting TTL for in-Firefox caching, right? AFAIK all Windows versions from XP up have local DNS cache running on every machine and on Linux glibc provides that though nscd. This leads to question if in-Firefox cache brings additional benefits or not.

Maybe it would be valuable to have a Firefox build with in-Firefox caching completely disabled and to do some performance tests to see if machine-wide cache works for Firefox. There would be no reason to bother with looking at TTL inside Firefox if the system-wide cache works for you.

Comment 22

•

9 years ago

((In reply to bugzilla.mozilla.org from comment #21)
> This whole thing is about getting TTL for in-Firefox caching, right? 

yes - each time we look at this we do need the cache because the the number of names that are looked up in the browser (most of them speculatively) tends to overwhelm the os cache. I'd be happy to look at data in a separate bug to revisit, of course.

Comment 23

•

9 years ago

(In reply to Patrick McManus [:mcmanus] from comment #19)
>
>> 
> 8] What I need to understand in the short term, for the sake of this bug, is
> if there is an exigent reason the current behavior needs to stop [quickly]

based on some more research and conversations I've decided we should make a quick change, based on changing the pref rather than backing out code. This is because I'm concerned about amplification and the perception thereof by security monitors.

The use of TCP would have also been a problem for me, but I believe that is just speculation from posters because of how ANY is often correlated with TCP - but our query is made with DNS_QUERY_ACCEPT_TRUNCATED_RESPONSE which is supposed to prevent tcp based retries.

The change will be to shift to [AAA]A for the ttl query. Steve is going to take it.

Comment 24

•

9 years ago

There is also this https://support.mozilla.org/en-US/questions/1049324

which makes very little sense on the merits (inbound to firefox connection alerts?) - but is obviously something we don't want to trigger.

Assignee

Comment 25

•

9 years ago

https://treeherder.mozilla.org/#/jobs?repo=try&revision=76a638cc3797

Assignee

Comment 26

•

9 years ago

https://treeherder.mozilla.org/#/jobs?repo=try&revision=373f2625093f

Assignee

Comment 27

•

9 years ago

Attached patch Disable type ANY request in DNS used to get TTL on Windows — Details — Splinter Review

Adding patch to disable the pref - network.dns.get-ttl -> false.

r=mcmanus based on external conversations.

Pushed to 'try' to get some builds and run unit tests; based on mozilla-release:
-- https://treeherder.mozilla.org/#/jobs?repo=try&revision=76a638cc3797

Second 'try' push based on mozilla-central in prep for landing:
-- https://treeherder.mozilla.org/#/jobs?repo=try&revision=373f2625093f

I would very much appreciate verification by those folk on this bug that ANY requests are no longer coming from Firefox in this build.

I've opened a second bug to work on the ANY -> A/AAAA change - Bug 1138242.

Assignee: nobody → sworkman

Status: NEW → ASSIGNED

Attachment #8571098 - Flags: review+

Assignee

Updated

•

9 years ago

Blocks: 1138242

Summary: DNS resolver should use AAAA than ANY → DNS resolver should not use 'ANY' to get cached records for TTL

bmozilla

Comment 28

•

9 years ago

(In reply to Patrick McManus [:mcmanus] from comment #22)
> ((In reply to bugzilla.mozilla.org from comment #21)
> > This whole thing is about getting TTL for in-Firefox caching, right? 
> 
> yes - each time we look at this we do need the cache because the the number
> of names that are looked up in the browser (most of them speculatively)
> tends to overwhelm the os cache. I'd be happy to look at data in a separate
> bug to revisit, of course.

Can you be more specific on what "overwhelm"s the os cache?  I assume all the names still need to be looked up in the os and thus get in the os cache.  Is it cache hits that are too many - that sounds unlikely.  Or is it entries with very low ttl that you are caching longer in the browser to avoid lookups?

Assignee

Comment 29

•

9 years ago

(In reply to Steve Workman [:sworkman] (please use needinfo) from comment #27)
> Pushed to 'try' to get some builds and run unit tests; based on
> mozilla-release:
> -- https://treeherder.mozilla.org/#/jobs?repo=try&revision=76a638cc3797
> ...
> I would very much appreciate verification by those folk on this bug that ANY
> requests are no longer coming from Firefox in this build.

Direct link to the builds to be verified (based on mozilla-release):
-- http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/sworkman@mozilla.com-76a638cc3797/try-win32/

Assignee

Comment 30

•

9 years ago

Comment on attachment 8571098 [details] [diff] [review]
Disable type ANY request in DNS used to get TTL on Windows

Approval Request Comment
[Feature/regressing bug #]: Bug 1084645
[User impact if declined]: Possible connection issues for Windows users; Firewall issues.
[Describe test coverage new/current, TreeHerder]: Verified locally.
[Risks and why]: Very little risk; reverts to previous behavior, same as non-Windows platforms.
[String/UUID change made/needed]: n/a

Inbound also requires approval at the moment. I'd like approval to land there too.

Attachment #8571098 - Flags: approval-mozilla-release?

Attachment #8571098 - Flags: approval-mozilla-beta?

Attachment #8571098 - Flags: approval-mozilla-aurora?

Liz Henry (:lizzard) (relman/hg->git project)

Assignee

Comment 31

•

9 years ago

Inbound open again; just landed the patch.
https://hg.mozilla.org/integration/mozilla-inbound/rev/97d3ba6cfed8

Loic

Updated

•

9 years ago

Keywords: regression

Comment 32

•

9 years ago

Comment on attachment 8571098 [details] [diff] [review]
Disable type ANY request in DNS used to get TTL on Windows

Approving for uplift based on conversation with Sylvestre on irc.

Attachment #8571098 - Flags: approval-mozilla-release?

Attachment #8571098 - Flags: approval-mozilla-release+

Attachment #8571098 - Flags: approval-mozilla-beta?

Attachment #8571098 - Flags: approval-mozilla-beta+

Attachment #8571098 - Flags: approval-mozilla-aurora?

Attachment #8571098 - Flags: approval-mozilla-aurora+

Ryan VanderMeulen [:RyanVM]

Comment 33

•

9 years ago

https://hg.mozilla.org/releases/mozilla-aurora/rev/cb533bea322d

status-firefox38: affected → fixed

Ryan VanderMeulen [:RyanVM]

Comment 34

•

9 years ago

https://hg.mozilla.org/releases/mozilla-beta/rev/082769bdd62a

status-firefox37: affected → fixed

Ryan VanderMeulen [:RyanVM]

Comment 35

•

9 years ago

https://hg.mozilla.org/releases/mozilla-release/rev/c37f6a5a0c8a

status-firefox36: affected → fixed

Carsten Book [:Tomcat]

Comment 36

•

9 years ago

https://hg.mozilla.org/mozilla-central/rev/97d3ba6cfed8

Status: ASSIGNED → RESOLVED

Closed: 9 years ago

status-firefox39: affected → fixed

Resolution: --- → FIXED

Target Milestone: --- → mozilla39

Sylvestre Ledru [:Sylvestre]

Updated

•

9 years ago

Flags: needinfo?(winfried.angele)

Comment 37

•

9 years ago

Added to the relnotes with "36.0.1 - Disable the usage of the ANY DNS query type (1093983)" as wording

relnote-firefox: --- → 36+

Comment 38

•

9 years ago

See also https://blog.cloudflare.com/deprecating-dns-any-meta-query-type/

winfried.angele

Comment 39

•

9 years ago

Attached image ANY returns to normal level. — Details

The fix works, thanks!

Chris

Comment 40

•

9 years ago

(In reply to Patrick McManus [:mcmanus] from comment #22)
> ((In reply to bugzilla.mozilla.org from comment #21)
> > This whole thing is about getting TTL for in-Firefox caching, right? 
> 
> yes - each time we look at this we do need the cache because the the number
> of names that are looked up in the browser (most of them speculatively)
> tends to overwhelm the os cache. I'd be happy to look at data in a separate
> bug to revisit, of course.

I hsve actually experimented with disabling the cache, and I do not notice a difference with it on or off.  Although I do disable prefetch.

I have always been of the opinion that apps using their own internal dns cache is annoying, I can understand why some do it such as smtp servers, but I am not convinced a browser should be doing it.

I also remember an incident a while back that when I toggled a dns option in the app it crashed my router, I did some debugging and found dns lookups were been sent to the query for each hostname even when the hostname matched, so e.g. lets say browsing youtube and is 30 images on the page, all the same hostname, firefox was sending a dns lookup for each image even tho was the same hostname and it somehow also bypassed the OS cache, this was a while ago tho and in a much older version of firefox, I think prefetching if I remembered right caused the behaviour hence me turning it off.

I have a few suggestions but I think none will be accepted.

1 - remove the firefox cache and disable prefetching by default.
2 - keep prefetching on but none prefetched dns lookups use the OS cache, although I think this wont be possible when clicking on linkls, only when typing/pasting them in.
3 - disable browser caching for mainstream url's which are very likely to be in OS cache, or dns cachers.

On my network, I have the OS cache, and my router as a cache.

For me I see prefetching as the problem here, is it really needed?