1306004 - prefetch requests should be displayed in network monitor

Status: NEW

(DevTools :: Netmonitor, enhancement, P2)

Description

[Ben Kelly [:bkelly, not reviewing]]

I believe we have network predictor and prefetch enabled in nightly and preparing to ride the trains.  We should ensure that any network requests made by prefetch show up in network monitor.

Comment 1

[Ruturaj Vartak]

Hi :jsnajdr,

I'll like to try fix this bug if it can be solved without any C / C++ and pure JS, HTML, CSS

Comment 2

[Ruturaj Vartak]

Adding some reference links

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Link_prefetching_FAQ
[2] https://css-tricks.com/prefetching-preloading-prebrowsing/
[3] https://www.igvita.com/2015/08/17/eliminating-roundtrips-with-preconnect/

Comment 3

[Jarda Snajdr [:jsnajdr]]

Netmonitor will display the prefetch request only if its channel's loadInfo/loadGroup/loadContext says that it belongs to the document displayed in the current tab.

However, the prefetch channel at [1] is created with the system principal, and null loadGroup. It's a system request that will be displayed only in browser toolbox. There's no way to know to which tab/document it belongs.

Questions for Nick:
- could the prefetch channel load with the document principal and with the right loadInfo? Is there a direct relation between the document and the prefetch actions?

- what's the best way to write a browser mochitest for this? I.e., load a page, perform some deterministic actions to trigger a prefetch, and then check that it appeared in Netmonitor?

Ruturaj, I'm afraid this is going to be a C++ work in the Necko's predictor code. The only pure JS will be the Netmonitor mochitest.

[1] http://searchfox.org/mozilla-central/source/netwerk/base/Predictor.cpp#1288-1294

Comment 4

[u408661]

So I'm going to flip this on its head, and ask whether or not this is actually the right thing to be doing. While the goal is to never prefetch things we don't end up using, we have no guarantee that it won't happen - so right there we have an issue where people looking at the net monitor could see resource loads that have no relation to the page.

Another thing to consider - we don't show system loads that happen in the background for things like blocklist updates, etc. I know those are in some way a different class of load, but they're also somewhat the same as predictive prefetches - things firefox has decided to do independent of the markup of any page you're looking at.

Finally, it's not like these resources will just not appear in netmonitor. The prefetch channel is something that happens secretly in the background. Firefox still opens up a "real" channel for that resource, it just so happens that that channel gets its response from the cache.

There would be a lot of work to be done to get prefetches loading with the "right" principal and loadinfo (though I take issue with using the term "right" here). Possibly including trying to serialize things to disk that are not meant to be serialized and/or serializing things that will take up a large amount of disk space to the point where the perf gains from prefetching are negated by all the extra work the cache and predictor have to do.

So what I think we might want to do is - instead of showing the channel that does the prefetch, let's instead add another state for "prefetched" to be shown in the netmonitor (which can be determined with data we already have) similar to how we currently show if a resource came from cache.

If we really want the timing info for prefetched resources, we could store that as metadata on the forced-valid entry (perhaps even in a way that makes it so that metadata is deleted when the entry is no longer forced-valid, we'd have to ask the cache people about that bit though).

Comment 5

[Ben Kelly [:bkelly, not reviewing]]

Nick, is this prefetch triggered via content using rel="prefetch" or something similar?  If its caused by content hinting, then it seems like devs should be able to see the effect they are having on the browser.

I would NI Nick here, but he is blocking requests.  Patrick, do you know?

Comment 6

[Patrick McManus [:mcmanus]]

the newish network prefetch code is not triggered via content. that's its best point :)

Comment 7

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Patrick McManus [:mcmanus] from comment #6)
> the newish network prefetch code is not triggered via content. that's its
> best point :)

So what would you expect developers to see in network panel when this feature kicks in?  A "cached" response for the request from the page?  Is there any way to indicate it was pre-fetched to avoid confusion?

Comment 8

[Jarda Snajdr [:jsnajdr]]

(In reply to Ben Kelly [:bkelly] from comment #7)
> So what would you expect developers to see in network panel when this
> feature kicks in?  A "cached" response for the request from the page?  Is
> there any way to indicate it was pre-fetched to avoid confusion?

My ideas on how Netmonitor could give developers more visibility into the predictor behavior:

1. There should be a new nsContentPolicy constant TYPE_PREFETCH that all prefetching channels will set as its contentPolicyType. Netmonitor will then show their "cause" field as "prefetch", instead of the generic "other".

2. Prefetches triggered by <link rel="prefetch"> are already visible in the tab's Netmonitor. The prefetches done by the predictor are not, and as Nick explained, probably never will. But they are visible in the browser toolbox. That seems OK, given they can be distinguished by the "cause" field.

3. Prefetched responses are now shown as "cached" in the tab's Netmonitor. We should be able to detect that the response comes from the predictor. I'm not sure how, but I don't expect it to be a trouble. We're listening on the http-on-examine-cached-response listener. There will be 3 ways how to get a response from a non-network source: cache, prefetched, service worker.

4. The predictor also pre-resolves DNS and pre-connects to servers. Netmonitor shows the request timing info, which includes the DNS and TCP times. The timings view could be extended to show that pre-resolution and/or pre-connection happened on a particular request.

Comment 9

[Patrick McManus [:mcmanus]]

:jsnajdr makes sense to me. file any bugs you need assistance with

Comment 10

[u408661]

(In reply to Jarda Snajdr [:jsnajdr] from comment #8)
> (In reply to Ben Kelly [:bkelly] from comment #7)
> > So what would you expect developers to see in network panel when this
> > feature kicks in?  A "cached" response for the request from the page?  Is
> > there any way to indicate it was pre-fetched to avoid confusion?
> 
> My ideas on how Netmonitor could give developers more visibility into the
> predictor behavior:
> 
> 1. There should be a new nsContentPolicy constant TYPE_PREFETCH that all
> prefetching channels will set as its contentPolicyType. Netmonitor will then
> show their "cause" field as "prefetch", instead of the generic "other".
> 
> 2. Prefetches triggered by <link rel="prefetch"> are already visible in the
> tab's Netmonitor. The prefetches done by the predictor are not, and as Nick
> explained, probably never will. But they are visible in the browser toolbox.
> That seems OK, given they can be distinguished by the "cause" field.

These two things seem exceptionally reasonable to me. I like it!

> 3. Prefetched responses are now shown as "cached" in the tab's Netmonitor.
> We should be able to detect that the response comes from the predictor. I'm
> not sure how, but I don't expect it to be a trouble. We're listening on the
> http-on-examine-cached-response listener. There will be 3 ways how to get a
> response from a non-network source: cache, prefetched, service worker.

I'd have to look at where the http-on-examine-cached-response notification happens, but it seems reasonable that we could do something there, with the channel keeping track of if it used a cached response that was forced valid (info it already has, but doesn't store anywhere once it makes the decision based on that info).

> 4. The predictor also pre-resolves DNS and pre-connects to servers.
> Netmonitor shows the request timing info, which includes the DNS and TCP
> times. The timings view could be extended to show that pre-resolution and/or
> pre-connection happened on a particular request.

This will be harder to know - for example, did we already have a connection because it was a preconnect, or because there was already a connection open to that host from another resource download? We do (IIRC) mark connections as preconnections, so there's probably a simple solution.

The preresolve thing is harder - we don't mark resolutions as speculative in any way (because DNS resolutions are so cheap in terms of bandwidth, we don't really care). Even harder, a resolution could happen super-fast because it was in the OS cache from some earlier resolution of the same hostname (whether from Firefox or some other process).