1348983 - BehindTheScene requests (fe. HSTS priming HEAD requests) not listed in Network DevTools

Status: UNCONFIRMED

(DevTools :: Netmonitor, enhancement, P3)

Description

[Dakota]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:52.0) Gecko/20100101 Firefox/52.0
Build ID: 20170316213829

Steps to reproduce:

Some requests are not shown in the DevTools Network Tab for example the HEAD requests for HSTS priming. uMatrix lists the requests and the Live HTTP Headers add-on also sees and lists the requests.
uMatrix marks the HEAD requests as BehindTheScene requests

The built-in devtools should also list *all* requests IMHO. At least there should be a pref to enable it.

(steps to reproduce)
0. run FF with a local proxy or install an add-on like uMatrix or Live HTTP Headers to monitor requests
1. clear the cache including the siteSettings
2. set security.mixed_content.block_display_content, security.mixed_content.use_hsts and security.mixed_content.send_hsts_priming all to true
3. open the Network Tab and go to https://earthlng.github.io/testpages/mixedcontent.htm


Actual results:

the 2nd and 3rd image are displayed (as expected) and the Network Tab only shows those 2 requests + the one for the document itself


Expected results:

all 5 requests should get listed in the Network Tab

1. the document itself (GET to earthlng.github.io)
2. HEAD request to cdn.ghacks.net
3. HEAD request to 1.f.ix.de
4. GET request to 1.f.ix.de
5. GET request to agrias.com.br

I noticed it in FF52.0.1 but also tested it in Nightly 55.0a1 (2017-03-20) and it's the same behavior ie. the requests are not shown in the Network tab

The built-in devtools should list *all* requests IMHO. At least there should be a pref to enable it. Thanks

Comment 1

[Loic]

NB: Retriage to Netmonitor if the component is wrong.

Comment 2

[Daniel Veditz [:dveditz]]

If the other add-ons can see the requests then it doesn't seem to be a problem on the "DOM: Security" end, but rather one of Dev Tools policy about which requests to include. Do CORS pre-flight requests have the same problem?

Comment 3

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Attachment: test.png

was following steps from comment #0, but I can see just one request in my Netmonitor panel. Is that expected?
(screenshot attached)

Honza

Comment 4

[Dakota]

Hi, thanks for giving it a shot. Idk why it doesn't work for you. Maybe an addon or pref is blocking something?
I re-tested it just now and the testpage still works for me. However in PB mode the 2nd image doesn't load with send_hsts_priming=false because I guess it doesn't see the cache or one of my pref settings is interfering.
The 3rd picture also requires network.stricttransportsecurity.preloadlist;true but since that's the default value I didn't mention it. It's weird that not even the 3rd image loads for you.

Also something seems to have changed again with the HSTS priming feature between FF52 and 55nightly.
Nightly now seems to behave the same as FF51 did in that use_hsts=true doesn't properly work without send_hsts_priming=true.

In FF52 in a normal window:
- the 2nd image loads with either send_hsts_priming;true OR (after clearing the cache again!) with send_hsts_priming;false after visiting the heise.de domain (which loads resources from https://1.f.ix.de which then sets the Hsts cache entry)

In a FF55nightly normal window:
- it requires both send_hsts_priming and use_hsts = true or it otherwise seems to ignore the HSTS cache entry even after visiting heise.de

I think the behavior in FF52 is how it should be.

Comment 5

[Dakota]

I forgot to mention that you only see the priming HEAD requests the first time after clearing the siteSettings because it keeps a cache.

Comment 6

[Dakota]

Maybe the changed behavior in FF55nightly compared to FF52 is the new implementation of isolating the HSTS cache that's been done as part of the TOR Uplift project (https://bugzilla.mozilla.org/show_bug.cgi?id=1323644). In my FF55nightly it doesn't even work with privacy.firstparty.isolate=false, but I'm not sure the implementation of isolation relies on that pref or is done no matter what. Idk what else to tell you. If not even the 3rd image loads for you something is definitely blocking it.

Comment 7

[Dakota]

Attachment: headers seen by Live HTTP Headers

After clearing everything and with 'security.mixed_content.send_hsts_priming;true' + 'security.mixed_content.use_hsts;true' these are the headers seen by Live HTTP Headers

Comment 8

[Dakota]

Attachment: network listing for the same site load

these are the requests listed in Network Monitor for the same page-load seen in the headers.txt attachment