1685862 - Cannot download HTTP resources in background script in HTTPS-Only Mode

Status: NEW

(Core :: DOM: Security, defect, P3)

Description

[dw-dev]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0

Steps to reproduce:

I am the developer of the Save Page WE extension. When saving an HTTP page, for which HTTP-Only Mode has been temporarily turned off, Save Page WE is unable to download the resources used by that page. This appears to be because the resources are downloaded by the background script.

Steps to reproduce:

1. Start Firefox.
2. Install Save Page WE.
3. Turn on HTTP-Only Mode.
4. Navigate to an HTTP page and temporarily turn off HTTP-Only Mode.
5. Save the page using Save Page WE.
6. View the saved page - the resources (images, etc) are missing.

Actual results:

The resources used by the HTTP page are not saved, because thay could not be downloaded by Save Page WE.

Expected results:

The resources should be downloaded, because HTTP-Only Mode has been temporarily turned off.

Comment 1

[dw-dev]

Just noticed that in several places "HTTP-Only Mode" should read "HTTPS-Only Mode".

Comment 2

[Tomislav Jovanovic :zombie]

Christoph, how do you think HTTPS only mode should interact with extension background pages? Should we ignore it for extension, or do we need an UI to turn it off for specific extensions?

Comment 3

[Christoph Kerschbaumer [:ckerschb]]

(In reply to Tomislav Jovanovic :zombie from comment #2)

Christoph, how do you think HTTPS only mode should interact with extension background pages? Should we ignore it for extension, or do we need an UI to turn it off for specific extensions?

In my opinion this is a plain bug. If https-only got disabled for the page then it should also be disabled for the extension.

Currently if the top-level document got exempt from https-only, then we set the flag HTTPS_ONLY_EXEMPT on the loadinfo for any resource loads. So I assume we are missing that somewhere for extension triggered loads.

Shane, is there one common place in our extension code where we create new channels? I could imagine we are missing to propagate the exempt flag there.

Comment 4

[Tomislav Jovanovic :zombie]

I'm not sure this is a simple bug, it seems there are a number of corner cases here.

(In reply to Christoph Kerschbaumer [:ckerschb] from comment #3)

Currently if the top-level document got exempt from https-only, then we set the flag HTTPS_ONLY_EXEMPT on the loadinfo for any resource loads. So I assume we are missing that somewhere for extension triggered loads.

So the exempt flag is set for the document in the tab, but extension is probably doing loads from their background page.

Background page is not visible in browser's UI, so there's no way for users to grant an exemption to extensions.

Also, lots of extensions have iframes that load (potentially) HTTP pages, and it seems there's no way to exempt those either. Clicking "Continue to HTTP site" in the iframe doesn't work, and there's nothing in the page info panel to exempt for the whole page.

Additionally, we don't expose the exemption status to extensions, or the HTTPS_ONLY status in the first place, so developers can't know what pages/domains are valid via http.

Comment 5

[Christoph Kerschbaumer [:ckerschb]]

As discussed on slack, there are a variety of things we should expore, some are more heavy weight than others:

• Introduce UI for extensions, heavy weight
• exempt all extensions from https-only, semi-optimal, because generally we want to move the web to https
• expose some signal to extensions if a page has been exempted, requires updates from extensions

Let's discuss more of those options in our weekly call early next week.

Comment 6

[Rob Wu [:robwu]]

Whick weekly call? If it doesn't conflict with an existing meeting I'd be interested in joining.

"exempt all extensions from https-only" doesn't look like an ideal long-term solution, because it breaks the user's expectation about their activity being https-only.

Comment 7

[erosman]

I was also asked about this issue in relation to user-scripts.

Situation:

HTTPS-Only Mode enabled and on a HTTP page that user has given temporary permission to

• HTTP XHR made from user-script using Web API fetch/XHR goes through
• HTTP XHR made using GM API and made from extension's background page fails

Possible Solutions:

• HTTP Temporary permission to be applied globally
• Allow extension background to bypass HOM .e.g. via a flag in fetch/XHR
• Create a permission to bypass HOM in manifest.json to allow extension background to bypass HOM

There are other situation where extensions that use HTTP for any other reason for their own functions will fail to function when HOM is enabled.

Comment 8

[Julian Gaibler]

Hi!

HTTPS-Only is rather straightforward I think: Wherever a new request gets created (and with it a new LoadInfo) we need to check if the principal is exempt by checking in with the SitePermissions API (code examples below).

If the permission says that the site is exempt, we need to set the HTTPS_ONLY_EXEMPT-bitflag on the httpsOnlyStatus field in the LoadInfo-object. The necko-code always calls ShouldUpgradeRequest() for each request, to see if it should get upgraded to HTTPS, which always returns false when HTTPS_ONLY_EXEMPT is set.

• Code example how to check if site is exempt in Javascript
• Code example how to exempt request in Javascript
• Function to check if site is exempt in C++
• Code example how to exempt request in C++

For regular top-level requests we set the exemption flag here and the flag gets propagated to subresources here.

I think that covers it. If I forgot something or it's unclear, feel free to ask :)

Comment 9

[Rob Wu [:robwu]]

nsHTTPSOnlyUtils::ShouldUpgradeRequest uses the triggering principal to decide whether or not exempt the request, provided that it's not the system principal.

Most downloads through the "downloads" extension API (here) (and also non-extensions, here) use the system principal as the triggeringPrincipal. For http(s) requests through the downloads API, the loadingPrincipal is set to the moz-extension:-principal.

This bug can be resolved in multiple ways, e.g.:

• Change nsHTTPSOnlyUtils::ShouldUpgradeRequest to also exempt requests if the loading principal is exempt (plus exempt (specific) moz-extension: principals, e.g. via (optional) extension permissions).
• Change NetUtil.newChannel (or one of its callees, or some/all of its callers), such that a "good" default is chosen for the new channel's loadInfo.httpsOnlyStatus member.

Both options (especially the second) affect way more than just the downloads API of extensions. This is probably desirable, but I don't have enough context to judge whether it really is.

Julian, which direction would you prefer?

Comment 10

[Julian Gaibler]

Thanks for providing these links! I see why we can't use the triggeringPrincipal or loadingPrincipal.

I think the best solution would be to create a new content principal based on download.source.url and give that to the permission manager. Then we could probably exempt the channel at the two codepoints you linked. Maybe here and here after the private-browsing check.

Code would maybe looks something like this:

// Create new content principal based on download URL
principal = Services.scriptSecurityManager.createContentPrincipal(
  download.source.url,
  loadingPrincipal.originAttributes
);
// Get permission state
const { state } = SitePermissions.getForPrincipal(
  principal,
  "https-only-load-insecure"
);
// If permission exempt, set httpsOnlyStatus to exempt aswell
if (state === Ci.nsIHttpsOnlyModePermission.LOAD_INSECURE_ALLOW_SESSION ||
	state === Ci.nsIHttpsOnlyModePermission.LOAD_INSECURE_ALLOW) {
  channel.loadInfo.httpsOnlyStatus |= Ci.nsILoadInfo.HTTPS_ONLY_EXEMPT;
}

That should fix the most important issue for now, and as we discussed we can try to improve it later when HTTPS-Only gains more traction :)

Comment 11

[Rob Wu [:robwu]]

With comment 10, all (http) downloads whose URLs are exempted will be downloaded over http, even for downloads not triggered by extensions. Is that really desirable?

And what is the expected behavior when redirects are involved (and if this differs from the currently implemented behavior, what's that)?
Based on the observation that subresource requests inherit the exemption from the document, I expect that the same would apply to downloads? I.e. the exemption is based on the initial URL (in the case of https://exempted.example.com/redir?url=http://not-exempted-but-inherit.example.com/dest, the http://not-exempted-but-inherit.example.com/dest request would NOT be upgraded due to the exemption).

Note: this would not fix the issues pointed out in comment 7. We discussed it last monday but didn't get to a resolution yet.

Comment 12

[Julian Gaibler]

Hi, sorry it took a while to respond!

all (http) downloads whose URLs are exempted will be downloaded over HTTP

That is what we want! I think we actually have a bug open because that's not happening right now.

And what is the expected behavior when redirects are involved

I'm pretty sure any redirects are handled by the HTTPS-Only Code within necko, so that shouldn't be a problem either.

Based on the observation that subresource requests inherit the exemption from the document, I expect that the same would apply to downloads?

You're right, we're not fixing the inheritance issue with this. But it seems there is no easy fix for that insight.

So I think we should do the simple fix here and then think long-term about how to address the other (valid issues). In our meeting, you talked about a general extension API proposal, where extensions can work within the security context of a page. How far along is that?

Comment 13

[Rob Wu [:robwu]]

(In reply to Julian Gaibler from comment #12)

all (http) downloads whose URLs are exempted will be downloaded over HTTP

That is what we want!

Just to double-check: If someone has whitelisted http://example.com, then any website can trigger a http:-request by triggering a download from a web page (e.g. <a download>). Is the ability for any web page to trigger http:-requests really what you want?

I think we actually have a bug open because that's not happening right now.

I can't find it, were you thinking of the already-fixed bug 1662359 ? Could you link the bug so I can get more context?

And what is the expected behavior when redirects are involved

I'm pretty sure any redirects are handled by the HTTPS-Only Code within necko, so that shouldn't be a problem either.

I searched for httpsOnlyStatus and HTTPS_ONLY_EXEMPT in Searchfox, but can't find that bit. To clarify, do you mean that the httpsOnlyStatus is preserved across redirects, or that the flag is cleared upon redirect? Based on your next reply below, I think that you mean that the httpsOnlyStatus flag with the exemption is preserved across redirects.

Based on the observation that subresource requests inherit the exemption from the document, I expect that the same would apply to downloads?

You're right, we're not fixing the inheritance issue with this. But it seems there is no easy fix for that insight.

So I think we should do the simple fix here and then think long-term about how to address the other (valid issues). In our meeting, you talked about a general extension API proposal, where extensions can work within the security context of a page. How far along is that?

No progress on that, bug 1670278 tracks this.

If everything written above looks as expected, then fixing this issue should be straightforward.

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::DOM: Security' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.