Closed Bug 1875378 Opened 1 year ago Closed 1 year ago

Refresh on HTTP page tries to access HTTPS version

Categories

(Core :: DOM: Security, defect)

Product:
Core
Component:
DOM: Security
Version:
Firefox 121
Type:
defect

Tracking

()

Status:
RESOLVED INCOMPLETE

People

(Reporter: mjurgens, Unassigned)

Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0

Steps to reproduce:

  1. Set up an internal HTTP only site (no HTTPS available - not even listening on the port).
  2. Access the site eg http://test.domain/test
  3. Once I manage to get that to load - press F5 to reload page

Actual results:

At Step 1: Internal test site fully operational
At Step 2: Firefox tries to load https:// version of site but fails, eventually I can get it to load, normally by clearing the cache - this is really annoying
At Step 3: Firefox tries to load https:// version of site but fails since there is no https:// version

Expected results:

At Step 2: Firefox should just load the http:// version of the page. Look I understand if there is an https:// version available, it should probably load in preference, but I don't have such a thing, nor do I want it for certain stages of testing
At Step 3: Firefox should just load the http:// version of the page. I've already forced it to load the http:// version and it worked successfully but a refresh should just reload what was already open and not try anything else.

This bug started as a Refresh problem but maybe the problem is that when an https:// site is not available, just use what you were originally told to use

I'm not sure when the refresh behaviour changed but this used to work fine (ie just reloaded what it already had open and did not try to https) maybe a couple of versions ago

When on the http:// version of the page, if you click any link, and let's say it loads ok, if you then click "back", we have the same problem - there was a successful http:// load but Firefox still tries to load https:// and fails.

I am now having great difficulty testing with Firefox. I hate to say it but Chrome works just fine. I hate having more reasons to move to Chrome

The Bugbug bot thinks this bug should belong to the 'Core::Networking: HTTP' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Networking: HTTP
Product: Firefox → Core

Thanks for the report. I think this is likely due to some https-first mode. I'm moving to Core :: Dom: Security component. If this was in error, please move back to Networking.

Component: Networking: HTTP → DOM: Security

Sorry, but I can't reproduce on a local HTTP site here.
Is this in a private browsing window?
Did you enable HTTPS-Only mode or something similar?

Flags: needinfo?(mjurgens)

Given the comments above, I tried on a fresh Firefox profile (Linux and Windows). It worked fine and as expected. I then tried again on the machine that had the problem that made me report this bug in the first place. I can no longer reproduce the problem (any of it). I have not idea what happened - it did just suddenly go bad, then I wrote up the bug report. Now it looks like, just as suddenly, it has started working again.

I guess you can close this bug

Flags: needinfo?(mjurgens)

Just for completeness - this was not in Private Browsing mode. I had not enabled any HTTPS-Only mode (intentionally) - my profile is really old and has been through many upgrades so not sure if that was a contributing factor

Although, now, I am getting it sometimes when I go back a page by hitting the back button. I will have to work out the steps to reproduce

There seems to be some time-related factor here.
If I use an http tab and then leave it for a bit eg 1 hour, sometimes less, then when I go back to the page and refresh it, it fails since it tries the https version and gives up. With Firefox startup restoring the last open tabs, sometimes and http tab that was open get reopened only to the https version and fails.

I have not yet been able to identify the exact steps or timeframes to reproduce.

I have reproduced it in Safe mode (so no addons causing the problem).

Load page over HTTP. Leave tab alone for 1 hour. Restart browser. Tab tries to reload in HTTPS and fails

There are a bunch of different reasons Firefox might try to upgrade to https. One of the first things we're going to have to do is figure out which mechanism is kicking in here.

I'm sure it's not HTTPS-Only: that would consistently load https, clearing your cache would have no effect, and would give clear error messages with an option to try the "http:" version. But we can quickly be sure: open preferences (about:preferences) and type HTTP-Only in the search box at the top.

It's not "HTTPS-First" because that would fall back to trying http: if the https: site didn't load. "http-first mode" is on by default in Private Browsing which is why Freddy asked about that, but there aren't any controls for it in settings (yet). It could be worth checking to make sure you're not in permanent implied private browsing. Back in the Privacy tab of our preferences page you looked at above you can look for the "History" section. By default it should say "Firefox will [Remember History]". If the button has been changed to read "Firefox will [Never remember history]" then you are always in Private Browsing even if we don't show the little icon. I'm pretty sure you're not, but it's a quick check to be sure.

There are a bunch of web features that trigger switching to https under various circumstances. All of those should cause the same behavior in other browsers like Chrome or Safari, though. Have you tried other browsers?

Does this happen on ALL sites that only have an http: version? For example, what about http://scripting.com/2023/10/22/012515.html ?

Does it only happen on that one single "test.domain" site? more than one site, but all internal? If so, anything in common?

How—exactly—are you clearing the cache? Some methods clear only the HTTP cache, and others also clear out some additional site-related data and metadata.

Why did you specify "internal" site? are you loading this using a private IP address like http://192.168.1.104/ ? Or does it use a name that you resolve through DNS?

If it's named, is it a single label ("testmachine") or does it literally have a domain part like your fake example "test.domain"?

If there's a domain part, are there a lot of internal sites that share the ".domain" part? If so do you visit them? Is it possible that you've visited one of them between when your test site was working over http: and the "later" when you refresh it and it breaks?

You were vague in your description of the affected machines and my theory might be ruled out by your answers to the above questions, but my current suspicion is that you are suffering from Strict-Transport-Security.

  • assuming the affected machine had a name like http://test1.devlab.mycompany.co.nz
  • you visit some other *.devlab.mycompany.co.nz or *.mycompany.co.nz site and it sends a strict-transport-security header with the includesubdomains attribute
  • now Firefox will upgrade any attempt to load the http: version of any subdomain of that site
  • whatever you're doing to "clear the cache" is clearing this setting also, which lets the site work again over http

When this happens and it tries to load the non-existent https: page, what does the error message say? If I'm correct the error page ought to mention Strict Transport Security somewhere. Be sure to press the "Advanced" button to get the detailed error description and code.

Flags: needinfo?(mjurgens)

There are a lot of questions there. I will try and cover them all.

  • https-only mode: not enabled
  • Firefox will [Remember History]: true
  • other browsers: I have briefly. I need to spend more time doing this. However, with Firefox it seems to only affect one computer. More on that below after covering all the questions
  • multiple sites: I have only tried one site test.domain. I will try others now.
  • cache clear method: https://addons.mozilla.org/en-US/firefox/addon/empty-cache-button/
  • internal site: using a full domain eg test.domain, that is resolved by my local dns
  • multiple internal sites: no, only one
  • other machines: only one machine seems affected
  • hsts: not mentioned in error message. Error message is "Unable to connect. An error occurred during a connection to test.domain". I can reproduce the error message any time just by trying to go to http://test.domain (since there is nothing listening on port 443)

One other thing I did was to try this on another machine. The other machine does not seem to exhibit the same problem. So I have zipped up the Firefox profile from the busted machine and the working machine and deployed them elsewhere. This testing is not complete but during this process I did this on the busted machine:

  • Firefox working ok on http://test.comain
  • Closed Firefox
  • Renamed %appdata%\Mozilla to %appdata%\Mozilla.busted
  • Put the known working profile in its place.
  • did other stuff for most of the day, could not replicate the issue
  • started building this message, realised I needed to check some settings on the busted profile
  • Renamed %appdata%\Mozilla to %appdata%\Mozilla.ok
  • Renamed %appdata%\Mozilla.busted to %appdata%\Mozilla
  • Started Firefox
  • Went to about:preferences to check stuff
  • I have session restore enabled and so I clicked on the tab that should be http://test.domain and it tried to load https://test.domain and failed

Wow. I did almost zero on that busted profile except wait for many hours.

I will be trying a few other combinations to try and work out if this is only on the one profile and only on Firefox and if time is the only factor in reproducing it.

I know this happening only on one profile would maybe just be - "oh don't worry about this and just use that profile that works ok" - but obviously something can go wrong and maybe it is worth it?

Flags: needinfo?(mjurgens)

I know this happening only on one profile would maybe just be - "oh don't worry about this and just use that profile that works ok" - but obviously something can go wrong and maybe it is worth it?

agreed, would be nice to make sure there's not something really broken here, at least to some level of time investment.

Your answers didn't clear things up as I'd hoped. If it's only happening with one site that might have something to do with the site configuration, but then why doesn't it happen to other profiles? If it's something particular to your browser configuration why isn't it affecting other sites?

Did you check to see if the problem happens with http://scripting.com ?

The add-on is using the browsingData.removeCache() API to clear the cache, which internally uses the CLEAR_ALL_CACHES definition of
CLEAR_NETWORK_CACHE | CLEAR_IMAGE_CACHE | CLEAR_CSS_CACHE | CLEAR_PREFLIGHT_CACHE | CLEAR_HSTS

That would support my theory that HSTS is involved, and it really does match your individual symptoms. But then we would expect every Firefox profile should behave the same, IFF you did the same kind of browsing in each of them. Since https://test.domain doesn't exist that can't be the source of the HSTS signal, but somewhere as you browse to other https://foo.domain site, one of them loads a resource from https://domain and THAT site sets HSTS with the includesubdomains attribute. Is it possible that when you tested the other browser/profile you didn't visit all the same internal sites?

To rule this out you can go to the test machine or profile that didn't have the problem and directly try to load https://domain, and then try loading http://test.domain again and see if the upgrade kicks in.

I don't know if this is the site in question, but going from your email I found https://edcint.co.nz/ does indeed send the header

strict-transport-security: max-age=63072000; includeSubDomains

Once you browse to that site then you won't be able to load any http://test.edcint.co.nz site until you clear the cache. Note that https://www.edcint.co.ns sends the same header, but that would only break http: sites under *.www.edcint.co.nz

If you try that kind of direct test on the machine that didn't have the problem, and still can't reproduce the problem there then what else could it be? Back in the problem profile what add-ons do you have installed? You can copy the list from the about:support page ("More Troubleshooting Information" on the Help menu).

Down a bit further in about:support you'll find a section titled "Important Modified Preferences". What are the differences between the "broken" profile and the one that doesn't have this problem?

Flags: needinfo?(mjurgens)

Once you browse to [https://edcint.co.nz] then you won't be able to load any http://test.edcint.co.nz site until you clear the cache.

That should be true in every Firefox profile and every other modern browser that supports the Strict Transport Security feature (e.g. Chrome, Edge, Safari, Brave, Opera, .... all of them).

Status: UNCONFIRMED → RESOLVED
Closed: 1 year ago
Resolution: --- → INCOMPLETE

Sorry, I should have updated this one.

I could reproduce the problem on any browser when going to [https://edcint.co.nz] then you trying to load [http://test.edcint.co.nz]

There is still one thing I have not figured out.

  • load [http://test.edcint.co.nz] succesfully
  • don't touch the browser for a period of time
  • refresh the page
  • page does not load as it tries to go to https://

Anyway, let's leave it there for now

Flags: needinfo?(mjurgens)
You need to log in before you can comment on or make changes to this bug.