1901120 - Fenix doesn't set `schemelessInput` in load info correctly

Status: RESOLVED FIXED

(GeckoView :: General, defect, P3)

Description

[Randell Jesup [:jesup] (needinfo me)]

See https://bugzilla.mozilla.org/show_bug.cgi?id=1813177#c2
Works on Chrome and FF Desktop

Comment 1

[Daniel Holbert [:dholbert] (PTO until 2/24)]

This is still reproducible, though it's been sort of papered-over in Nightly with Bug 1719271's pref-flip.

To test this bug, you need to be testing in a fresh profile (where you've never before visited the https version of this site)[1], and you need to disable the pref dom.security.https_first. With that pref disabled in a fresh profile with today's Android Nightly, if I type "rbi.org.in" into my URLbar and submit it, I end up at an Azure error page (the same one that I see from curl -vvv http://rbi.org.in).

Whereas if I perform the same process on Desktop (fresh profile, disable the pref, type rbi.org.in), I end up at the https version of the site.

However: I can get rbi.org.in to give me the same azure error page if I additionally disable the pref dom.security.https_first_schemeless. So it looks like maybe the bug here is that dom.security.https_first_schemeless was being ignored on Android, at least in this case?

[1] if/when I explicitly visit the https version of the site, we seem to remember that and automatically redirect http to https from that point on (even with the https_first pref turned off)

Comment 2

[Daniel Holbert [:dholbert] (PTO until 2/24)]

Here's a profile I just captured of this in today's Android nightly (with dom.security.https_first disabled): https://share.firefox.dev/47YBqah

The network track shows that we make a connection to http://rbi.org.in/ despite the fact that I typed rbi.org.in into my URLbar (schemeless) which should make it automatically choose https as the protocol.

Comment 3

[Daniel Holbert [:dholbert] (PTO until 2/24)]

SIMPLER STR (with less site-specific weirdness involved):

1. Start Fenix, ideally with a fresh profile (not sure a fresh profile is actually necessary but it's helpful to reduce variability).
2. Visit about:config and toggle dom.security.https_first to false. (BUT, ensure that dom.security.https_first_schemeless is true)
3. Open a new tab and type example.org and submit.

ACTUAL RESULTS:
You end up at http://example.org (note HTTP, no S), indicated with a broken lock icon.

EXPECTED RESULTS:
You should end up at https://example.org (note HTTPS), indicated with a full lock icon.

REASONING:
dom.security.https_first_schemeless should cause your schemeless domain entry to produce a HTTPS-first-style request, which should give you the HTTPS page since that attempt succeeds.

Firefox on Desktop gives expected-results.

I'm marking this as depending on bug 1812192 where this https_first_schemeless feature was implemented, and adjusting the component accordingly. (Not sure if this involves necko code after all.)

Comment 4

[Daniel Holbert [:dholbert] (PTO until 2/24)]

Bit of a shot-in-the-dark, but I'm guessing this commit from bug 1812192...

https://hg.mozilla.org/integration/autoland/rev/cecb7eec7fa1
Address bar should mark navigations that were schemeless r=mak,Gijs,tabbrowser-reviewers

...needs some equivalent logic for the Fenix URLbar code, to set wasSchemelessInput as-needed for user-entered URLs.

Comment 5

[Malte Jürgens [:maltejur] (ooo until 2025-02-21)]

(In reply to Daniel Holbert [:dholbert] from comment #2)

The network track shows that we make a connection to http://rbi.org.in/ despite the fact that I typed rbi.org.in into my URLbar (schemeless) which should make it automatically choose https as the protocol.

This is more or less expected, as schemeless HTTPS-First was only intended to ship on Desktop. We should have probably documented that better.

(In reply to Daniel Holbert [:dholbert] from comment #4)

Bit of a shot-in-the-dark, but I'm guessing this commit from bug 1812192 [...] needs some equivalent logic for the Fenix URLbar code, to set wasSchemelessInput as-needed for user-entered URLs.

Yes, that sounds about right if we wanted to ship schemeless HTTPS-First on Android as well. But as the current plan is to ship full HTTPS-First (Bug 1921221), like was already done in Nightly, I am not sure if we should focus our efforts on bringing schemeless HTTPS-First to Android first.

But on the other hand, we are currently also discussing using wasSchemelessInput to decide when we should not do a regular HTTPS-First upgrade (Bug 1919544). Considering that, I think we should actually do work on setting wasSchemelessInput correctly on Android.

I am not sure if this is something the Android team wants to work on. If not, I can also try to set up an Android environment and work on this, but that may take a bit longer than if somebody already experienced with the native android address bar code does it. Ah, I just saw that Sunil already has assigned this to himself.

Comment 6

[Simon Friedberger (:simonf)]

:pollymce, could you comment on how hard it would be to do this on Android? Since this is a front-end thing, is it something we could kick to your team?

Comment 7

[Polly [:polly]]

hello, thanks for reaching out!
dumb question, but do we have documentation for these about:config preferences anywhere?
I'd like to try and understand what a user might be trying to achieve by having dom.security.https_first = false but dom.security.https_first_schemeless = true, so that i am clear what the expected behaviour is here. It's not immediately obvious to me!

Comment 8

[Simon Friedberger (:simonf)]

https_first_schemeless only upgrades connections to sites which have been entered without a scheme http:// or https://. General https_first tries to upgrade connections no matter which scheme the user entered.

https_first_schemeless is currently already enabled in Desktop Fx. https_first only in Nightly.

Comment 9

[Polly [:polly]]

Thanks for the info! :raised_hands:


So, we don’t have about:config on the release build of android, it is only on the nightly.
i tested on android nightly and this is what i get.... I put “example.org” into the browser to see where i ended up.
(this doesn’t format as a table sadly!)

| https_first | https_first_schemeless | result url scheme |
| true | true | https |
| true | false | https |
| false | true | http |
| false | false | http |

So it looks like we get the expected behaviour if both https_first and https_first_schemeless agree.

If they are different - https_first overrides https_first_schemeless.
In this mixed-up scenario, it is not clear to me from the about:config page which of these settings would “win”.
If there are no docs to lead a user to expect which setting would override the other, I don’t see a clear rule that we need to implement here.
But please let me know if i am missing something!

Comment 10

[Malte Jürgens [:maltejur] (ooo until 2025-02-21)]

I think there still is some confusion here. The two prefs in question are documented very briefly here. What HTTPS-First does is try to upgrade all top level loads, whether they originate from the address bar, or from a user clicking on a link on a page. "Schemeless HTTPS-First" will only try to upgrade address bar inputs, and only if they do not contain an explicit scheme. For example, with schemeless, entering example.com should get upgraded, but http://example.com shouldn't. So schemeless HTTPS-First is a subset of HTTPS-First, enabling HTTPS-First only in some cases. This also means if https_first is enabled, https_first_schemeless doesn't matter, because HTTPS-First already upgrades all top-level loads.

The problem we are trying to fix now is that https_first_schemeless was only implemented on Desktop, which means it currently does nothing on Android. The missing piece for it to also work on Android is the information whether a load from the address bar was started with an input with or without a scheme (wasSchemelessInput).

I'd like to try and understand what a user might be trying to achieve by having dom.security.https_first = false but dom.security.https_first_schemeless = true, so that i am clear what the expected behaviour is here.

Also to clear up things here, these prefs are not meant to be set by users through some UI. They are mostly just there to support different feature configurations during our full HTTPS-First rollout. Once HTTPS-First is enabled everywhere, we may even consider removing the https_first_schemeless pref again, because it was mainly meant to partially roll out HTTPS-First. But it would still be really useful to correctly set wasSchemelessInput, and in turn make https_first_schemeless work on Android, because as per my comment, we probably also want to use the information if an address bar input contained a scheme for the main HTTPS-First in the future.

Comment 11

[Pablo Gonzalez Reyes]

(In reply to Malte Jürgens [:maltejur] from comment #5)

(In reply to Daniel Holbert [:dholbert] from comment #2)

The network track shows that we make a connection to http://rbi.org.in/ despite the fact that I typed rbi.org.in into my URLbar (schemeless) which should make it automatically choose https as the protocol.

This is more or less expected, as schemeless HTTPS-First was only intended to ship on Desktop. We should have probably documented that better.

(In reply to Daniel Holbert [:dholbert] from comment #4)

Bit of a shot-in-the-dark, but I'm guessing this commit from bug 1812192 [...] needs some equivalent logic for the Fenix URLbar code, to set wasSchemelessInput as-needed for user-entered URLs.

Yes, that sounds about right if we wanted to ship schemeless HTTPS-First on Android as well. But as the current plan is to ship full HTTPS-First (Bug 1921221), like was already done in Nightly, I am not sure if we should focus our efforts on bringing schemeless HTTPS-First to Android first.

But on the other hand, we are currently also discussing using wasSchemelessInput to decide when we should not do a regular HTTPS-First upgrade (Bug 1919544). Considering that, I think we should actually do work on setting wasSchemelessInput correctly on Android.

• I am not sure if this is something the Android team wants to work on. If not, I can also try to set up an Android environment and work on this, but that may take a bit longer than if somebody already experienced with the native android address bar code does it. Ah, I just saw that Sunil already has assigned this to himself.

Comment 12

[Simon Friedberger (:simonf)]

:pollymce so to come back to this, the question would be, how difficult is it to implement "this" on Android, where "this" is "detecting that the users entered a URL without a scheme and putting that information in the LoadInfo so the network code can react to it".

Comment 13

[Polly [:polly]]

i think we would need to understand what is being asked for in a bit more detail, i am still a bit confused, sorry... just feel like i am missing a lot of context.
probably the best way forward for this is to fill in an intake request. instructions here, but basically there is a jira ticket template to fill in which asks a few more questions - the aim being to shape the work and fully understand the requirements and drivers behind it, so we can plan it in appropriately.

Comment 14

[Daniel Holbert [:dholbert] (PTO until 2/24)]

(In reply to Polly McEldowney [:pollymce] from comment #13)

i think we would need to understand what is being asked for in a bit more detail, i am still a bit confused, sorry... just feel like i am missing a lot of context.

Adding a bit more details here:
(1) When a user just types example.org into the Fenix location bar and hits enter, the location-bar code fixes up that string to add the "http://" scheme, and asks Gecko to load the resulting URL http://example.org. (The user's input was schemeless (no http or https), and the location-bar code adds the "http" scheme automatically in order to make it a valid URL for Gecko to load. It uses insecure "http" for historical reasons.)
(2) There's now a flag that you can pass (added in bug 1812192) that Fenix can use in this circumstance to tell Gecko that the user's actual input was schemeless (i.e. that the location-bar code added the http scheme automatically).
(3) Fenix is not currently passing that flag (which is understandable because that flag was only ~recently added), but ideally it should pass that flag. Hence this bug.
(4) The benefit here is that connections will be secure-by-default more often, and will just-work-by-default more often too (in cases where a site happens to have a broken insecure-http configuration, like rbi.org.in (per this bug) and nih.gov (per bug 1868167)

Let me know if that makes sense or if clarification would help on any part there. (Happy to chat over slack too if that's easier.)

probably the best way forward for this is to fill in an intake request. instructions here, but basically there is a jira ticket template to fill in which asks a few more questions - the aim being to shape the work and fully understand the requirements and drivers behind it, so we can plan it in appropriately.

Do you actually want folks to use that intake form for relatively-small bugs? Clicking through it, it looks like it's for substantial projects and new-feature-work; whereas this bug here is likely pretty small/targeted. We likely just need to have someone who knows Fenix's location-bar code take a look at the Firefox-for-desktop patch https://hg.mozilla.org/mozilla-central/rev/cecb7eec7fa1 (from Bug 1812192) and make the equivalent changes to Fenix's location-bar code.

Comment 15

[Daniel Holbert [:dholbert] (PTO until 2/24)]

[also/alternately it looks like malte was open to working on this in comment 5, but didn't have an Android dev setup and it looked at that point like this bug had been assigned. At this point I'm not sure whether it's easier for someone like malte working on DOM::Security code to get up-to-speed on the android build process and location-bar code, vs. someone working on Android location-bar code to get up-to-speed on this wasSchemelessInput flag so they can pass it in the right situations. Probably a bit of a hurdle in either direction, and it's just a question of who's got the relevant cycles. :) But one way or another, there are a few associated webcompat bugs that'll benefit from having this fixed, so we should ideally get it resourced.]

Comment 16

[Daniel Holbert [:dholbert] (PTO until 2/24)]

(In reply to Daniel Holbert [:dholbert] from comment #14)>

Do you actually want folks to use that intake form for relatively-small bugs? Clicking through it, it looks like it's for substantial projects and new-feature-work; whereas this bug here is likely pretty small/targeted.

I guess I see that template mentions t-shirt sizing that includes small-sized projects, so I guess it's worth doing. I went ahead and filed https://mozilla-hub.atlassian.net/browse/FXMO-501 if that helps here.

Comment 17

[Polly [:polly]]

thanks for filling in the ticket! i think the idea of the intake form is for planning and prioritisation as well as gathering all the details of the work, so definitely helpful for the team to have that.

Comment 18

[Malte Jürgens [:maltejur] (ooo until 2025-02-21)]

Thanks Daniel for staying on this and filing the Jira ticket.

One small addition to

(2) There's now a flag that you can pass (added in bug 1812192) that Fenix can use in this circumstance to tell Gecko that the user's actual input was schemeless (i.e. that the location-bar code added the http scheme automatically).

The format of what was the wasSchemelessInput flag has slightly changed in Bug 1919544. Instead of a bool, it is now an enum just called schemelessInput. That enum has the possible values "unknown / this load isn't coming from the address bar", "url had a scheme" and "url didn't have a scheme". That change was needed, as Bug 1919544 is now also using the same flag for determining whether a load came from a *schemefull* address bar load, and !wasSchemelessInput alone could mean both the load isn't coming from the address bar, and that it isn't schemefull.

Also, I will be adding similar changes to correctly set that enum for loads from a URL passed as a command line argument on desktop in Bug 1937852. Once I have a patch there, you can maybe use that as guidance on how to do the changes for Android.

(In reply to Daniel Holbert [:dholbert] from comment #15)

[also/alternately it looks like malte was open to working on this in comment 5, but didn't have an Android dev setup and it looked at that point like this bug had been assigned. At this point I'm not sure whether it's easier for someone like malte working on DOM::Security code to get up-to-speed on the android build process and location-bar code, vs. someone working on Android location-bar code to get up-to-speed on this wasSchemelessInput flag so they can pass it in the right situations. Probably a bit of a hurdle in either direction, and it's just a question of who's got the relevant cycles. :) But one way or another, there are a few associated webcompat bugs that'll benefit from having this fixed, so we should ideally get it resourced.]

Yes, me or somebody from my team working on this is also still an option. I was thinking that this may be a bit less work for the Android team than us, but if the Android team doesn't see it that way or doesn't have the resources right now, we can also take over working on this.

Comment 19

[Malte Jürgens [:maltejur] (ooo until 2025-02-21)]

Renaming the bug as it doesn't just cause dom.security.https_first_schemeless not having any effect, but also an inconsistent behavior on Android in regard to Bug 1919544. And as already shortly mentioned in comment #5, the latter is arguably even more important, as we soon plan to generally enable HTTPS-First. That would mean the less strong dom.security.https_first_schemeless wouldn't have an effect anymore anyhow.

Comment 20

[Malte Jürgens [:maltejur] (ooo until 2025-02-21)]

Judging from the fact that there still hasn't been any response on Jira, it looks like the Android team may not have many resources to work on this right now. We still want to fix this, ideally with the HTTPS-First release in 136, to make sure that we have a consistent behavior around explicit http:// schemes. Considering that, I will start working on this myself.

Comment 21

[Malte Jürgens [:maltejur] (ooo until 2025-02-21)]

Attachment: Bug 1901120 - Set schemelessInput correctly on Android r?simonf!,#android-reviewers!

Comment 22

[Matt Tighe [:matt-tighe]]

Hey there! Been taking a peek at this patch from an Android reviewer perspective, and I had a question about the overall API. I am happy to move the patch forward as is given the fact that we are looking to catch a train, but I was wondering:

Would it make sense to instead pass the raw address bar input down to Gecko, along with a client-side formatted version? So instead of passing the URI and the schemeless input boolean, we would pass the URI and something like originalInputString? It seems like we'd have more flexibility on the Gecko side for determining logic or introducing additional features down the road.

Comment 23

[Malte Jürgens [:maltejur] (ooo until 2025-02-21)]

(In reply to Matt Tighe [:matt-tighe] from comment #22)

Would it make sense to instead pass the raw address bar input down to Gecko, along with a client-side formatted version? So instead of passing the URI and the schemeless input boolean, we would pass the URI and something like originalInputString? It seems like we'd have more flexibility on the Gecko side for determining logic or introducing additional features down the road.

That does actually sound like a better way of passing through this information. I'll look into changing it to that, which I think I can do that rather quickly, as we are still passing the information through the same places.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b6cd2921f411
Set schemelessInput correctly on Android r=simonf,android-reviewers,geckoview-reviewers,m_kato,matt-tighe

Comment 25

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3eead22382a9
Backed out changeset b6cd2921f411 for causing android build bustages. CLOSED TREE

Comment 26

[agoloman]

Backed out for causing android build bustages.

• Backout link
• Push with failures
• Failure Log

Comment 27

[Malte Jürgens [:maltejur] (ooo until 2025-02-21)]

Looking into it, this build error doesn't happen locally or on the auto try chooser

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1f419782aefe
Set schemelessInput correctly on Android r=simonf,android-reviewers,geckoview-reviewers,m_kato,matt-tighe

Comment 29

[amarc]

Backed out for causing android build bustages

• Backout link
• Push with failures
• Failure Log 1
• Failure Log 2
• Failure Log 3
• Failure Log 4
• Failure Log 5

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/34b17ee9dd42
Set schemelessInput correctly on Android r=simonf,android-reviewers,geckoview-reviewers,m_kato,matt-tighe

Comment 31

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/34b17ee9dd42

Comment 32

[Simon Friedberger (:simonf)]

Attachment: Bug 1901120 - Set schemelessInput correctly on Android r?simonf!,#android-reviewers!

This passes through the information about wether the input in the address bar
got fixed up with a http:// scheme in the beginning. This information is used to
exempt a site from HTTPS-First upgrading if the user specified an explicit
http:// scheme.

Original Revision: https://phabricator.services.mozilla.com/D234324

Comment 33

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: Users don't have an option to prevent HTTPS-First upgrades
• Code covered by automated testing: no
• Fix verified in Nightly: yes
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: Visit upgradable.httpsonly.polar.onl and note that an HTTPS connection is made. Then visit http://upgradable.httpsonly.polar.onl and the connection should be HTTP.
• Risk associated with taking this patch: Low
• Explanation of risk level: This just passes some information from the URL bar down to the network code
• String changes made/needed: No
• Is Android affected?: yes

Comment 34

[Andrei Jolțan [:Andi]]

Hi,

Just wanted to let you know that the 2 UI tests failed 1x today.
I've logged 1948000 and 1948001 to keep track.

Comment 35

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/dee617a8e60a