1296999 - Location bar suggests bare HTTP URL for sites even if only HTTPS URLs are in history

Status: RESOLVED WORKSFORME

(Firefox :: Address Bar, defect)

Description

[Paul]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0
Build ID: 20160728203720

Steps to reproduce:

Let's say you visit https://www.facebook.com/messages so frequently it's the most visited Facebook URL in your history. Let's also say that you never visit http://*.facebook.com/*, and have no such entries in your history or bookmarks.

Well, start typing "facebook" in the location bar.


Actual results:

The top suggestion is http://facebook.com/, seemingly invented by Firefox.


Expected results:

The first suggestion should be the most commonly visited URL related to the string being entered, in this case https://www.facebook.com/messages. If someone decides that the root of the site should always come first, OK I'm not going to have that argument here but it should definitely be https://www.facebook.com/ and not http://facebook.com/, which is insecure and non-canonical. Foremost, we should never prefer HTTP!

Comment 1

[Paul]

The drop-down also suggests extremely unprofitable domains, e.g. try typing a 0 (zero), q, or some other uncommon character. The top suggestion will have that character at the start of the domain name, ignoring every common URL with that character in it that you actually visit.

Comment 2

[Marco Bonardo [:mak]]

(In reply to Paul from comment #0)
> The top suggestion is http://facebook.com/, seemingly invented by Firefox.

That seems unlikely, the only way for that to happen would be to type the non secure version, abut visit the secure version only through links.
What we do is reuse the most secure scheme out of every time the url was typed.
The top suggestion should also not show a scheme at all, may I see a screenshot of the issue please?

Comment 3

[Paul]

Attachment: Screenshot: After typing 'f'

Comment 4

[Paul]

Attachment: Screenshot: After typing 'f' and hitting Enter

Comment 5

[Paul]

Attachment: Screenshot: History has only attempts from today for http://facebook.com/

Comment 6

[Paul]

Attachment: Screenshot: After typing 'j'

Comment 7

[Paul]

Attachment: Screenshot: History has only 2 old visits to this URL

Comment 8

[Paul]

Attachment: about:config filtered for "urlbar"

Comment 9

[Paul]

@mak You're correct that the suggestion doesn't show a scheme. I thought this implied HTTP because it navigates to http://facebook.com/ ... however today it *sometimes* seemed to navigate directly to https://www.facebook.com/, but I may have just not noticed the redirect occurring.

I have provided six screenshots:
3 for f, f⏎, and http://facebook.com/ history
2 for j and related history (per comment 1)
1 showing urlbar settings in about:config

Note that there was no http://facebook.com/ when the first two screenshots were taken; I've been manually deleting it over the past week or two.

Comment 10

[Paul]

To clarify: no http://facebook.com/ in the browser's history when the first two screenshots were taken.

Comment 11

[Marco Bonardo [:mak]]

(In reply to Paul from comment #9)
> @mak You're correct that the suggestion doesn't show a scheme. I thought
> this implied HTTP because it navigates to http://facebook.com/ ...

No, what we show there may not be the url we visit. Even when we show "facebook.com" we may be requesting "https://facebook.com" instead. It's done mostly to avoid confusing the user (showing the same string in the textfield and in the first entry), but it's also true it doesn't really show the security level... Maybe we could append a security indicator to entries, though that may make users think the page is "safe" when it's just encrypted... Security indicators are a complicate matter.

> today it *sometimes* seemed to navigate directly to
> https://www.facebook.com/, but I may have just not noticed the redirect
> occurring.

Sounds like you may be willing to use a network logger to check what actually happens, relying on eyes doesn't work very well in these cases.
YOu can do that in Firefox, by enabling Chrome Debugging in developer tools, open browser toolbox, go to Network, and then start the visit.
Under HTML, select the first connection to the page and then the Request URL in the Headers Pane should tell you what we requested.

It seems strange to me that sometimes we visit https, sometimes http, since once https is typed once, it should always be used for any future request.
For example in my profile typing "f[ENTER]" always goes directly to "https://www.facebook.com/"

Comment 12

[Paul]

Unfortunately an Xorg crash took down my Firefox instance, and the new instance is refusing to misbehave. Which is interesting ... but I may not be able to reproduce the problem for a while. However you can refer to the Facebook screenshot attached to this bug for now.

Is there a way to prevent the Developer Toolbar in a fresh tab from closing when the first page loads? This only happens the first time in a given tab. Failing that I can just use tcpdump.

Comment 13

[Marco Bonardo [:mak]]

(In reply to Paul from comment #12)
> Is there a way to prevent the Developer Toolbar in a fresh tab from closing
> when the first page loads? This only happens the first time in a given tab.
> Failing that I can just use tcpdump.

I think Content devtools are per page, that's why I suggested using Browser devtools (Browser Toolbox).

Comment 14

[Paul]

Ah, I wasn't familiar with it. Configured it per https://developer.mozilla.org/en-US/docs/Tools/Browser_Toolbox and confirmed it can track network connections across tabs. Will see if I can reproduce the bug again after a few days, when the Firefox process's data structures are bloated and timing effects start to appear.

Comment 15

[Daniel Kabs, reporting bugs since 2002]

I think I see this behaviour, too. 

Using Firefox 50.0.2, I have accessed both URLs
- http://192.168.1.254/ and 
- https://192.168.1.254/ 
So both URLs are stored in my browser's history.

When I start entering "192" into the location bar, the http version of the URL is suggested first "192.168.1.254/ - Visit". So entering "192" and pressing return sends me to the http URL.

I don't like this at all. I agree that the https version of the URL should be preferred, always. 

However, I think this is already discussed in bug 902338 and bug 902582.

Comment 16

[Marco Bonardo [:mak]]

(In reply to Daniel Kabs, reporting bugs since 2002 from comment #15)
> I think I see this behaviour, too. 
> 
> Using Firefox 50.0.2, I have accessed both URLs
> - http://192.168.1.254/ and 
> - https://192.168.1.254/ 
> So both URLs are stored in my browser's history.

it doesn't matter that they are in history, but that they are "typed" urls.
Also don't look what is "suggested" in the first entry, since we strip the scheme, look at what we actually visit when you Enter.
can you please try to type the https url and confirm it (I actually think paste/enter would do, but I'm not totally sure) and then check with a network traffic logger (even firefox browser toolbox network would do) what we request to the server?

Comment 17

[Daniel Kabs, reporting bugs since 2002]

Marco, you are correct. I didn't know that the access method is not displayed in the first suggested entry ( "192.168.1.54/ -- Visit" ).

Further, after I deleted every "http:" entry pointing to 192.168.1.254 from my browser's history, entering "192." into the Location Bar and pressing return sends me to a secure page, i.e., a "https:" URL.

So I have to take back what I said. I don't see the behaviour as specified in the bug description.

Comment 18

[Paul]

@mak, I've just found how to reproduce part of the bug, by forcing the protocol:

1. Ensure that you have visits for https://www.facebook.com/ in your history, but not {http,https}://facebook.com/
2. Start typing http://face...
3. Firefox completes the URL to http://facebook.com/, *even though it's not in your history*

I can't reproduce the full bug on demand yet, but it still happens with some regularity despite the Firefox 50 URL bar fixes. By the way, I run Firefox with keyword.enabled set to false, not sure if that's relevant to the full bug but it isn't to this part.

Comment 19

[Marco Bonardo [:mak]]

(In reply to Paul from comment #18)
> 1. Ensure that you have visits for https://www.facebook.com/ in your
> history, but not {http,https}://facebook.com/
> 2. Start typing http://face...
> 3. Firefox completes the URL to http://facebook.com/, *even though it's not
> in your history*

that's not a bug, you typed http:// explicitly, we respect your choice.

Comment 20

[Paul]

I see your perspective, although this confirms that under some conditions Firefox invents FQDNs that are not in history.

I will continue to try to capture the HTTP part of the issue.

Comment 21

[Paul]

As of Firefox 53, this issue hasn't occurred in several months. Presumably it was resolved as part of the long tail of URL bar fixes.

Please feel free to close this ticket.

Comment 22

[Paul]

Attachment: Packet capture of a YouTube visit which exhibits the bug

OK, I can replicate the bug 100% now, as follows:

1. Remove any http://www.youtube.com/ entries from history
2. Open a new tab
3. In the location bar, type "yout", this is expanded to "youtube.com/"
4. Hit <Enter>

The browser initially navigates to http://www.youtube.com/, then to https://www.youtube.com/. Both of these URLs appear as the most recent two in the History window, as in previous screenshots.

I've attached a packet capture (following a clean restart of Firefox). It shows the browser performing a DNS lookup and immediately initiating an HTTPS connection with Google's servers, indicating that the failover to HTTPS occurs within the browser itself, not as a result of an HTTP redirect. (The browser does perform an HTTP OCSP request, but this is unlikely to be the cause of the problem.)

So there probably isn't a security issue, but the spurious HTTP entry shouldn't
be added to the history nonetheless. Thoughts?

Comment 23

[Marco Bonardo [:mak]]

We recently fixed bug 1341350 that will make autofill prefer https more often, before we were picking https only if ALL the visits were going to https, after that change we'll use an average. It's not the final change to this behavior, we still want to improve the algo in bug 1239708.
We also changed the location bar first entry to actually show which url will be visited, and not a cut version of it.
So starting from Firefox 55 we should prefer https more AND show https in the first autocomplete popup entry.

There are many moving parts here, included the locationbar, hsts cache in the browser (that at a network level will directly decide to go to https even if we request https), and server side redirects. Additionally in the middle there are add-ons and antivirus that could decide to rewrite a request.
I can't exclude a bug at one of these levels that will end up adding http to global history even if effectively we'll go straight to https, but considering the final target is correct and the only downside is an history entry, I don't think the cost of time investigating it will be balanced by the gain.

Thanks for reporting though, I'll keep this in mind and if it ends up being a widespread annoyance we could decide to reprioritize its investigation.

Comment 24

[Paul]

Oh thanks, that combination sounds like it may resolve this issue ... and if it doesn't we should see what's going on, right in the URL bar.

I don't care if my particular bug reports get fixed, I just want our browser to be awesome!