1618094 - address bar search term is interpreted as scheme when Firefox runs in an environment where flatpak claims to understand all protocols/schemes

Status: RESOLVED FIXED

(Firefox :: Shell Integration, defect, P3)

Description

[grmat]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0

Steps to reproduce:

Using the address bar for both search and direct address input (as is the default).

Searching for

"site:mozilla.org firefox"

as that's a common query for different search engines.

Actual results:

Firefox attempts to open the address "site:mozilla.org firefox", which results in a dialog to open the "site" with the default system handler for this scheme.

The search term is misinterpreted as scheme part of an URL. As soon as the query contains a colon and a character after it, the hint in the address bar changes from "search with x" to "visit q", resulting in the corresponding action.

I know that "//" are not necessarily part of an URI, but I think the space(s) in the search query should make it obvious that the user is not requesting an address directly.

Expected results:

The default search engine should be queried with the query string. I'm marking this as bug rather than feature request because it's a regression.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[Gingerbread Man]

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0
20200217142647

Works for me. I can only reproduce this with keyword.enabled set to false which is expected behavior.

Comment 3

[grmat]

keyword.enabled is set to true (default) in my config. Changing it doesn't have any effect on the described behaviour.

Maybe it's also worth noting that this happens on a fresh profile, and also in developer edition 74.0b7

Comment 4

[Marco Bonardo [:mak]]

WFM as well, tested in Both Firefox 73 and Nightly 75.
Does it happen only with "site:test test" or also with random things like "bacon:test test"?
Please attach a log from about:support.

Comment 5

[grmat]

Attachment: about:support

Comment 6

[grmat]

Attachment: video recording

attached the log from about:support, and also a small video showing the behaviour. Note that the hint changes to "Visit" instead of "Search". It happens as well with made-up things like "bacon" or "asdf".

Comment 7

[Marco Bonardo [:mak]]

Could you please enable Chrome Debugging from Firefox devtools options, then open the Browser Console (not the Web Console) and evaluate this code (For help see https://developer.mozilla.org/en-US/docs/Tools/Browser_Console)?

Services.uriFixup.getFixupURIInfo("site:test", Ci.nsIURIFixup.FIXUP_FLAG_FIX_SCHEME_TYPOS | Ci.nsIURIFixup.FIXUP_FLAG_ALLOW_KEYWORD_LOOKUP).keywordAsSent

Comment 8

[grmat]

The return value is an empty String ("")

Comment 9

[Marco Bonardo [:mak]]

Thank you. Here is the problem, on all of my systems, including Ubuntu, that call returns "site:test"; if it returns an empty string, it means URIFixup thinks this is a uri that should be resolved. For some reason this happens in your archlinux Firefox build.

Gijs, any idea?
Maybe a problem with the search service?

Comment 10

[:Gijs (out for now; he/him)]

Can you reproduce on a mozilla.org build instead of an archlinux build?

Comment 11

[grmat]

:Gijs yes, same behaviour.

Comment 12

[:Gijs (out for now; he/him)]

(In reply to grmat from comment #11)

:Gijs yes, same behaviour.

So on a clean profile, on a mozilla.org build, you're seeing different behaviour from Marco? (comment #4, comment #9)

This doesn't make a lot of sense.

Is the about:support you attached earlier complete? Some of the sections (Firefox Features, modified prefs) seem to be lacking entries I'd expect to see there.

Do you have any kind of enterprise/global preference settings installed/enabled that could be affecting Firefox's behaviour?

Comment 13

[:Gijs (out for now; he/him)]

Oh, no, hang on, I have an idea - in the browser console (comment #7), what is the result of this (note: code snippets may scroll, make sure to get everything when copy/pasting):

Cc["@mozilla.org/uriloader/external-protocol-service;1"].getService(Ci.nsIExternalProtocolService).externalProtocolHandlerExists("site")

?

This should be returning false. But I bet it returns true on your machine. If so, it'd be worth checking two more things:

1. if you get the same result for other stuff, like passing "gobbledygook" instead of "site"
2. what the output is for

Cc["@mozilla.org/uriloader/external-protocol-service;1"].getService(Ci.nsIExternalProtocolService).getApplicationDescription("site")

(e.g. if I run this on a mac, and use "mailto" instead of "site", it would say "Mail", ie Mail.app, which would be the default mailto handler)

Comment 14

[:Gijs (out for now; he/him)]

(In reply to :Gijs (he/him) from comment #13)

Oh, no, hang on, I have an idea - in the browser console (comment #7), what is the result of this (note: code snippets may scroll, make sure you get everything):

Cc["@mozilla.org/uriloader/external-protocol-service;1"].getService(Ci.nsIExternalProtocolService).externalProtocolHandlerExists("site")

?

This should be returning false. But I bet it returns true on your machine. If so, it'd be worth checking two more things:

1. if you get the same result for other stuff, like passing "gobbledygook" instead of "site"
2. what the output is for

Cc["@mozilla.org/uriloader/external-protocol-service;1"].getService(Ci.nsIExternalProtocolService).getApplicationDescription("site")

(e.g. if I run this on a mac, and use "mailto" instead of "site", it would say "Mail", ie Mail.app, which would be the default mailto handler)

Comment 15

[Marco Bonardo [:mak]]

(In reply to :Gijs (he/him) from comment #13)

1. if you get the same result for other stuff, like passing "gobbledygook" instead of "site"

I asked for that in comment 4, and it's still broken also for "bacon:"

It will still be interesting to get the output.

Comment 16

[grmat]

This should be returning false. But I bet it returns true on your machine.

Yes it does.

1. has been answered in comment 6, as :mak said.
2. output: "System Handler" (that's what it says in the download/open window as well.

Comment 17

[grmat]

The "system Handler" thing seemed suspicious to me. I found out, that the XDG desktop portal is causing this issue. I assumed that before and tested it but apparently setting GTK_USE_PORTAL to 0 or "" isn't enough, it has to be unset instead.

Comment 18

[grmat]

Sorry for the noise, but I think I wasn't clear enough: unsetting the env var GTK_USE_PORTAL before starting Firefox resolves the issue.

So does FF "support" the use of XDG desktop portal? It's used with flatpak setups and on KDE or other non-GNOME/GTK desktops to get a more consistent, native feel.

Comment 19

[:Gijs (out for now; he/him)]

(In reply to grmat from comment #18)

Sorry for the noise, but I think I wasn't clear enough: unsetting the env var GTK_USE_PORTAL before starting Firefox resolves the issue.

So does FF "support" the use of XDG desktop portal? It's used with flatpak setups and on KDE or other non-GNOME/GTK desktops to get a more consistent, native feel.

I don't know enough about XDG / GTK to be able to say anything about what we "support" in this respect. But it seems this usecase wasn't considered when the flatpak support was implemented in bug 1411579 and friends. https://searchfox.org/mozilla-central/rev/b2ccce862ef38d0d150fcac2b597f7f20091a0c7/toolkit/system/gnome/nsGIOService.cpp#420-427 is what is doing this - it returns a handler irrespective of the input provided.

Hopefully :jhorak or other folks who know about our GTK/flatpak support have ideas about what to do here.

Comment 20

[:Gijs (out for now; he/him)]

Mihai, I believe you know about our flatpak support based on activity in bug 1628004 (if not, please redirect to someone better-placed). Do you know what we can do about this? If we're going to evangelize people using the flatpak option and we break features like this, that would be unfortunate.

Comment 21

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to :Gijs (he/him) from comment #20)

Mihai, I believe you know about our flatpak support based on activity in bug 1628004 (if not, please redirect to someone better-placed). Do you know what we can do about this? If we're going to evangelize people using the flatpak option and we break features like this, that would be unfortunate.

Thanks for raising this. I've got good knowledge on the release automation pipeline, less so with the under-the-hood flatpak; let me poke around some people to find some answers.

Keeping the NI open until then.

Comment 22

[Mihai Tabara [:mtabara]⌚️GMT]

@stransky hey! do you know who could help with this? As far as I can tell, this is not necessarily a flatpak thing, but more likely the portal that flatpak uses under-the-hood?

Comment 23

[Jan Horak [:jhorak]]

It looks like you have the application with which handles 'site' scheme installed. It's the same situation with the magnet uri: https://en.wikipedia.org/wiki/Magnet_URI_scheme. The magnet uri does not have // divider.

Please check the grep -r x-scheme-handler/site /usr/share/applications or xdg-mime query default x-scheme-handler/site.
You can also try to run xdg-mime query in the flatpak environment, ie. after running flatpak run --command=sh org.mozilla.Firefox Do the list contains any application?

Comment 24

[Jan Horak [:jhorak]]

Btw what we're dealing with is this:
https://searchfox.org/mozilla-central/source/uriloader/exthandler/unix/nsOSHelperAppService.cpp#1024
https://searchfox.org/mozilla-central/source/uriloader/exthandler/unix/nsGNOMERegistry.cpp#12
I think the external uri handlers can be disabled in the about:config by setting network.protocol-handler.external-default to false, but it's just a workaround.

Comment 25

[:Gijs (out for now; he/him)]

(In reply to Jan Horak [:jhorak] from comment #23)

It looks like you have the application with which handles 'site' scheme installed. It's the same situation with the magnet uri: https://en.wikipedia.org/wiki/Magnet_URI_scheme. The magnet uri does not have // divider.

Please check the grep -r x-scheme-handler/site /usr/share/applications or xdg-mime query default x-scheme-handler/site.
You can also try to run xdg-mime query in the flatpak environment, ie. after running flatpak run --command=sh org.mozilla.Firefox Do the list contains any application?

In comment 4 / comment 6, it was already established this is happening for all protocols, even "bacon" which is clearly bogus...

(In reply to Jan Horak [:jhorak] from comment #24)

https://searchfox.org/mozilla-central/source/uriloader/exthandler/unix/nsGNOMERegistry.cpp#12

This clearly asks the gioservice, which has https://searchfox.org/mozilla-central/rev/b2ccce862ef38d0d150fcac2b597f7f20091a0c7/toolkit/system/gnome/nsGIOService.cpp#420-427 , per comment #19, which always says "yes, we have a handler", for anything, when running in a flatpak environment...

Comment 26

[Martin Stránský [:stransky] (ni? me)]

Thanks for the hints Gijs, so it's clearly bug at nsGIOService::GetAppForURIScheme() for flatpak.

Comment 27

[grmat]

(In reply to Jan Horak [:jhorak] from comment #23)

Please check the grep -r x-scheme-handler/site /usr/share/applications or xdg-mime query default x-scheme-handler/site.

No result.

You can also try to run xdg-mime query in the flatpak environment, ie. after running flatpak run --command=sh org.mozilla.Firefox Do the list contains any application?

I don't have flatpak installed.

To clarify again:

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #22)

As far as I can tell, this is not necessarily a flatpak thing, but more likely the portal that flatpak uses under-the-hood?

Yes. I haven't used Flatpak versions of Firefox or even flatpak. This issue seems to be caused by the use of the xdg-desktop-portal, which is also (but not exclusively) used by flatpak apps. AFAIU support for xdg-desktop-portal was introduced to FF because of Flatpak.

xdg-desktop-portal might also be used by users on non-GNOME desktops in order to get their DE-native filepicker instead of GTK/GNOME's.

Comment 28

[Daniel]

This bug is super-annoying when doing local web development. It affects the official Firefox release from Mozilla on Flathub.

Typing e.g. localhost:3000 into the address field will show a prompt for opening the localhost protocol in another application.

Comment 29

[:Gijs (out for now; he/him)]

Jan or Martin, is there some way to convince xdg-desktop-portal to tell us the truth instead of making stuff up?

We can probably create hackarounds for localhost:1234 but that still won't help comment 0.

Comment 30

[:Gijs (out for now; he/him)]

(In reply to Daniel from comment #28)

This bug is super-annoying when doing local web development. It affects the official Firefox release from Mozilla on Flathub.

Typing e.g. localhost:3000 into the address field will show a prompt for opening the localhost protocol in another application.

Looks like potentially setting widget.use-xdg-desktop-portal in about:config to false will work as a workaround. Can you confirm if this helps? It might negatively impact things you do want to open with external apps though (both files you download and protocols).

Comment 31

[Daniel]

Looks like potentially setting widget.use-xdg-desktop-portal in about:config to false will work as a workaround. Can you confirm if this helps? It might negatively impact things you do want to open with external apps though (both files you download and protocols).

This options is false by default. I tried enabling it and restarted Firefox, but that made no difference.

Comment 32

[Martin Stránský [:stransky] (ni? me)]

I'm afraid the only solution here is to explicitly block some schemas, like localhost. This "enable all" thing is a flatpak sandboxing feature where an application can't probe system resources and access to system resources is controlled on user/system level only.

Comment 33

[:Gijs (out for now; he/him)]

(In reply to Martin Stránský [:stransky] from comment #32)

I'm afraid the only solution here is to explicitly block some schemas, like localhost. This "enable all" thing is a flatpak sandboxing feature where an application can't probe system resources and access to system resources is controlled on user/system level only.

This will fix the web developer case from comment #28, but not all the other examples in this bug (like site:... in the URL bar to search with the default search engine, or other things that are not schemes). Anyway, schemas are often app-specific these days, as they're the main way of launching local applications from the internet (and passing them information) when the user e.g. clicks a button or a link, without having the app set up a local http server which has all kinds of access control issues. Which means getting an exhaustive list of things that "are schemes" is not possible

Heck, even if we could get such a list, hardcoding those would break the case where the user does not have a handler but is e.g. searching magnet: handler app on linux or whatever.

The "whole point" of all this logic is that it adapts to the user's machine and doesn't depend on Firefox developers knowing that the protocol/application exists...

Is there any chance we could convince the xdg desktop people to add an API for this, perhaps one that requests user permission or something?

Otherwise, I guess we should just ignore ALL attempts to load protocols in the URL bar entirely (except perhaps items for which an network.protocol-handler.expose pref exists?) if we're using flatpak / xdg, and always do a search (which I'm sure will upset some other segment of the user population...).

Comment 35

[Martin Stránský [:stransky] (ni? me)]

https://github.com/flatpak/xdg-desktop-portal-gtk/issues/330 is related

Comment 36

[Jan Horak [:jhorak]]

I actually filled an issue on flatpak portal upstream: https://github.com/flatpak/xdg-desktop-portal-gtk/issues/330. Hacking and workarounds does not help there. In case it won't be fixed we'll have to disable scheme handling by system in the flatpak.

Comment 37

[Martin Stránský [:stransky] (ni? me)]

Jan is going to look at it. We need to discuss that with gnome/flatpak folks how to handle it.

Comment 41

[violetmage]

related: 1688966

Comment 42

[gudvinr]

This is reproducible under KDE Neon but doesn't happen in Arch Linux.

Comment 44

[Will Shanks]

Is there a workaround other than typing out http:// for the localhost case when one still wants the flatpak to open other schemes correctly? I tried setting network.protocol-handler.external.localhost to false, but that just stops Firefox from trying to use the system handler. It doesn't actually load the page from localhost. I think the only other workarounds mentioned above are variants of disabling all scheme handling?

Also, is there a workaround for urls that have dots in them? For example, www.mozilla.org:443 also tries to use the system handler for me.

Comment 45

[:Gijs (out for now; he/him)]

Marco, could we specialcase this situation in the address bar? I imagine people are unlikely to put zoommtg:blah or roblox or gotomeeting or msteams or whatever URLs directly in the address bar - they'll use the webpage that loads that protocol, and that will work. So perhaps on Linux, for URL fixup, when we establish that we get bogus results about external protocols, we could ignore those and rely only on the distinction between protocols we know we support and "everything else"?

Comment 46

[Marco Bonardo [:mak]]

(In reply to :Gijs (he/him) from comment #45)

Marco, could we specialcase this situation in the address bar? I imagine people are unlikely to put zoommtg:blah or roblox or gotomeeting or msteams or whatever URLs directly in the address bar - they'll use the webpage that loads that protocol, and that will work. So perhaps on Linux, for URL fixup, when we establish that we get bogus results about external protocols, we could ignore those and rely only on the distinction between protocols we know we support and "everything else"?

I suppose we could do some detection, similar to the DNS wildcard one, where we generate 2 random words and check if protocols for those exist, if so, we're in a case where the external protocol check cannot be trusted... Then your suggestion would be to not ask to the OS, but just trust internally registered protocols? It would solve some concerns here, but of course it would act wrongly for some protocols that are not tracked internally... I guess it's matter of picking the less evil here? Do we have a workaround for this, can users add the protocol to Firefox to ensure it works if they use to paste certain urls in the address bar?

To sum up, the workaround would be: if wildcard detection is positive, avoid the externalProtocolHandlerExists check and assume it's false.
If we agree implementing this workaround for now, while we wait for upstream, we could maybe ask Daisuke to look into it, since he made other changes to URIFixup in the past and knows this code.

Comment 47

[:Gijs (out for now; he/him)]

(In reply to Marco Bonardo [:mak] from comment #46)

(In reply to :Gijs (he/him) from comment #45)

Marco, could we specialcase this situation in the address bar? I imagine people are unlikely to put zoommtg:blah or roblox or gotomeeting or msteams or whatever URLs directly in the address bar - they'll use the webpage that loads that protocol, and that will work. So perhaps on Linux, for URL fixup, when we establish that we get bogus results about external protocols, we could ignore those and rely only on the distinction between protocols we know we support and "everything else"?

I suppose we could do some detection, similar to the DNS wildcard one, where we generate 2 random words and check if protocols for those exist, if so, we're in a case where the external protocol check cannot be trusted... Then your suggestion would be to not ask to the OS, but just trust internally registered protocols? It would solve some concerns here, but of course it would act wrongly for some protocols that are not tracked internally... I guess it's matter of picking the less evil here?

I think that's exactly what it is, and I think for the address bar, we should err on assuming a search query.

Do we have a workaround for this, can users add the protocol to Firefox to ensure it works if they use to paste certain urls in the address bar?

We could continue to check HandlerService.exists, which only checks for stored information in handlers.json? There'd be stored information if the protocol had been used before, and for a technical user it'd be possible to either change the file and/or create a webpage that would make it easy for users to invoke such a URI once, such that data gets stored in the file, to make it work in the address bar later.

To sum up, the workaround would be: if wildcard detection is positive, avoid the externalProtocolHandlerExists check and assume it's false.

Right, potentially substituting HandlerService.exists instead?

If we agree implementing this workaround for now, while we wait for upstream,

FWIW, based on the replies here and on similar bugs, I'm not convinced upstream will ever fix this.

we could maybe ask Daisuke to look into it, since he made other changes to URIFixup in the past and knows this code.

That sounds great. If the plan above still sounds workable, can you pass this on to Daisuke?

Comment 48

[Marco Bonardo [:mak]]

I filed bug 1744243. We can keep this bug to track upstream, if it will ever happen.

Comment 49

[Rémy Hubscher [:natim]]

A workaround I found was to add the key network.protocol-handler.external.localhost set to false in about:config

Comment 50

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 4 duplicates.
:jhorak, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 51

[Jan Tojnar]

(In reply to Will Shanks from comment #44)

Is there a workaround other than typing out http:// for the localhost case when one still wants the flatpak to open other schemes correctly? I tried setting network.protocol-handler.external.localhost to false, but that just stops Firefox from trying to use the system handler.

I installed the following desktop file:

[Desktop Entry]
Exec=firefox http://%u
MimeType=x-scheme-handler/localhost
Name=Open localhost protocol as http
NoDisplay=true
Type=Application
Version=1.4

You can put it to e.g. ~/.local/share/applications/localhost-proto-handler.desktop

You can also add more “schemes” by separating the fake MIME types with semicolons.

Comment 52

[Marco Castelluccio [:marco]]

Jan, since this seems to be more important than initially thought given you increased the severity, do you have any plans to fix this?

Comment 53

[Jan Horak [:jhorak]]

Until issue https://github.com/flatpak/xdg-desktop-portal-gtk/issues/330 is resolved, I don't think I can fix it.

Comment 54

[Marco Bonardo [:mak]]

We have a workaround bug on file at bug 1744243, it's just matter of finding an owner.

Comment 55

[Alex]

Kubuntu 22.04 now forces installing Firefox via snap and this bug now happens also there.

Comment 56

[Donal Meehan [:dmeehan]]

Setting 103 to Won't Fix, :mak is there any plans to address the workaround for bug 1744243?
Comment 55 mentions this is now impacting snap builds

Comment 57

[Marco Bonardo [:mak]]

I will directly look into the workaround fix.

Comment 58

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last months and this bug has severity 'S2'.
:bhearsum, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 59

[Marco Castelluccio [:marco]]

Sorry, there was a problem with the detection of inactive users. I'm reverting the change.

Comment 60

[Kestrel]

Bug 1744243 appears to have fixed it for me with the latest Nightly snap 105.0a1 (2022-08-14) but I had to delete handlers.json to remove unwanted protocols that had accumulated (eg site:) since there is no way to remove them in the UI. Now there is the opposite problem of not being able to easily register a new protocol handler.

Another workaround until the fix reaches stable is to set widget.use-xdg-desktop-portal.mime-handler = 0.

Comment 61

[[e-mail] (:bhearsum)]

I talked to :Mak about this elsewhere. According to him, a workaround was landed in https://bugzilla.mozilla.org/show_bug.cgi?id=1744243, and it should be okay to reduce the severity of this bug.

Comment 63

[Jan Horak [:jhorak]]

Attachment: Bug 1618094 Use SchemeSupported dbus method to check if system supports scheme; r?mak

The SchemeSupported has been added to the org.freedesktop.portal.OpenURI interface.
That should fix problems with wrong uri scheme handling by checking whenever system
is able to open specified scheme.

Comment 64

[Jan Horak [:jhorak]]

There's the SchemeSupported method implementation: https://github.com/flatpak/xdg-desktop-portal/pull/1203

Comment 65

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9cdd0a07644d
Use SchemeSupported dbus method to check if system supports scheme; r=stransky

Comment 66

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/9cdd0a07644d