1667639 - Ending a keyword query with a question mark (?) searches default search engine instead and restricts to search

Status: VERIFIED FIXED

(Firefox :: Address Bar, defect, P3)

Description

[Asif Youssuff]

As seen on https://www.reddit.com/r/firefox/comments/j0cwyp/question_mark_at_end_of_keyword_search/

Steps to reproduce:

1. Create a bookmark keyword like location https://crash-stats.mozilla.org/search/quick/?query=%s and any keyword
2. start query in address bar with keyword chosen in step 1, space, any query, and append a ?

What happens:

Firefox suggests a search on the user's default search engine instead of the bookmark query.

Expected result:

Query is submitted to bookmark query with ? sent to search.

20:34.01 INFO: Narrowed integration regression window from [fb9c01b7, 446aa773] (3 builds) to [fb9c01b7, 711b8615] (2 builds) (~1 steps left)
20:34.01 INFO: No more integration revisions, bisection finished.
20:34.01 INFO: Last good revision: fb9c01b719fac0644f1a3d25b4c9371d322d05d1
20:34.01 INFO: First bad revision: 711b8615bb3f978a68a27314719247fdde9a6aed
20:34.01 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=fb9c01b719fac0644f1a3d25b4c9371d322d05d1&tochange=711b8615bb3f978a68a27314719247fdde9a6aed

Comment 1

[Asif Youssuff]

[Tracking Requested - why for this release]:

Regression in user facing feature.

Comment 2

[Marco Bonardo [:mak]]

I'm not completely sure what's the right fix here, ? is a restriction override, as such it also overrides keywords.
My best bet is that we should stop supporting restriction character at the end, that would also be more compatible with update 2. Maybe do that when update 2 is enabled, so that will be the future state.
Drew, wdyt?

Comment 3

[Marco Bonardo [:mak]]

I don't think we're going to uplift a fix for this, it's an edge case.

Comment 4

[Asif Youssuff]

The way I see it, by using the bookmark keyword, I have already chosen a search engine, kinda like doing @google (for an example of a built in search keyword). If I do @google and end my query with a % or *, I don't search open tabs and bookmarks - that query is sent unmodified to the selected search engine.

I think the restriction character at the end of the query does make sense if I have not already selected an engine, however, so removing support for the restriction character at the end of any query would be undesirable.

Comment 5

[Drew Willcoxon :adw]

IIRC I previously argued (not strongly) that we shouldn't support restriction chars at the end, so I would personally be OK with that. However, the fact that Asif says that would be undesirable makes me wonder how often people actually do that -- maybe more often than I assumed.

I also think what Asif says about the keyword makes sense. Maybe we could generalize that into something like, keywords or restriction chars at the start of the query take precedence over chars at the end? Off hand I'm not sure how hard that would be to implement or whether there are edge cases, and it has the downside of adding yet more complexity and magic around restriction chars.

One other thing is that we want to unify the various kinds of keywords, so as I understand it, at some point bookmark keywords will enter search mode just like engine aliases -- at least bookmark keywords with %s substitutions? If that's the case, we may want to defer this bug until then, unless we think the regression in functionality is worth fixing soon.

Comment 6

[Marco Bonardo [:mak]]

(In reply to Drew Willcoxon :adw from comment #5)

IIRC I previously argued (not strongly) that we shouldn't support restriction chars at the end, so I would personally be OK with that. However, the fact that Asif says that would be undesirable makes me wonder how often people actually do that -- maybe more often than I assumed.

Yes, but my point is that now we have a more visible way to achieve the same result through the local one-off buttons. Before, once you had typed the search string, the only way to restrict was to type the restriction at the end; now you can click or ALT+UP to the local one-off.

I also think what Asif says about the keyword makes sense. Maybe we could generalize that into something like, keywords or restriction chars at the start of the query take precedence over chars at the end?

Of course that would be the solution to this bug, I thought the same. But for it to work the tokenizer should immediately know if the first word is a keyword/alias, that is something I already suggested doing in the past (by making tokenize() async and having a memory cache). For just this case the cost is much higher than the benefit.

One other thing is that we want to unify the various kinds of keywords, so as I understand it, at some point bookmark keywords will enter search mode just like engine aliases -- at least bookmark keywords with %s substitutions? If that's the case, we may want to defer this bug until then, unless we think the regression in functionality is worth fixing soon.

That would also solve this, but we'll still be stuck with the "searchstring *" non-visual search mode (no chiclet indicator), unless we start converting restriction tokens at the end into search mode, that would still require at least typing a space after the token and cause a lot of visual flicker... It sounds bad :(

That's why from a consistency point of view, now it looks like an acceptable solution would be to only accept restriction at the beginning, where it can be converted into the search mode chiclet. It wasn't a given until we had search mode.

Comment 7

[Asif Youssuff]

(In reply to Marco Bonardo [:mak] from comment #6)

That's why from a consistency point of view, now it looks like an acceptable solution would be to only accept restriction at the beginning, where it can be converted into the search mode chiclet. It wasn't a given until we had search mode.

It is not consistent today that a restriction token with a space before it at the end of a non-scoped query searches based on the token, though. It also seems like restriction tokens today work in the middle of a non-scoped query if they are surrounded by spaces.

In any case, I like that I can choose to use a restriction token at the end of a query because I sometimes get irrelevant history suggestions from the awesomebar, and it would be incredibly cumbersome to remove my search query, type the restriction character, then retype my query to find a result that I am currently able to just type % or * for.

Comment 8

[Marco Bonardo [:mak]]

(In reply to Asif Youssuff from comment #7)

It also seems like restriction tokens today work in the middle of a non-scoped query if they are surrounded by spaces.

It should not, restriction only work at the beginning or at the end, this changed some time ago.

In any case, I like that I can choose to use a restriction token at the end of a query because I sometimes get irrelevant history suggestions from the awesomebar, and it would be incredibly cumbersome to remove my search query, type the restriction character, then retype my query to find a result that I am currently able to just type % or * for.

You should not, with search mode you can just click on the restriction button or ALT+UP until you reach it.

Comment 9

[Asif Youssuff]

(In reply to Marco Bonardo [:mak] from comment #8)

(In reply to Asif Youssuff from comment #7)

It also seems like restriction tokens today work in the middle of a non-scoped query if they are surrounded by spaces.

It should not, restriction only work at the beginning or at the end, this changed some time ago.

Okay, then that may be another new bug. I have only ever used the restriction token at the end or the beginning, FWIW.

In any case, I like that I can choose to use a restriction token at the end of a query because I sometimes get irrelevant history suggestions from the awesomebar, and it would be incredibly cumbersome to remove my search query, type the restriction character, then retype my query to find a result that I am currently able to just type % or * for.

You should not, with search mode you can just click on the restriction button or ALT+UP until you reach it.

Clicking doesn't really make a lot of sense because my hands are on the keyboard already, and sorry - alt-up is not discoverable and still cumbersome - especially when you are proposing removing a feature that continues to work today.

Tab would be more discoverable, but I know that is still in place for users who prefer tab moving through suggestions. Also, ctrl-up works for me as well - is that expected? I know I have used that in the search toolbar.

Comment 10

[Asif Youssuff]

Also, just realized that I don't really care if the chiclet appears when typing a restriction token at the end of a query, because I am modifying my existing query instead of restricting my results based on prior scoping. There is serious convenience and power afforded by this approach - one that the scopes in the "this time search with" certainly helps for keyboard + mouse driven users, but nothing really comes close to typing a restriction character vs. doing alt-up/down until I get to the "engine" I am looking for.

Comment 11

[armonth]

Hello!

I usually use bookmarks with keywords for almost anything (yt for youtube, web for web archive, wikien and wikies for both english and spanish Wikipedia, etc). Recently I found this bug and is, as a minimum, annoying.

One of my keywords is dl for DeepL used to translate phrases so you can imagine what happens 99% of the times:

dl how are u?
dl ¿cómo estás?

At first I thought that the bookmark had been deleted/corrupted somehow. And is the "?" at the end what changes behavior from DeepL to default search engine. Since we are talking about translators, using "?" or not, changes the meaning of the phrase and the web results.

Comment 12

[Marco Bonardo [:mak]]

(In reply to Asif Youssuff from comment #9)

Okay, then that may be another new bug. I have only ever used the restriction token at the end or the beginning, FWIW.

Not a bug, it was our decision.

Clicking doesn't really make a lot of sense because my hands are on the keyboard already, and sorry - alt-up is not discoverable and still cumbersome - especially when you are proposing removing a feature that continues to work today.

How is typing *, % or ^ more discoverable than ALT+UP? both are things you must know about.

Tab would be more discoverable, but I know that is still in place for users who prefer tab moving through suggestions. Also, ctrl-up works for me as well - is that expected? I know I have used that in the search toolbar.

CTRL+UP is the same as UP afaik. UP would also work but you must press it twice to skip the settings button, while ALT directly goes to one-off.

(In reply to armonth from comment #11)

At first I thought that the bookmark had been deleted/corrupted somehow. And is the "?" at the end what changes behavior from DeepL to default search engine. Since we are talking about translators, using "?" or not, changes the meaning of the phrase and the web results.

Yes, that's why we are discussing whether having undiscoverable characters that typed at the end of a phrase can break expectations.

Comment 13

[Asif Youssuff]

(In reply to Marco Bonardo [:mak] from comment #12)

(In reply to Asif Youssuff from comment #9)

Clicking doesn't really make a lot of sense because my hands are on the keyboard already, and sorry - alt-up is not discoverable and still cumbersome - especially when you are proposing removing a feature that continues to work today.

How is typing *, % or ^ more discoverable than ALT+UP? both are things you must know about.

Tab would be more discoverable, but I know that is still in place for users who prefer tab moving through suggestions. Also, ctrl-up works for me as well - is that expected? I know I have used that in the search toolbar.

I meant undiscoverable as opposed to Tab, not as opposed to the restriction tokens. alt/control-up/down is a lot less discoverable than Tab because Tab is standardized across apps and OSes for moving focus to the next input element, whereas alt/control-up/down is a completely unknown invention.

Per the documentation, Control-up/down switches the default engine, FWIW, whereas Alt-up/down changes it for the current search: https://support.mozilla.org/kb/keyboard-shortcuts-perform-firefox-tasks-quickly#w_search

Alt-up/down is still more slower and more error-prone than the restriction token, since a restriction token doesn't force me to move focus to another UI element one by one - I can simply type a single character and get the result I am looking for.

Comment 14

[Marco Bonardo [:mak]]

(In reply to Asif Youssuff from comment #13)

Per the documentation, Control-up/down switches the default engin

only in the legacy search bar, it never did (and will unlikely do) in the urlbar.

Alt-up/down is still more slower and more error-prone than the restriction token

I'm not discussing whether it's faster or simpler, I'm discussing whether it's consistent with the general urlbar direction. I agree with you that for an advanced user it's simpler to type a char than click a button. The fact is fixing this bug in the status-quo may not be trivial (the code went a different direction), and requires some rearchitecture, and the result is not consistent with the direction of making things more discoverable.
It's matter of figuring out a compromise, or spend quite some time to rearchitecture keywords detection.

Comment 15

[Jan Moesen]

It might be worth adding “bookmarklet” to the summary. At first I thought it only impacted my JavaScript bookmarks (starting with javascript: and containing %s) but could not find any relevant bugs.

Proposed new summary: “Ending a query with a question mark (?) in a bookmark/bookmarklet keyword searches default search engine instead”

Although that would prevent exact matches for “"bookmark keyword"”, I guess.

Anyway, +1 for use cases like in comment 11, or even bookmarklets that don’t navigate away from the page. For example, I have a bookmarklet with keyword len to tell me the length of a string, and len foo bar baz? now goes to the default search engine instead of showing a simple alert box saying that string has 12 characters. This unexpected unloading of the page instead of showing the alert box has actually caused me some minor dataloss, though I admit this is probably very niche usage. (Though niche can be nice.)

Comment 16

[Drew Willcoxon :adw]

(In reply to Jan Moesen from comment #15)

Although that would prevent exact matches for “"bookmark keyword"”, I guess.

Neither of Bugzilla's simple search nor advanced search does exact phrase matching by default, so it should be OK. You can do exact phrase matching with the advanced search, but if you're doing that in the first place, you should know enough to know not to rely on it alone, so no worries.

Comment 17

[ariasuni]

In the end, is this easily fixable or not? Should we wait for bug 1653261 or not?

Comment 18

[B.J. Herbison]

Question mark behaves differently from other overrides. It you type "keyword query^" (no space before ^) Firefox preserves the keyword search, while "keyword query ^" (space before ^) switches to the default engine, but for ? the switch happens even without a space before the ?.

Is this difference intentional? In https://support.mozilla.org/en-US/kb/address-bar-autocomplete-firefox#w_changing-results-on-the-fly it says that the special character must be "separated by spaces" to have the special function.

(In reply to Marco Bonardo [:mak] from comment #2)

I'm not completely sure what's the right fix here, ? is a restriction override, as such it also overrides keywords.

Where is this keyword override behavior documented? Looking at https://support.mozilla.org/en-US/kb/address-bar-autocomplete-firefox#w_changing-results-on-the-fly it says that the characters change the displayed search suggestions. The page doesn't say that the modifiers change what happens when you press Enter.

Comment 19

[Michal Čaplygin [:myf]]

Minimal STRs:

0. Add "echo" Bookmarklet

• Either using Menu > Bookmarks > Manage Bookmarks > Organize > Add Bookmark
• or right clicking inside the Bookmark toolbar and choosing "Add Bookmark…"

Name: (test, or whatever)
URL: javascript:alert("%s")
Keyword: echo

1. Invoke with "hello!"

• Focus URL bar
• type echo hello!
• press Enter

2. Invoke with "hello?"

• Focus URL bar
• type echo hello?
• press Enter

Expected outcome

Both 1. and 2. should produce alert modal dialogue with text "hello!" (1.) / "hello?" (2.) respectively.

Actual outcome

Second test (2.) makes a default search engine search instead.

---

?Leading question mark

I'd like to also point out a related nuisance that even leading question mark NOT separated by a space currently ruins the functionality: for example while you can find this issue by its ID (1667639) using URL bar to switch to this tab from another one using the number (or id=1667639, or cgi?id=1667639) entering ?id=1667639 instead will also kick you to web search.

---

Solution?

Could the current "non-separated question mark functionality" be altered, so it would be only the "last resort"? Concretely: IF the current query matches search keyword, rest of the query will be passed to that keyword and not processed as implicit web search?
(Already proposed in #4.)
And analogically, if there is URL history match (?id=1667639) or title match, prefer such result.
So in short: whatever? (or ?whatever) should do the search strictly only if there is no whatever? in the history and if the query does not start with a keyword.

(Or alternatively, ditch the non-separated leading question mark functionality, and trailing question mark override functionality altogether.)

Comment 21

[github.com.k]

I think it would be fair to rethink the priority with the growth of conversational LLM based search engines. Questions are the default search input here.

Comment 22

[Michal Čaplygin [:myf]]

Good point, chat interfaces (of LLM based apps) are likely to grow radically in the near future. Real-world example: currently you can ask Bing Chat (soon "Copilot") a question from URL (so it starts responding when page loads without additional interaction) using keyword search:

https://www.bing.com/search?q=%s&showconv=1&sendquery=1

… just you must not end your question with a question mark, what feels super odd. Technically, punctuation likely makes no difference, both

• Which browsers support bookmarks for keyword searches

and

• Which browsers support bookmarks for keyword searches?

yields (almost) the same outcome. But having to refrain from natural and correct punctuation while writing the question-query is particularly difficult here.

Comment 23

[Mandy Cheang [:mcheang][she/her]]

Attachment: Bug 1667639 - Make keyword at the start of the string take precedence over restrict symbols at the end of the string. r=mak!

This patch modifies PlacesUtils to cache the keywords and adds functionality to
find a keyword. The Tokenizer calls PlacesUtils to know if the first word
is a keyword and if the first word is a keyword, then the rest of the search
string will be treated as text and Tokenizer won't filter any other restrict
symbols.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4e1ff4f27b7c
Make keyword at the start of the string take precedence over restrict symbols at the end of the string. r=mak

Comment 25

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/4e1ff4f27b7c

Comment 26

[Ardelean Oana, Desktop Test Engineering [:oardelean]]

Reproducible on a 2024-01-15 Nightly build on Windows 10 using the steps from Comment 19.
Verified as fixed on Firefox 123.0b7 and Firefox Nightly 124.0a1 on Windows 10, macOS 12, Ubuntu 22.