1994400 - Korean/Japanese composition unexpectedly stops when mouse select triggers Firefox builtin AI chatbot popup icon

Status: NEW

(Core :: Machine Learning: Frontend, defect, P1)

Description

[Raul Bucata]

Environment:
Operating system: Windows 10
Firefox version: Firefox 144.0

Preconditions:

• Korean input installed

Steps to reproduce:

1. Navigate to: https://www.google.com/
2. Searching for something
3. Change the keyword with mouse select
4. Type something in Korean
5. Observe the input field

Expected Behavior:
Input does not break

Actual Behavior:
The first letter breaks

Notes:

• Reproduces regardless of the status of ETP
• Reproduces in firefox-nightly, and firefox-release
• Does not reproduce in chrome

Created from https://github.com/webcompat/web-bugs/issues/182893

Comment 1

[Raul Bucata]

Attachment: firefox drag select korean bug.mkv

Comment 2

[Kagami Rosylight [:saschanaz] (they/them)]

(In reply to Raul Bucata from comment #0)

1. Navigate to:

To https://google.com I assume?

Comment 3

[Kagami Rosylight [:saschanaz] (they/them)]

STR needs to include enabling AI chatbot.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Since nightly and release are affected, beta will likely be affected too.
For more information, please visit BugBot documentation.

Comment 5

[Kagami Rosylight [:saschanaz] (they/them)]

The chatbot icon started to still the focus, which is the problem. Bisecting now.

Comment 6

[Kagami Rosylight [:saschanaz] (they/them)]

Affects macOS and Linux (fcitx5) too.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

:ngrato, since you are the author of the regressor, bug 1922623, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 8

[Nick]

@yjamora this might be something to look into.

Comment 9

[Pascal Chevrel:pascalc]

Is that really a website issue if the regressor is a Firefox code change? Is the Severity ok? Thanks

Comment 10

[Dennis Schubert [:denschub]]

Well, we track everything as a site-report that's breaking a website, no matter "why" it's broken. So from that point of view, yeah, it's a WebCompat bug (that's caused by a regression).

The S4 is based on our rating that this only affects some users, not all users, which drops the WebCompat impact score a lot. However, since this affects Japan, I think we should treat this higher. I'll bump this to S2, which it would be if we'd say it affects "all" users, which seems fair.

Comment 11

[Jeff Muizelaar [:jrmuizel]]

Yubin, can you take a look at this per comment 8?

Comment 12

[Yubin Jamora]

Thank you for reporting the bug. Kagami Rosylight [:saschanaz] (they/them) is right.

The chatbot icon started to still the focus, which is the problem. Bisecting now.

Will work on that.

Comment 13

[Jeff Muizelaar [:jrmuizel]]

Yubin, any update on this?

Comment 14

[Yubin Jamora]

Sorry for the delay. I looked into the issue today and as Kagami Rosylight [:saschanaz] (they/them) mentioned the problem is hidePopup() steals focus, which breaks Korean IME composition when typing in input fields.

@mardak,
wonder would it be acceptable to skip showing aiActionButton for the shortcut when user selects text inside input/textarea/contenteditable elements? This avoids the hidePopup() focus issue entirely. or the only way I could fix it was moving the panel offscreen with panel.moveTo(-9999, -9999) instead of calling hidePopup() which i don't want to do it. - actually we need the shortcut for the proofread in input/textarea...

Comment 15

[Ed Lee :Mardak]

i suppose there could be 2 parts of this bug of showing the shortcut in the first place when selecting text with a mouse click hhttps://searchfox.org/firefox-main/rev/93aad2a6615f670b1279c229dd37f7397236131a/browser/components/genai/GenAIChild.sys.mjs#62-69,85-89

or hiding when the selection changes https://searchfox.org/firefox-main/rev/93aad2a6615f670b1279c229dd37f7397236131a/browser/components/genai/GenAIChild.sys.mjs#110

link preview had a related korean typing issue bug 1967131 where we worked around with some typing backoff heuristic (and later removed the shift shortcut completely for other shift issues)

it is a desired feature to get prompts like proofread for inputs, so that would require product input. is there a way to be more narrowly targeting when the popup appears such as the content or previous composing actions? i suppose the tricky part is the user didn't necessarily type into this particular input box yet so we don't know if there will be future composition

i suppose just to check, deleting the text first before typing avoids the issue?

Comment 17

[Kagami Rosylight [:saschanaz] (they/them)]

i suppose just to check, deleting the text first before typing avoids the issue?

That removes text selection, which then won't open popup, so yes.

Comment 18

[Donal Meehan [:dmeehan]]

:mardak, what's the plan for a fix here? Wondering if we will have in time for Fx148 or what is the plan?

Comment 19

[Ed Lee :Mardak]

yubin, is there a quick/small fix we could potentially uplift? it seems unlikely product would accept some of the proposed fixes unless maybe we narrowly target to some locales or input languages perhaps ja and ko?

i'll also ping jolie but i suspect smart window will be higher priority

Comment 20

[Yubin Jamora]

Instead of doing panel hide, we could do a really hacky thing like moving the panel offscreen with panel.moveTo(-9999, -9999). Not sure if it's acceptable, and also I don't know what other regressions it would cause..

Comment 21

[Alice0775 White]

https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/textarea is also affected.
Google Spreadsheets(https://docs.google.com/spreadsheets) and Excel(https://excel.cloud.microsoft) are also affected.

Comment 22

[Alice0775 White]

[Tracking Requested - why for this release]: IME user cannot input text if pre-filled text is selected in textarea. Please back out the offending patch.

Comment 23

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox148 (beta) and tracked for firefox149 (nightly). However, the bug still isn't assigned.

:Mardak, could you please find an assignee for this tracked bug? Given that it is a regression and we know the cause, we could also simply backout the regressor. If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 24

[Ed Lee :Mardak]

looks like address bar does this position fixed top: -999px https://searchfox.org/firefox-main/rev/e315c373bd165aa45994c218186b3b5df83a66da/browser/themes/shared/urlbar-searchbar.css#1270-1273

i guess we can try your comment 20 suggestion for those who want to keep the text selection shortcut functionality? we'll want to make sure it doesn't break other typing/focus by keeping the panel open

Comment 25

[Donal Meehan [:dmeehan]]

:yubin jamora, next week is the last week of beta for Fx148.
I see this was set as a P1, so mentioning hopefully we'll have a patch in time that is safe to uplift?