1911609 - AI shortcut doesn't appear near the selected text when the page is not 50% zoomed

Status: RESOLVED FIXED

(Core :: Machine Learning, defect, P1)

Description

[Chris Peterson [:cpeterson]]

Attachment: screenshot.png

Steps to reproduce

1. Enable the AI shortcut and set page zoom to 100% (default).
2. Load a page (such as https://example.com) and double-click to select some text on a web page.
3. See that the AI shortcut is displayed below and to the right of the selected text.
4. Increase page zoom to 200% (using Ctrl + +)
5. Double-click to select some text on the page again.

Expected result

The AI shortcut should still be displayed below and to the right of the selected text.

Actual result

The AI shortcut is display far above and to the left of the selected text. The more you zoom the page, the farther above and to the left the AI shortcut is displayed. See the attached screenshot.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:tarek, could you have a look please?

For more information, please visit BugBot documentation.

Comment 2

[Ed Lee :Mardak]

https://phabricator.services.mozilla.com/D226847#inline-1266845
review comment in bug 1922623 suggests this might want to be scaled with * this.contentWindow.devicePixelRatio

https://searchfox.org/mozilla-central/rev/fcf53e1685bfb990b5abc7312ac1daf617f0991f/browser/actors/ContextMenuChild.sys.mjs#764-765

Comment 3

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1911609 - Fix zoom handling in GenAI shortcuts. r=ngrato

Comment 4

[Ed Lee :Mardak]

bug 1922623 landed with some devicePixelRatio calculation, but it seems to be quite wrong now when not zoomed. it's correct when zooming out to 50% https://searchfox.org/mozilla-central/rev/548b6981501f59e3c9f2f7851c013e7d53c4e72f/browser/components/genai/GenAI.sys.mjs#629-641

Comment 5

[Ed Lee :Mardak]

Attachment: 100% zoom.png

Comment 6

[Ed Lee :Mardak]

Attachment: 30% zoom.png

Comment 7

[Ed Lee :Mardak]

also, the position seems to be quite wrong for external monitors that i can't even see where it's opening up

Comment 8

[Ed Lee :Mardak]

there was a recent change perhaps bug 1931692 that seemed to break the behavior that landed with bug 1922623 that was working fine a month ago (but didn't make it to m-c yet). turns out event.screenX changes between the original mouseup event and when we access it via a setTimeout debounce

https://searchfox.org/mozilla-central/rev/548b6981501f59e3c9f2f7851c013e7d53c4e72f/browser/components/genai/GenAIChild.sys.mjs#48,65,78

so we'll just grab the value before setTimeout

Comment 9

[Nick]

Attachment: Bug 1911609 - AI shortcut zoomed incorrect location r=Mardak

Zoom functionality was taken care of in another patch, however there was a new bug in nightly that distorted the formula to create the popup destitation. This patch fixes some scoping issues with the variables to keep the intentional values and produce an accurate placement of the popup

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5045cb4ff24b
AI shortcut zoomed incorrect location r=Mardak,firefox-ai-ml-reviewers

Comment 11

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/5045cb4ff24b