1907766 - On screen keyboard does not resize browser so it covers text boxes

Status: RESOLVED FIXED

(Core :: Widget: Win32, defect, P2)

Description

[Valley]

Steps to reproduce:

It happens constantly and seems to be a consistent bug, There is no way to reproduce it

Actual results:

Only on firefox, the in-built windows keyboard does not resize the browser to allow for people to see the text . I have tried to see if this issue is on other browsers like chrome but it is not.

Expected results:

The browser should resize so the text box selected is visible

Comment 1

[:Gijs (out for now; he/him)]

Some more details would be helpful here, because bug 1834394 suggests that we resize the entire browser window when the onscreen keyboard appears.

Can you outline exactly what version of Windows this is, what your relevant windows settings are, and how you're triggering the onscreen keyboard?

Comment 2

[Valley]

I use windows 11, up to date. i am not sure what settings would be relevant to this is bug, the onscreen keyboard pops up when tapping on a space where i can enter text or if by pressing the touch keyboard button on the taskbar

Comment 3

[Ray Kraesig [:rkraesig]]

Can you provide a screen-capture video of the behavior you're seeing?

Comment 4

[Valley]

Attachment: Firefox keyboard bug

Comment 5

[Valley]

Attachment: new video hoping this works

Comment 6

[Valley]

i cant seem to upload the video, with the attachment featurem he is the link to it to https://vimeo.com/989060846?share=copy

Comment 7

[Ray Kraesig [:rkraesig]]

Unfortunately you've uploaded the video to Vimeo, not to us; and Vimeo provides no way to download the video file.

Descriptive summary for future readers, after the Vimeo link inevitably goes stale:

(The video appears to be taken with a phone camera in the off-hand. Only the lower three-quarters or so of the monitor is visible.)

[00:00] User has Slack open in a web browser, with the on-screen keyboard open. The keyboard is fully visible; above the keyboard is the lower edge of the Slack webpage, with the text-entry field for a new channel named #test also fully visible.

[00:01] User enters "chrome" on the on-screen keyboard, causing the letters to appear in the text-entry field. The user then [00:03] presses Enter. The message is successfully sent, and immediately visible.

[00:06] User presses the X in the upper-right of the on-screen keyboard. The webpage moves down behind the keyboard only a handful of frames before the keyboard disappears, revealing that the webpage's lower edge has become flush with the Windows 11 taskbar. The taskbar itself reveals that Google Chrome is the active application.

[00:08] User taps a non-interactive space in the Slack interface, then [00:09] taps the text-field, summoning the on-screen keyboard again. This initially conceals the Slack interface, but after a moment, it is displaced upwards to become fully visible above the keyboard again.

[00:12] User performs an action out of view of camera which dismisses the keyboard and — according to the taskbar — brings Firefox into the foreground. Firefox displays the same Slack interface, with the already-sent "chrome" message visible.

[00:14] User taps a non-interactive space in the Slack interface, then [00:15] taps the text-field, summoning the on-screen keyboard again. Unlike with Chrome, this does not cause the webpage to shift upwards to become visible. User enters "firefox", to no visible effect, before [00:19] pressing Enter, which shifts the visible portion of the webpage up slightly [a].

[00:20] User presses the X in the upper-right of the on-screen keyboard, dismissing it. The webpage does not move; it is revealed that the message "firefox" was sent as expected. The shift [a], above, is demonstrated to have been due to the new message having come in as expected, rather than anything to do with the keyboard itself.

[00:21] User taps the text-field, summoning the on-screen keyboard again. The text-entry field and messages do not visibly move, and are again concealed thereby.

[00:24] User dismisses the on-screen keyboard, presumably by tapping the X (which is no longer visible in the camera's field of view, presumably due to drift). The rendered webpage continues not to move.

After a couple of false starts, [00:26] user again resummons and [00:28] re-dismisses the on-screen keyboard, demonstrating no change in behavior.

Comment 8

[Ray Kraesig [:rkraesig]]

I've been able to reproduce this locally. It's noteworthy that this is very different from the behavior seen in the video provided by :saschanaz in bug 1834394 comment 1 — we no longer resize the window — and mozregression confirms that this is due to the recent fix for bug 1645571, which would have just made it out to Release in v128, and which does approximately what :saschanaz identified.

Unfortunately, bug 1645571 explicitly prevents Windows from resizing our window, which was what originally prevented this from being a problem. What I think we need to do at this point is to render the document at an offset, so that it still thinks it has a full n pixels' worth of height (and therefore CSS media rule-changes are not triggered), but actually only k pixels' worth of the document is drawn.

Since this a) relies on knowing where in the webpage the text field is (so that it can be scrolled into view), and b) should affect only the document's viewport and not that of the Firefox chrome, this is probably mostly not Widget-level. Forwarding to... DOM, I suppose?

Comment 9

[Kagami Rosylight [:saschanaz] (they/them) (OOO until Dec 11)]

I think it doesn't have to know the text field position. What Chrome is doing on Windows is putting the whole page in a parent scroll container and make it fit the remaining screen area. That's out of scope of DOM I think. Maybe widget (to know the keyboard size) + Firefox frontend? Thoughts?

(Having said that, there is VirtualKeyboard Chromium-only API (bug 1730568) which allows the page to manually handle it, which would be in the DOM scope but out of scope of this bug)

Comment 10

[Kagami Rosylight [:saschanaz] (they/them) (OOO until Dec 11)]

(We might need to back out the fix until we figure this out)

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1645571

Comment 12

[Ray Kraesig [:rkraesig]]

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #9)

That's out of scope of DOM I think. Maybe widget (to know the keyboard size) + Firefox frontend? Thoughts?

Sounds reasonable. Forwarding to Firefox :: General for further analysis and rebucketing, since none of the categories jump out at me as appropriate.

Widget will still need to be involved, as you note (and also to acquire and propagate the "keyboard is visible" event in the first place)... but I'm not sure whether that just means "Widget :: Win32", or if Mac and Linux have similar behavior we need to account for. I know Android does, and even handles all this correctly, but that may be at the wrong level of abstraction to be relevant.

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

:cmartin, since you are the author of the regressor, bug 1645571, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 14

[:Gijs (out for now; he/him)]

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #10)

(We might need to back out the fix until we figure this out)

Given this is Windows 11 and happens 100% of the time per comment 0, and the other bug is taskbar auto-hide only (and apparently tablet mode only, which is gone from Win11 - but there are comments there saying it reproduces on Win11?!), that seems like what we should do, and apparently would be a DOM change? Anyway, can/should we prioritize that for 128, 129, the upcoming 128 dot release and/or esr? Who can drive that work?

(In reply to Ray Kraesig [:rkraesig] from comment #12)

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #9)

That's out of scope of DOM I think. Maybe widget (to know the keyboard size) + Firefox frontend? Thoughts?

Sounds reasonable. Forwarding to Firefox :: General for further analysis and rebucketing, since none of the categories jump out at me as appropriate.

Widget will still need to be involved, as you note (and also to acquire and propagate the "keyboard is visible" event in the first place)... but I'm not sure whether that just means "Widget :: Win32", or if Mac and Linux have similar behavior we need to account for. I know Android does, and even handles all this correctly, but that may be at the wrong level of abstraction to be relevant.

Android frontend is completely different from desktop so yeah, I expect that that doesn't help us here. Really, I'd expect the widget layer to deal with all of this - it can resize any browser window all by itself (and would then be in a better position to deal with potential duplicate/racy events etc.), it doesn't need the frontend for that.

macOS doesn't have a touch mode that I'm aware of. Linux is... Linux. Wayland is so opinionated about what apps can and cannot do that we probably couldn't do much ourselves even if we wanted to.

This is a Windows-specific problem, caused at least in part by MS continually changing their mind about how they want to support tablets and providing... imperfect... APIs for the same.

What Chrome is doing on Windows is putting the whole page in a parent scroll container and make it fit the remaining screen area. That's out of scope of DOM I think.

I don't understand what this means. But either way I'm confused - last time I tried this on Windows (don't have a machine that would do this to hand atm), the entire window resized. This seems to describe something different but I don't know what it is.

Comment 15

[Kagami Rosylight [:saschanaz] (they/them) (OOO until Dec 11)]

(In reply to :Gijs (he/him) from comment #14)

and apparently tablet mode only, which is gone from Win11 - but there are comments there saying it reproduces on Win11?!

The tablet mode taskbar has been restored since 2022, it's technically not gone even though it's not fully immersive as it was in Win8 (I'm not sure about the bug itself though): https://blogs.windows.com/windows-insider/2022/02/24/announcing-windows-11-insider-preview-build-22563/

Tap-to-open-onscreen-keyboard has been there forever, if this is what we call tablet mode here.

apparently would be a DOM change?

Unsure why a widget patch backout would be a DOM change? If it means a proper fix, not sure it's something we can do in a short term and I think it should be a widget/frontend change. Widget should be able to resize the window itself, but resizing window would retrigger bug 1834394 as it will resize viewport too. We need a way to not resize viewport as Chrome does (comment #9) which would be a non-straightforward work.

I don't understand what this means. But either way I'm confused - last time I tried this on Windows (don't have a machine that would do this to hand atm), the entire window resized. This seems to describe something different but I don't know what it is.

That was the behavior before this regression. Now it does not resize the window (which is kinda preferred behavior but it needs more work, as this bug explains).

Comment 16

[:Gijs (out for now; he/him)]

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #15)

Tap-to-open-onscreen-keyboard has been there forever, if this is what we call tablet mode here.

It's not, win10 tablet mode changed e.g. window controls on the apps to get rid of minimize/maximize/restore, and changed taskbar/app-switching mechanics. The Win11 taskbar thing you linked looks different yet again. MS can't seem to make their mind up on how their OS should behave in these modalities.

apparently would be a DOM change?

Unsure why a widget patch backout would be a DOM change?

OK, clearly I got muddled - either way, someone involved with the original patch should prioritize getting it backed out for 128 if still possible (and definitely ESR + beta + nightly). And then (separately/later) iterate on what the "right" solution is like.

Needinfo Ray for this, I guess, then, if it's not DOM.

We need a way to not resize viewport as Chrome does (comment #9) which would be a non-straightforward work.

I don't understand your description of what Chrome does and cannot try it. The explanation in comment 9 does not make sense to me. Can you please try to explain again? From a user/visual perspective, what are you suggesting should happen (rather than resizing the window, which was the pre-regression behaviour)?

Comment 17

[:Gijs (out for now; he/him)]

Didn't mean to clear Chris's needinfo.

Comment 18

[Kagami Rosylight [:saschanaz] (they/them) (OOO until Dec 11)]

Attachment: chrome-onscreen-keyboard.webm

So this is how it works on Chrome with data:text/html,<style>html, body { height: 100%; margin: 0; background: linear-gradient(black, white) }</style><input style="position:absolute; bottom: 0"> (a definitely unscrollable page)

As you see the unscrollable page becomes scrollable, as if the whole page is now contained in a fixed size iframe. The point is that the viewport size is kept fixed before and after opening onscreen keyboard on Chrome.

(And scratch my earlier point, this does need scrollIntoView which is in DOM 😅)

Comment 19

[Chris Martin [:cmartin]]

I see what's happened here. So... The change that we made in Bug 1645571 was to fix a glitchy behaviour that was ultimately caused by Windows trying (and intermittently failing) to resize our window a bunch of times on our behalf without us asking for it, causing the touch keyboard to flicker and disappear (leaving the touch user without a way to type).

Unfortunately, it appears that we were actually relying on Windows to resize our window for us instead of doing it ourselves, and now that we've "opted out" of that glitchy behaviour, we need to do something about it ourselves. That could be resizing our window (which is what Windows mostly tried to do previously), or (as :saschanaz suggested) we could try to resize the scrollable area.

Either way though, I think either solution will take more time than we have, and I agree that the number of people that would be affected and annoyed by the original bug pales in comparison to the number of people that will now be affected and annoyed by this.

I suggest we back it out and we can figure out what to do later. 😪

Comment 20

[:Gijs (out for now; he/him)]

[Tracking Requested - why for this release]:

(In reply to Chris Martin [:cmartin] from comment #19)

Either way though, I think either solution will take more time than we have, and I agree that the number of people that would be affected and annoyed by the original bug pales in comparison to the number of people that will now be affected and annoyed by this.

I suggest we back it out and we can figure out what to do later. 😪

Comment 21

[Ray Kraesig [:rkraesig]]

Either way though, I think either solution will take more time than we have, and I agree that the number of people that would be affected and annoyed by the original bug pales in comparison to the number of people that will now be affected and annoyed by this.

Roger roger. (I'd mistaken the other bug as being of greater severity than it sounds like it actually was?)

For the record, :Gijs has already started that ball rolling; the build sheriffs are reverting the patch, and that should be in the already-planned release of v128.0.3.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

This has been backed out from Nightly and also for 129.0b9, 128.0.3 and 128.1esr.
https://hg.mozilla.org/integration/autoland/rev/79645123c6f65973d21724d63d16e315ca1efeed
https://hg.mozilla.org/releases/mozilla-beta/rev/29b8a0134a20936a810831cea3197c6ac515ac3c
https://hg.mozilla.org/releases/mozilla-release/rev/70d19c6228c3030dc64235906cee7476f059e131
https://hg.mozilla.org/releases/mozilla-esr128/rev/8bb02a780758a4e7ce169d4cbe0d81aeb74852dc

Comment 23

[Ryan VanderMeulen [:RyanVM]]

Added to the 128.0.3 relnotes.

Fixed the Windows on-screen keyboard potentially concealing the webpage when displayed.