1831649 - Implement the 'interactive-widget' meta viewport key, to allow controlling viewport resize behaviour when showing a virtual keyboard

Status: RESOLVED FIXED

(Core :: Layout: Scrolling and Overflow, enhancement, P3)

Description

[Botond Ballo [:botond]]

Filing this bug to track a Firefox implementation of the interactive-widget meta viewport key, as a mechanism for allowing page authors to control the viewport resize behaviour when showing a virtual keyboard (or potentially other interactive widgets).

A few relevant links:

• Explainer doc for proposal: https://github.com/bramus/viewport-resize-behavior/blob/main/explainer.md#proposal
• CSSWG tracking issue: https://github.com/w3c/csswg-drafts/issues/7767
• Mozilla standards-position ticket: https://github.com/mozilla/standards-positions/issues/693

My understanding is that this is a newer alternative proposal to the Virtual Keyboard API tracked in bug 1730568.

Comment 1

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Set inputMode="none" in test_event_target_radius.html to avoid showing the software keyboard. r?masayuki!,botond!

test_event_target_radius.html doesn't require showing the software keyboard.
The keyboard shown by the test persists there for some reasons even after the
test has finished. Thus a new mochitest which will be introduced in this commit
series will not properly run, the new test needs to start without the software
keyboard.

Comment 2

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Parse `interactive-widget`. r?botond

It's defined in the CSS viewport spec.
https://drafts.csswg.org/css-viewport/#interactive-widget-section

Comment 3

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Detect the software keyboard height change. r?#geckoview-reviewers

Comment 4

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Propagate the software keyboard height change into nsPresContext. r?botond!,#geckoview-reviewers!

Comment 5

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Expand the display size used as input to mobile viewport sizing to include the software keyboard area in overlays-content and resizes-visual mode. r?emilio!,botond!

So that even if the chrome window gets resized by the software keyboard, the
contents can be laid out as if there's no software keyboard for overlays-content
and resizes-visual modes.

Comment 6

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Tweak the visual viewport size for resizes-visual and overlays-content modes. r?botond

|aDisplaySize| is based on the document viewer size which is changed by the
software keyboard, thus to make the visual viewport size unchange for
overlays-content mode we need to compensate the keyboard size.

Comment 7

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Enforce a reflow in the case where interative-widget value is changed. r?botond

Without this reflow changing the interactive-widget value will do nothing.
Though it's a rare case, we do it in a mochitest.

NOTE: The actual route to kick a reflow when the value is changed is;

1. MobileViewportManager is listening meta viewport changes
2. the listener invokes MobileViewportManager::RefreshViewportSize

Comment 8

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - interactive-widget tests. r?botond

A caveat; test_interactive_widget.html changes the top level content document so
that after the all test cases finished the changes need to be reverted.

The reason why we take this approach rather then opening a new tab is that on
the GeckoView test runner the software keyboard never appears in newly opened
documents. Newly opened documents on the GeckoView test runner are active but
background, it's a weird setup, I haven't found out any ways to make the
software keyboard work there without any troubles.

Comment 9

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Explicitly set android.windowSoftInputMode to GeckoView test runner. r?#geckoview-reviewers

And add a caveat about windowSoftInputMode in a document.

We implement interactive-widget [1] in a way that the content is laid out as if
the window doesn't get resized even if the application window gets resized by
the software keyboard. To make this way work, using adjustResize is mandatory.

An alternative approach to implement it is to change windowSoftInputMode value
depending on each content's interactive-widget value in each Android Activity,
i.e., set to adjustResize on interactive-widget=resizes-content mode, set to
adjustUnspecified on interactive-widget=resizes-visual or on
interactive-widget=overlays-mode. It will force applications to do more work.

[1] https://drafts.csswg.org/css-viewport/#interactive-widget-section

Comment 10

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Tweak viewport stuff while the dynamic toolbar is moving for interactive-widget. r?botond

Comment 11

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Stop synchronizing nsView size with the content visible area on Android in the case of resizes-visual and overlays-content modes. r?tnikkel!,botond!

On Android there's no window-ed mode so that even if those size are not in sync,
it will not cause any problems at least the visible size is larger than nsView
size. Though I am not 100% confident whether there's any exception or not.

Comment 12

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Trigger MobileViewportManager::RefreshViewportSize in the case where nsDocumentViewport boundaries changes but nsView boundaries unchanges. r?tnikkel!,botond!

Comment 13

[Hiroyuki Ikezoe (:hiro)]

Attachment: resizes-visual test case

Comment 14

[Hiroyuki Ikezoe (:hiro)]

Jon or Titouan, I need you guys help.

One of the patches (D204164) here uses ViewCompat.setOnApplyWindowInsetsListener, it works perfectly on GeckoView example. But it doesn't work on Fenix.

The reason is that one of our android-components also uses ViewCompat.setOnApplyWindowInsetsListener with null listener in Activity.exitImmersiveMode and unfortunately Activity.exitImmersiveMode gets called after GeckoView.onAttachedToWindow gets called where I tried to use ViewCompat.setOnApplyWindowInsetsListener (you can checked it in D204164). That means, the listener I attached is nullified soon after its registration. How can I solve this conflict? One odd thing I've observed is that Activity.exitImmersiveMode gets called without any Activity.enterImmersiveMode, I suppose they should be paired.

Comment 15

[Titouan Thibaud [:tthibaud]]

Hi Hiro! I'll take some time to look into this on Thursday.

Comment 16

[Jonathan Almeida [:jonalmeida]]

Back from PTO - clearing NI to let Titouan respond.

Comment 17

[Sebastian Zartner [:sebo]]

The implementation is obviously ongoing and interative-widget has already been added to the spec. in the meantime.
Though I want to point out that someone noted on the related GitHub issue that there doesn't seem to be a resolution yet for adding this feature. And nobody has answered that comment so far.

Emilio, as one of the authors of the spec., I guess it's on you to clarify that.

Regarding documentation, once this lands, the page https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta/name#standard_metadata_names_defined_in_other_specifications needs to be updated and probably reworked as well as the related browser compat data (which currently doesn't include any of the viewport keys). As there are a bunch of keys already, I guess it also makes sense to additionally create a separate page for <meta name="viewport"> under https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta/name/viewport.

Sebastian

Comment 18

[Emilio Cobos Álvarez (:emilio)]

I added it to the agenda to get it reviewed.

Comment 19

[Titouan Thibaud [:tthibaud]]

(In reply to Hiroyuki Ikezoe (:hiro) from comment #14)

Jon or Titouan, I need you guys help.

One of the patches (D204164) here uses ViewCompat.setOnApplyWindowInsetsListener, it works perfectly on GeckoView example. But it doesn't work on Fenix.

The reason is that one of our android-components also uses ViewCompat.setOnApplyWindowInsetsListener with null listener in Activity.exitImmersiveMode and unfortunately Activity.exitImmersiveMode gets called after GeckoView.onAttachedToWindow gets called where I tried to use ViewCompat.setOnApplyWindowInsetsListener (you can checked it in D204164). That means, the listener I attached is nullified soon after its registration. How can I solve this conflict? One odd thing I've observed is that Activity.exitImmersiveMode gets called without any Activity.enterImmersiveMode, I suppose they should be paired.

Sorry for the delay, I finally took time to investigate the ViewCompat.setOnApplyWindowInsetsListener issue.

I haven't figured out a good way to solve it. There's a conflict between Fenix and GeckoView applying a listener on the same view, with indeed Fenix reseting the listener to null in Activity.exitImmersiveMode. Activity.exitImmersiveMode gets called "preventively" at several times, without checking if the app was in immersive mode before. We could try to check if immersive mode is on before applying the code to exit from it, but I'd say it would be hard to make sure we don't leave any corner case behind.

I wonder if it would be possible to expose an API from GeckoView, like windowInsetsChanged(...), so that Fenix would be responsible to tell GeckoView about the changes. It would make it easier to handle the conflict, as Fenix would handle both the code related to exitImmersiveMode and to the keyboard resizing.

I'm not 100% sure it would be the best option, but for now I couldn't come up with a better idea. Any thought?

Comment 20

[Hiroyuki Ikezoe (:hiro)]

(In reply to Titouan Thibaud [:tthibaud] from comment #19)

(In reply to Hiroyuki Ikezoe (:hiro) from comment #14)

Jon or Titouan, I need you guys help.

One of the patches (D204164) here uses ViewCompat.setOnApplyWindowInsetsListener, it works perfectly on GeckoView example. But it doesn't work on Fenix.

The reason is that one of our android-components also uses ViewCompat.setOnApplyWindowInsetsListener with null listener in Activity.exitImmersiveMode and unfortunately Activity.exitImmersiveMode gets called after GeckoView.onAttachedToWindow gets called where I tried to use ViewCompat.setOnApplyWindowInsetsListener (you can checked it in D204164). That means, the listener I attached is nullified soon after its registration. How can I solve this conflict? One odd thing I've observed is that Activity.exitImmersiveMode gets called without any Activity.enterImmersiveMode, I suppose they should be paired.

Sorry for the delay, I finally took time to investigate the ViewCompat.setOnApplyWindowInsetsListener issue.

I haven't figured out a good way to solve it. There's a conflict between Fenix and GeckoView applying a listener on the same view, with indeed Fenix reseting the listener to null in Activity.exitImmersiveMode. Activity.exitImmersiveMode gets called "preventively" at several times, without checking if the app was in immersive mode before. We could try to check if immersive mode is on before applying the code to exit from it, but I'd say it would be hard to make sure we don't leave any corner case behind.

Yeah, I agree.

I wonder if it would be possible to expose an API from GeckoView, like windowInsetsChanged(...), so that Fenix would be responsible to tell GeckoView about the changes. It would make it easier to handle the conflict, as Fenix would handle both the code related to exitImmersiveMode and to the keyboard resizing.

An annoying point here is that GeckoView is just a View, not an application (I should use Activity in terms of Android development?). It would also make things awkward if GeckoView forces every application (Activity) to do something for the application global things? For example, if GeckoView provided the API to be notified WindowInsets change for the application, it will be a static global function looks like;

static public void GeckoView.addWindowInsetsListener(final Activity activity, OnApplyWindowInsetsListener listener);

And a call site of ViewCompat.setOnApplyWindowInsetsListener here will be GeckoView.addWindowInsetsListener(this, ...).

Is it okay? I would say that an ideal way of providing such APIs is probably to create a new repo for it, named Android ViewCompat utils or some such, then use the repo both in GeckoView and android-components. But it's somewhat opposed to our recent Fenix repo integration into mozilla-central. :/

I'm not 100% sure it would be the best option, but for now I couldn't come up with a better idea. Any thought?

Anyways, I have no great idea to solve this conflict, so tried to implement the API in GeckoView and it works as expected. I will upload it soon. Thanks!

Comment 21

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Add a new API to listen the root view's WindowInsets changes with multiple listeners. r?#geckoview-reviewers

Comment 22

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1831649 - Use GeckoView.addWindowInsetsListener/removeWindowInsetsListener in android-components. r?#android-reviewers

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6e96a1a01c24
Set inputMode="none" in test_event_target_radius.html to avoid showing the software keyboard. r=botond,m_kato
https://hg.mozilla.org/integration/autoland/rev/1e6ed8b78e8f
Parse `interactive-widget`. r=botond,emilio
https://hg.mozilla.org/integration/autoland/rev/4ddecb37950f
Detect the software keyboard height change. r=geckoview-reviewers,owlish
https://hg.mozilla.org/integration/autoland/rev/22538f7bdc88
Explicitly set android.windowSoftInputMode to GeckoView test runner. r=geckoview-reviewers,m_kato
https://hg.mozilla.org/integration/autoland/rev/2d0e7e662530
Propagate the software keyboard height change into nsPresContext. r=botond,geckoview-reviewers,m_kato
https://hg.mozilla.org/integration/autoland/rev/becfb23cd6b4
Expand the display size used as input to mobile viewport sizing to include the software keyboard area in overlays-content and resizes-visual mode. r=emilio,botond
https://hg.mozilla.org/integration/autoland/rev/a05aa1c75ece
Tweak the visual viewport size for resizes-visual and overlays-content modes. r=botond
https://hg.mozilla.org/integration/autoland/rev/b2718b2f730d
Tweak viewport stuff while the dynamic toolbar is moving for interactive-widget. r=botond
https://hg.mozilla.org/integration/autoland/rev/17fb50534fae
Trigger MobileViewportManager::RefreshViewportSize in the case where nsDocumentViewport boundaries changes but nsView boundaries unchanges. r=tnikkel,botond
https://hg.mozilla.org/integration/autoland/rev/279fdf36aac1
interactive-widget tests. r=botond
https://hg.mozilla.org/integration/autoland/rev/21235ab42839
Add a new API to listen the root view's WindowInsets changes with multiple listeners. r=geckoview-reviewers,owlish,ohall
https://hg.mozilla.org/integration/autoland/rev/501dc3134ae1
Use GeckoView.addWindowInsetsListener/removeWindowInsetsListener in android-components. r=android-reviewers,tthibaud

Comment 24

[Serban Stanca [:SerbanS]]

Backed out for causing multiple failures.

• Backout link

---

• Push with failures - AC-android-all opt
• Failure Log
• Failure line: file:///builds/worker/checkouts/gecko/mobile/android/android-components/components/ui/widgets/src/test/java/mozilla/components/ui/widgets/behavior/EngineViewScrollingBehaviorTest.kt:566:10: error: Class 'DummyEngineView' is not abstract and does not implement abstract member public abstract fun addWindowInsetsListener(key: String, listener: OnApplyWindowInsetsListener?): Unit defined in mozilla.components.concept.engine.EngineView

---

• Push with failures - focus-android-all opt
• Failure Log
• Failure line: file:///builds/worker/checkouts/gecko/mobile/android/focus-android/app/src/test/java/org/mozilla/focus/BrowserFragmentTest.kt:78:1: error: Class 'DummyEngineView' is not abstract and does not implement abstract member public abstract fun addWindowInsetsListener(key: String, listener: OnApplyWindowInsetsListener?): Unit defined in mozilla.components.concept.engine.EngineView

---

• Push with failures - android wpt failures
• Failure Log
• Failure line: TEST-UNEXPECTED-FAIL | /webdriver/tests/classic/element_click/scroll_into_view.py | test_partially_visible_does_not_scroll[9] - assert (490, 681) == approx((490 ± 1.0e+00, 1386 ± 1.0e+00))

---

• Push with failures - fenix-ui-test opt
• Failure Log
• Failure line: TEST-UNEXPECTED-FAIL | org.mozilla.fenix.ui.HistoryTest#verifyHistoryMenuWithHistoryItemsTest | androidx.test.espresso.PerformException: Error performing 'single click - At Coordinates: 1032, 1846 and precision: 16, 16' on view 'view.getContentDescription() is "Menu"'.

---

Also, please check this mochitests failure.

Comment 25

[Hiroyuki Ikezoe (:hiro)]

I've fixed the first two failure in comment 24.

For the wpt one, I can't run it locally due to bug 1908788.

For the fenix-ui-test one, I can't find any ways to run it locally, I've asked it in #Fenix, but I haven't got responses yet.

For the mochitest failure, I couldn't run it yesterday because of bug 1908713, now I can run it, but can't reproduce the failure locally. :/

Comment 26

[Hiroyuki Ikezoe (:hiro)]

For the wpt failure, the culprit is D204165 which just added android:windowSoftInputMode="stateUnspecified|adjustResize" to the TestRunner activity. The fact makes me suspect the test itself should have been unstable if we did run the test on GeckoView example or Fenix. :/

Comment 27

[Aaron Train [:aaronmt]]

(In reply to Hiroyuki Ikezoe (:hiro) from comment #25)

For the fenix-ui-test one, I can't find any ways to run it locally, I've asked it in #Fenix, but I haven't got responses yet.

#mobile-android-team (Slack) is where the team lives

Hi Hiroyuki, you will need to build Fenix, we have documentation here (Kotlin Espresso Tests) that run on a connected device (physical/emulator) either through Gradle/Android Studio:

https://firefox-source-docs.mozilla.org/mobile/android/fenix/UI-Tests.html

They can also be scheduled with ./mach try --preset firefox-android which will run {Fenix/AC/Focus} these suits on Firebase Test Lab on a variety of devices.

Matrix: https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/8459306338494513927/details?stepId=bs.3bc345a966ef66d5&testCaseId=1 (videos/logs/screenshots/test results)

As this broke a vast (~290 tests) tests prior to back-out in our suite, let's ensure these changes are tested. Please reach out for further questions.

For what it's worth it looks like the vast majority are hitting an issue where the address-bar is merged with the system UI (see attachment), viewport all wonky. A large number of our tests confirm the current URL and it can't detect it behind the system UI.

Comment 28

[Aaron Train [:aaronmt]]

Attachment: merged address-bar with system UI (Android).png

Comment 29

[Hiroyuki Ikezoe (:hiro)]

Thanks Aaron. Yeah, I had read the document and had built Fenix locally, but the command in the document, "/gradlew blah" didn't work locally. That's way I did ask.

Anyways, the screenshot is quite helpful to tell what's going on. It makes me suspect the recent Fenix's navbar change did something I wasn't expecting. All changesets here had done before the navbar change. :/

Comment 30

[Hiroyuki Ikezoe (:hiro)]

FWIW, the culprit caused fenix-ui-test failure is D210894. Odd.

Comment 31

[Hiroyuki Ikezoe (:hiro)]

I've fixed all failure in comment 24 other than the (intermittent) mochitest failure. That's said, while I was checking whether there's any other failure or not, I realized there's a perma failure in screenshots-arm job and interestingly it fails on m-c.

Aaron, is it worth filing a bug? Or it's a known failure?

Comment 32

[Aaron Train [:aaronmt]]

Thanks. I filed a bug about it.

For the Try above, using Fuzzy query=fenix arm did not schedule a complete UI test run. Can you run ./mach try --preset firefox-android to ensure a run against Fenix/Focus/AC is scheduled? Presets found in https://searchfox.org/mozilla-central/source/tools/tryselect/try_presets.yml#134

Comment 33

[Hiroyuki Ikezoe (:hiro)]

I figured out how the test_interactive_widget.html fails.

The test inserts a meta viewport tag in the top level content document at the beginning of the test, then in the first test case (for interactive-widget=resizes-content) we wait for a resize event triggered by appearing the software keyboard. But in a race condition, a resize event triggered by the meta viewport tag insertion is received while we are waiting for a resize event in the first test.

Comment 34

[Hiroyuki Ikezoe (:hiro)]

Now I've addressed all failure cases I know of here, I am waiting for reviewing bug 1908828.

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a39d8020f6ed
Set inputMode="none" in test_event_target_radius.html to avoid showing the software keyboard. r=botond,m_kato
https://hg.mozilla.org/integration/autoland/rev/60c098037541
Parse `interactive-widget`. r=botond,emilio
https://hg.mozilla.org/integration/autoland/rev/94b7afd117cb
Detect the software keyboard height change. r=geckoview-reviewers,owlish
https://hg.mozilla.org/integration/autoland/rev/bfbf94d9ecbc
Explicitly set android.windowSoftInputMode to GeckoView test runner. r=geckoview-reviewers,m_kato
https://hg.mozilla.org/integration/autoland/rev/5717fc0000f5
Propagate the software keyboard height change into nsPresContext. r=botond,geckoview-reviewers,m_kato
https://hg.mozilla.org/integration/autoland/rev/8b197fb54abc
Expand the display size used as input to mobile viewport sizing to include the software keyboard area in overlays-content and resizes-visual mode. r=emilio,botond
https://hg.mozilla.org/integration/autoland/rev/2b6b61d3db94
Tweak the visual viewport size for resizes-visual and overlays-content modes. r=botond
https://hg.mozilla.org/integration/autoland/rev/1a16365ecd2c
Tweak viewport stuff while the dynamic toolbar is moving for interactive-widget. r=botond
https://hg.mozilla.org/integration/autoland/rev/9fa674dd294e
Trigger MobileViewportManager::RefreshViewportSize in the case where nsDocumentViewport boundaries changes but nsView boundaries unchanges. r=tnikkel,botond
https://hg.mozilla.org/integration/autoland/rev/ee42ede208d0
interactive-widget tests. r=botond
https://hg.mozilla.org/integration/autoland/rev/951c1ac69475
Add a new API to listen the root view's WindowInsets changes with multiple listeners. r=geckoview-reviewers,owlish,ohall
https://hg.mozilla.org/integration/autoland/rev/fae4fd3feb6b
Use GeckoView.addWindowInsetsListener/removeWindowInsetsListener in android-components. r=android-reviewers,tthibaud

Comment 36

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/a39d8020f6ed
https://hg.mozilla.org/mozilla-central/rev/60c098037541
https://hg.mozilla.org/mozilla-central/rev/94b7afd117cb
https://hg.mozilla.org/mozilla-central/rev/bfbf94d9ecbc
https://hg.mozilla.org/mozilla-central/rev/5717fc0000f5
https://hg.mozilla.org/mozilla-central/rev/8b197fb54abc
https://hg.mozilla.org/mozilla-central/rev/2b6b61d3db94
https://hg.mozilla.org/mozilla-central/rev/1a16365ecd2c
https://hg.mozilla.org/mozilla-central/rev/9fa674dd294e
https://hg.mozilla.org/mozilla-central/rev/ee42ede208d0
https://hg.mozilla.org/mozilla-central/rev/951c1ac69475
https://hg.mozilla.org/mozilla-central/rev/fae4fd3feb6b