1961157 - Add support for synthesizing widget events in background tabs

Status: NEW

(Core :: DOM: UI Events & Focus Handling, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

While implementing support for widget events in the Remote Agent for Marionette and WebDriver BiDi, I encountered an issue where events dispatched to a background tab are incorrectly delivered to the foreground tab instead.

The root of the problem appears to be in how the event is passed into EventUtils: it's sent to the Chrome window associated with the background tab, but the synthesized event ends up being targeted at the currently active (foreground) tab rather than the intended one. This leads to unexpected side-effects and incorrect test behavior.
Impact

For WebDriver, interacting with elements in background tabs is a required feature. Some clients (e.g., Playwright) run multiple tests in parallel using separate Firefox containers within a single window. Since only one tab can be foregrounded at any time, forcing a tab switch before every widget event is not a viable solution.

Without a way to reliably synthesize widget events in background tabs, we cannot enable widget events as the default mechanism for either Marionette or WebDriver BiDi.

So two questions from my side for now:

• Is there a supported mechanism to direct widget events to a specific background tab within a Chrome window? We will most likely need that for key events in the future as well if those have to be widget events.

• Could APZ or another layer allow dispatching events to a tab regardless of its foreground state? Maybe via the focus manager test mode?

Comment 1

[Hiroyuki Ikezoe (:hiro)]

A caveat here is that APZ doesn't work for background tabs at all. Given that background tabs can not receive any user inputs.

Comment 2

[Hiroyuki Ikezoe (:hiro)]

To be more precise, APZ doesn't have any data for background tabs, thus there's nothing APZ can do for background tabs.

Changing this principal architecture requires significant amount of work. I can't think of any workarounds. I would say every test using user input events, such as wheel events, needs to be run in a foreground tab.

Comment 3

[Botond Ballo [:botond]]

There is some previous discussion of this topic in bug 1848957 comment 26 through comment 30.

I believe Hiro is right that we cannot do APZ processing of events targeting background tabs, for the reason discussed there:

(In reply to Botond Ballo [:botond] from comment #26)

I think trying to send events through APZ to a page in a background tab might cause other problems.

When APZ processes an event, it performs a hit-test using rendering-related state, such as the WebRender scene and other things derived from the display list. I think those pieces of rendering state are not created for background tabs.

Given that our main motivation for doing APZ processing of WebDriver events is to pass certain WPTs, perhaps we could enable the "widget" codepath for WPTs only, and not other clients of WebDriver?

Comment 4

[James Graham [:jgraham]]

I think there are legitimate reasons for real sites to want to test via the APZ codepath as well as web-platform-tests.

That said, if it's impossible for background tabs to actually use the APZ codepath, we need to figure out what's reasonably possible to implement, doesn't cause undue maintenance burden, and doesn't cause surprise to users.

From the implementation point of view, it sounds like going via APZ in background tabs is impossible. Does widget events in background tabs without APZ make any sense here, or are the two inextricably linked? If that was possible we could at least have a more unified codepath on the WebDriver side. Otherwise it seems like we'll need to continue to be able to dispatch content events in background tabs and widget events in foreground tabs. Also: how does this all interact with headless mode?

From a user perspective, we clearly need some way to offer APZ support for foreground tabs. That could either be by default, or we could make it e.g. a session-level capability (or pref) that people have to opt in to. The latter seems more conservative since you get the same behaviour regardless of focus.

Comment 5

[Hiroyuki Ikezoe (:hiro)]

(In reply to James Graham [:jgraham] from comment #4)

Does widget events in background tabs without APZ make any sense here, or are the two inextricably linked?

A key factor is this widget. With the current approach Henrik has worked, the widget is an nsIWidget instance corresponding to the top browser window, any events synthesized with the widget are delivered to documents in the foreground tab or to the browser window, they will never be delivered into background tabs regardless of APZ, e.g. without APZ on non-E10S mode. To be clearer, it's either the browser window document or APZ that determines where the given input event is delivered, and the browser document only handles the foreground documents, APZ does the same.

If that was possible we could at least have a more unified codepath on the WebDriver side. Otherwise it seems like we'll need to continue to be able to dispatch content events in background tabs and widget events in foreground tabs.

A approach I can think of is to extend EventUtils.synthesizeWheelAtPoint to accept a BrowsingContext, and if the BrowsingContext is for a document in a background tab, then

1. query the BrowserParent instance corresponding the BrowsingContext
2. invoke BrowserParent::SendMouseWheelEvent

so that the event will be delivered to the document in a background tab.

There are some concerns I can think of right now;

a) there's ApzAwareEventRoutingToChild call, we probably need to tweak it, unless multiple tests running at the same time both in foreground and background tabs will be messy, will be broken unexpectedly
b) events in background tabs will be only delivered to the originated document, e.g. given that a document has an out-of-process iframe, a synthesized event in the parent document will never be delivered to the iframe document even if the event position is on the iframe

Anyways, this approach will be implemented mostly in WebDriver and event handling implementations, APZ is less related. The above ApzAwareEventRoutingToChild is the only one relevant part.

Also: how does this all interact with headless mode?

Even on headless mode, the structure of widgets is same, there's no difference.

Comment 6

[Hiroyuki Ikezoe (:hiro)]

Moving into DOM:Events since I believe most of work for supporting background tabs will be in DOM and relevant parts. (I suppose DOM:Generic would be the best fit but there's no such component)

Comment 7

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Well, DOM: UI Events & Focus Handling component manages how to consider the destination of the user input (including synthesized events within the async mode).

I'm not so familiar with IPC part, but sounds like it's reasonable to change the APIs to have browsing context which overrides the redirect from the main process to a specific remote process.