702605 - Allow interaction events to be injected into the Event Queue

Status: RESOLVED WONTFIX

(Core :: DOM: UI Events & Focus Handling, defect)

Description

[David Burns :automatedtester]

Raising this from https://bugzilla.mozilla.org/show_bug.cgi?id=566671#c8

We need to have a mechanism to allow native events to be injected into the event queue so that we can run tests (Mozmill and Selenium) without Firefox having to be in focus. 

By allowing us to do this we can run tests on the same machine in parallel. Example of how Opera and Chromium do it are linked below[1].

[1]http://www.google.com/codesearch#OAMlx_jo-ck/src/chrome/browser/automation/testing_automation_provider.h&q=SendOSLevelKeyEventToTab&exact_package=chromium&type=cs&l=1145

[2] https://github.com/operasoftware/operadriver/blob/bc5946242ea2f9d50684a1136f39baa885dda3af/src/com/opera/core/systems/scope/services/ums/OperaExec.java


This simplifies how we test B2G, Desktop and Mobile in the future.

Comment 1

[Bill Walker [:bwalker] [@wfwalker]]

I believe this is a blocker for testing how we integrate Apps with B2G and therefore deserves pretty high priority.

Comment 2

[Olli Pettay [:smaug][[e-mail]]]

What events are you talking about? Events in event loop, or user interaction events?
I assume latter.
Is DOMWindowUtils not enough? There could be a driver (some component) which gets
messages from some other process and dispatches events to Firefox UI using DOMWindowUtils.

Comment 3

[Marlena Compton [:marlenac]]

You are correct in assuming user interaction events.  I've asked :automatedtester to comment on your 2nd question.

Comment 4

[David Burns :automatedtester]

I seem to have missed :smaug question! 

I will need to have a play with this. Currently Marionette and Selenium is doing this via synthesized events from the Selenium project.

I will need to try this out but I think it probably has been tried before by the Selenium project but I cant see any obvious attempt at it.

I will try do this ASAP!

Comment 5

[Olli Pettay [:smaug][[e-mail]]]

Um, looks like DOMWindowUtils::SendNativeKeyEvent won't work on all the platforms.
nsIWidget::SynthesizeNativeKeyEvent isn't just implemented in all the cases.

But would DOMWindowUtils::SendKeyEvent be enough?

Comment 6

[Eran Mes]

The short answer is: I'm not sure if DOMWindowUtils::SendKeyEvent is enough, but I have an extensive test suite to verify that.
Background: The Selenium project currently has native XPCOM components that emulate user input events by using an OS-specific mechanism to send those messages to the Firefox process (on Windows SendMessage is used, on Linux  GDK events are directly queued using gdk_event_put). We'd like to get rid of these native components since they require a lot of maintenance (re-building at every Firefox release), are platform-specific and very sensitive to focus issues.

The alternative we're after is a way to inject input events into Firefox's event queue directly. Is DOMWindowUtils::SendKeyEvent the right way to go? Alternatively, is there any special difficulty in implementing SendNativeKeyEvent for more platforms?

The main requirement is accuracy in emulating a user. Because of this requirement, synthesizing events at the DOM level is not good enough - things like CSS styles don't get applied properly and we have to manage focus switching between elements ourselves.

Comment 7

[Olli Pettay [:smaug][[e-mail]]]

SendKeyEvent doesn't put anything to event queue (event loop), but it does dispatch events
via the platform specific widget code, and whoever is calling SendKeyEvent is obviously
currently being executed in the event loop.

SendNativeKeyEvent gives the keyboard layout stuff etc. Do you need that? Is Selenium testing
various keyboard layouts? If not, or if not yet, perhaps SendKeyEvent is enough. It certainly isn't
a DOM level thing, but dispatches events in a way lower level - at the level where
Gecko gets events from OS normally.

Comment 8

[Simon Stewart]

For reference, the ChromeDriver already injects events directly into webkit's event queue. The path it follows in chrome starts here:

http://code.google.com/searchframe#OAMlx_jo-ck/src/chrome/browser/automation/testing_automation_provider.cc&exact_package=chromium&q=SendWebkitKeyEvent&type=cs&l=5397

The OperaDriver does something similar, though it's closed source so I can't point you at their implementation :)

The important thing in both implementations, and what's being suggested here, is that these events bypass the normal mechanisms provided by the OS to simulate events. This means that tests using webdriver can be run in parallel and without focus. 

A naive implementation of this on Windows would use Win32's SendInput API in order to emulate user interactions. This is deeply undesirable, as SendInput sends window events to whichever window currently happens to have OS focus, reducing machine utilization and rendering the machine being used to run the test totally useless while the test is running (and since devs often run the tests in the background while reading email/getting on with other work, this is a showstopper)

Comment 9

[Olli Pettay [:smaug][[e-mail]]]

(In reply to Simon Stewart from comment #8)
> For reference, the ChromeDriver already injects events directly into
> webkit's event queue. The path it follows in chrome starts here:
> 
> http://code.google.com/searchframe#OAMlx_jo-ck/src/chrome/browser/automation/
> testing_automation_provider.
> cc&exact_package=chromium&q=SendWebkitKeyEvent&type=cs&l=5397

So that does bypass all the low level keyboard layout related things, I think.


> The important thing in both implementations, and what's being suggested
> here, is that these events bypass the normal mechanisms provided by the OS
> to simulate events. This means that tests using webdriver can be run in
> parallel and without focus. 
DOMWindowUtils::SendKeyEvent doesn't do anything with focus. It just send key event to the
"widget".

You may want to set focusmanager.testmode to true for testing though. See bug 704583.

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I see mention of keyboard interactions, but we will also need support for mouse interactions against multiple parallel instances of Firefox.

Comment 11

[Olli Pettay [:smaug][[e-mail]]]

DOMWindowUtils have methods for that.

Comment 12

[Simon Stewart]

DOMWindowUtils has two major drawbacks that make it unsuitable:

1) The sendNative*Event methods aren't implemented on every platform (notably OS X)

2) The implementations that these delegate down to on Windows (the most common platform) are very fragile to concurrent user interaction with the keyboard.

The first is "simply" a matter of implementing the missing methods for other platforms. The latter is more serious. The key piece of implementation is here and looks similar to the equivalent code in webdriver:

http://mxr.mozilla.org/mozilla-central/source/widget/windows/nsWindow.cpp#5667

Our experience has been that the keyboard state is changed when the user hits modifier keys. 

There are an additional three other potential problems:

1) Handling multibyte input (that is, simulating i18n input when the keyboard mapping doesn't support that particular character set)

2) Handling "special" keys (eg: the function keys) My read of the code suggests that those aren't handled, but that's a quick skim through so I may have missed something.

3) I suspect that if the focus manager is on then keyboard input may be ignored, though perhaps the implementation of "OnChar", "OnKeyDown" and "OnKeyUp" avoids this?

Comment 13

[Olli Pettay [:smaug][[e-mail]]]

(In reply to Simon Stewart from comment #12)
> DOMWindowUtils has two major drawbacks that make it unsuitable:
> 
> 1) The sendNative*Event methods aren't implemented on every platform
> (notably OS X)

Why do you need sendNative*Event?


> 2) The implementations that these delegate down to on Windows (the most
> common platform) are very fragile to concurrent user interaction with the
> keyboard.
Use the testing mode in focusmanager?

Comment 14

[Simon Stewart]

We're attempting to emulate user level interactions as effectively as possible. Synthesizing events on the DOM isn't terribly accurate and is wildly complex (consider the events being fired during drag and drop. Between two frames. In different domains. In different versions of Firefox) Fortunately, browsers are adept at generating the correct sequence of JS events from the (considerably simpler) OS level ones.

The testing mode in focusmanager prevents any keyboard input from reaching the window. Even if it didn't, our experience on the selenium project suggests that the keyboard state gets read when the events are processed, and if the user is holding down (for example) the shift key that's registered in the events that are generated.

Comment 15

[Olli Pettay [:smaug][[e-mail]]]

Who is talking about synthesizing events on the DOM?
nsDOMWindowUtils::send*Event certainly dispatch events at way lower level
(pretty much at the same level as sendNative*Event)

The API provided my DOMWindowUtils is used for Firefox testing, including drag and drop between
frames (drag and drop testing is painful).

Comment 16

[Olli Pettay [:smaug][[e-mail]]]

Maybe I'm missing something here. What are the requirements?

(Requirements aren't quite clear since for example https://github.com/operasoftware/operadriver/blob/bc5946242ea2f9d50684a1136f39baa885dda3af/src/com/opera/core/systems/scope/services/ums/OperaExec.java certainly doesn't dispatch OS level events, but something in higher level)

Comment 17

[Eran Mes]

Thanks for the response, Olli. We will try nsDOMWindowUtils.sendKeyEvent and nsDOMWindowUtils.sendMouseEvent.
There is no need to start the emulation at the OS layer. The requirement is to allow interaction with the application-under-test in a way that is indistinguishable from a real user driving the browser. Synthesizing events at the DOM level definitely doesn't achieve this goal. All Selenium needs is a way to instruct the browser to dispatch events as if they were coming from the user. Generating these events at the OS level ("native events") is a guaranteed way to achieve this goal. However when there's a mechanism from the browser to dispatch events then there's no need to generate the events at the OS level.
As you've said (comment #7) dispatching event at the level where Gecko normally gets OS events from sounds just like the right place.

Comment 18

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Hey David, is this bug still relevant?

Comment 19

[David Burns :automatedtester]

actually with the work for Marionette it probably isn't required anymore as we have trusted events working properly for most things except for touch but there is a separate bug for that.