639546 - Create a framework for sending native events

Status: RESOLVED WONTFIX

(Testing Graveyard :: Mozmill, enhancement)

Description

[Andrew Halberstadt [:ahal]]

We want to be able to send os level events from within Mozmill which will allow us to do things such as dragging and dropping a file onto Firefox.

We want to take advantage of functions of a higher level than straight mouse click and move events (e.g use Gtk in Linux instead of X)

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Anything we want to have a look at the webdriver implementation? Has there been made a final decision yet?

Comment 2

[Andrew Halberstadt [:ahal]]

I worked on this a fair amount last summer (using the webdriver code) so I plan to start there and modify things as I go. I actually had some basic events working and integrated with Mozmill already (only keypress and click in linux).  It will take some work to get that merged into master and working properly in Minefield again though.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Andrew, we now have a good contact to Simon and Eran, who both are working on that code for webdriver.

Comment 4

[Andrew Halberstadt [:ahal]]

(In reply to comment #3)
> Andrew, we now have a good contact to Simon and Eran, who both are working on
> that code for webdriver.
Yep, I actually met up with them a couple times last summer ;)

Comment 5

[Andrew Halberstadt [:ahal]]

So Clint and I had decided that passing the absolute coordinates of the element to the os event system was the best way to go. I figured out (pretty sure I figured out) how to calculate this, but I'm unable to get Firefox to receive a click event.

I'm pretty sure that the reason is because we're sending the event to the GDK_WINDOW_ROOT instead of the Firefox window handle. In other words, I don't think you can just send a click event to a certain part of the screen and expect the application with focus to get that event (you have to explicitly pass the window handle in).

In Firefox 4 only the top level window has an associated window handle. Unfortunately ctypes can only call into C and to get this window handle you need XPCOM (C++). We'll have to either:
1) Figure out how to get this in javascript and pass it through ctypes (don't think this is possible, but may be wrong)
2) Scrap ctypes and use an XPCOM component
3) Create a shim library that gets called by ctypes and calls into C++

I would imagine that 3 would be the easiest.

Comment 6

[Andrew Halberstadt [:ahal]]

So 1) and 3) are impossible due to ctypes inability to pass through arbitrary objects.

I'm currently trying 4) Get window handle from the gdk side.

Comment 7

[Andrew Halberstadt [:ahal]]

For reference development of this bug is ongoing here: https://github.com/mozilla/pow-wow

I'm unassigning from myself for now, but feel free to ping me with any questions.

Comment 8

[Andrew Halberstadt [:ahal]]

Found in triage.