Routing hardware key events to keyboard app when an input field is focusing.

RESOLVED FIXED in Firefox 48

Status

()

defect
RESOLVED FIXED
5 years ago
3 years ago

People

(Reporter: jj.evelyn, Assigned: chunmin)

Tracking

unspecified
mozilla48
ARM
Gonk (Firefox OS)
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(feature-b2g:2.2r+, firefox48 fixed, b2g-v2.2r affected, b2g-master affected)

Details

(Whiteboard: [ft:conndevices])

Attachments

(11 attachments, 79 obsolete attachments)

227.47 KB, application/vnd.openxmlformats-officedocument.wordprocessingml.document
Details
53.71 KB, application/vnd.openxmlformats-officedocument.presentationml.presentation
Details
14.56 KB, patch
Details | Diff | Splinter Review
13.40 KB, patch
Details | Diff | Splinter Review
7.68 KB, patch
chunmin
: review+
Details | Diff | Splinter Review
3.17 KB, patch
chunmin
: review+
Details | Diff | Splinter Review
14.18 KB, patch
chunmin
: review+
Details | Diff | Splinter Review
37.66 KB, patch
chunmin
: review+
Details | Diff | Splinter Review
3.57 KB, patch
chunmin
: review+
Details | Diff | Splinter Review
1.67 KB, patch
chunmin
: review+
Details | Diff | Splinter Review
18.60 KB, patch
chunmin
: review+
Details | Diff | Splinter Review
Proposal:

    1. Let Keyboard app specify its key set for a layout in app's manifest
    2. System app uses mozContentEvent to notify Gecko when a keyboard layout is ready, as well as its handling key set. 
    3. Once step#2 is done, Gecko bypass those key strokes to Keyboard app directly.  (foreground app won't get them)
This should be an Gaia System:input mgmt + Gecko InputMethod API work.

We need to define how (3) works though, since the input app frame by default never takes focus, it doesn't make sense for it to receive normal DOM KeyboardEvent.
Component: General → Runtime
How about dispatch the KeyboardEvent to navigator.mozInputMethod or navigator.mozInputMethod.inputcontext?
(In reply to Kan-Ru Chen [:kanru] from comment #2)
> How about dispatch the KeyboardEvent to navigator.mozInputMethod or
> navigator.mozInputMethod.inputcontext?

So keyboard app will not receive KeyboardEvent from EventListener but from that dedicated API.
This avoid that "non-focused window should not receives DOM KeyboardEvent".

oh...good point.
what about adding a dedicated interface 'inputevent' in InputMethod with keyCode/charCode/Timestamp as attributes.
These attributes will be exposed to keyboard app via navigator and a listener required to listen to 'inputevent'
by listening to 'inputeventchange' we can take keyCode/charCode/Timestamp attributes and we can convert those based on IME loaded, then we can convert the charCode/keyCode as per IME and timestamp[here timestamp is required for multitap]
after converting the keyCode/charCode we can send them back to inputMethod to dispatch it to Editoreventlistener  via domWindowUtils.
(In reply to Marco Chen [:mchen] (PTO 12/10 ~ 12/12) from comment #3)
> So keyboard app will not receive KeyboardEvent from EventListener but from
> that dedicated API.
> This avoid that "non-focused window should not receives DOM KeyboardEvent".
> 
> oh...good point.

Would it violate the spec/convention etc if we reuse the KeyboardEvent instance for our API? If not I am totally for it.

By the way, we need to properly vet the API to ensure hardware behavior can be properly reflected on the target app. For example, |sendKey()| currently sends all three key events at once. We would need to change that to make sure keyboard app is enabled to relay the hw keyboard as close as possible.
(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TPE) (please ni?) from comment #7)
> (In reply to Marco Chen [:mchen] (PTO 12/10 ~ 12/12) from comment #3)
> > So keyboard app will not receive KeyboardEvent from EventListener but from
> > that dedicated API.
> > This avoid that "non-focused window should not receives DOM KeyboardEvent".
> > 
> > oh...good point.
> 
> Would it violate the spec/convention etc if we reuse the KeyboardEvent
> instance for our API? If not I am totally for it.
> 
> By the way, we need to properly vet the API to ensure hardware behavior can
> be properly reflected on the target app. For example, |sendKey()| currently
> sends all three key events at once. We would need to change that to make
> sure keyboard app is enabled to relay the hw keyboard as close as possible.

The repeat use case was covered, my mistake:
http://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/forms.js#523-526

The other detail to settle is probably only like key properties etc. But let's not worry about this given the we won't need this too much with the new proposal.
(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TPE) (please ni?) from comment #8) 
> The other detail to settle is probably only like key properties etc. But
> let's not worry about this given the we won't need this too much with the
> new proposal.

and Luke will update the new proposal here, ni him as a note. Simply speaking it's just chaining keyboard app into the event routing path of the current KeyboardEvent. [1] 
Active keyboard app will get events before the target element.

[1] https://wiki.mozilla.org/WebAPI/BrowserAPI/KeyboardEvent#Dispatch_KeyboardEvent_across_BrowserElements
Flags: needinfo?(lchang)
The new proposal is updated on Wiki [1]. Please refer to it for further discussion.

[1] https://wiki.mozilla.org/WebAPI/InputMethod_API_with_hardware_keyboard
Flags: needinfo?(lchang)
(In reply to Luke Chang [:lchang] from comment #10)
> The new proposal is updated on Wiki [1]. Please refer to it for further
> discussion.
> 
> [1] https://wiki.mozilla.org/WebAPI/InputMethod_API_with_hardware_keyboard

Hi XuLei,

This proposal introduces some APIs on mozInputMethod. It would be nice if you give us some feedback. Thanks a lot.
Flags: needinfo?(xyuan)
Looks good to me.

Just a comment about registering/unregistering a listener to handle the key events in keyboard app:

    mozInputMethod.inputcontext.addEventListener('keydown', function handler() {});
    mozInputMethod.inputcontext.removeEventListener('keydown', handler); 

It appears like a general key event handler to catch both software and hardware key events. I suggest we change the API a bit to tell that it only catches the hardware key events. It may be something like that:

    mozInputMethod.inputcontext.hardwareinput.addEventListener('keydown', function handler() {});
    mozInputMethod.inputcontext.hardwareinput.removeEventListener('keydown', handler);
Flags: needinfo?(xyuan)
To my knowledge, "keydown" and "keyup" usually stand for hardware key events so I'm not sure should we need to clarify it in particular.

Hi Tim, what do you think?
Flags: needinfo?(timdream)
(In reply to Yuan Xulei [:yxl] from comment #12)
> It appears like a general key event handler to catch both software and
> hardware key events. I suggest we change the API a bit to tell that it only
> catches the hardware key events. It may be something like that:

Yeah. InputContext, which inherits EventTarget, only dispatches selectionchange and surroundingtextchange events currently. I don't think adding keydown and keyup event will confuse people. What do you mean?
Flags: needinfo?(timdream) → needinfo?(xyuan)
The usage of `mozInputMethod.inputcontext.addEventListener('keydown'...)` implies it will listen to all key events regardless of whether a hardware a keyboard is attached or whether the events are generated by mozInputMethod.inputcontext.sendKey(...).

When we don't have a physical keyboard and use software keyboard only, should `mozInputMethod.inputcontext.addEventListener('keydown'...)` catch the `keydown` events from sendKey()?

If so, just forget about all my comments :-)
If not, I suggest change the API to make it clear that virtual key events won't be listened.
Flags: needinfo?(xyuan)
(In reply to Yuan Xulei [:yxl] from comment #15)
> The usage of `mozInputMethod.inputcontext.addEventListener('keydown'...)`
> implies it will listen to all key events regardless of whether a hardware a
> keyboard is attached or whether the events are generated by
> mozInputMethod.inputcontext.sendKey(...).
> 
> When we don't have a physical keyboard and use software keyboard only,
> should `mozInputMethod.inputcontext.addEventListener('keydown'...)` catch
> the `keydown` events from sendKey()?
> 
> If so, just forget about all my comments :-)
> If not, I suggest change the API to make it clear that virtual key events
> won't be listened.

Hi Mr.Yuan Xulei,

For the question,
> should `mozInputMethod.inputcontext.addEventListener('keydown'...)` catch
> the `keydown` events from sendKey()?

Yes i think it will listen soft key also when sendKey()is called, which will lead to never ending loop.
the solution for this is event Location comparison. for all soft key the event location is Zero[DOM_KEY_LOCATION_STANDARD] and for all hardkey the event location will be 4[DOM_KEY_LOCATION_MOBILE]
[https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent]
based on this we can separate which key is soft and which key is from hardware.So we can process the hardkey event only one time and send back the processed key using sendKey()
(In reply to Luke Chang [:lchang] from comment #10)
> The new proposal is updated on Wiki [1]. Please refer to it for further
> discussion.
> 
> [1] https://wiki.mozilla.org/WebAPI/InputMethod_API_with_hardware_keyboard

I discussed with Luke about the proposal a bit and was wondering about this particular scenario:
1. mozInputMethod.inputcontext dispatches the hardware keyXXX event to KeyboardApp.
2. KeyboardApp decides to process the event, and calls evt.preventDefault in that event handler.
3. KeyboardApp sends the (processed) key to the "client app" by calling mIM.inputcontext.sendKey(), and gets a Promise.
4. "client app" rejects the key (maybe by its keydown event's preventDefault(), etc) and KeyboardApp's Promise gets rejected.

5. At this time, should we allow Gecko to be notified about such rejection? What's the possible use case, and is there anything which we might need to do, but which requires knowledge from Gecko and not just KBApp?
Note that currently it's not possible to notify Gecko because the synchronous event handler at steps 1 and 2 is already over.
Instead of adding a listener to Keydown in mozInputMethod.inputcontext, why can't we use another new interface in mozInputMethod[ex:inputHardwareEvent] which sends keycode/charcode to gaia for further processing.
In this case we have to implement the interface and make a way via forms.js, Keyboard.jsm, MozKeyboard.js 
 
we will add a listener in forms.js for keypress, So we will be listening to only editor hardware keypress and sending them to gaia for further processing. 


Currently we have a channel to editor via mozInputMethod from gaia [sendKey() soft touch]
with this new approach we will be adding reverse channel for hardware keys which gives the event info to gaia and processes then sends back to mozInputMethod by existing path [sendKey()]

With this approach we will not disturb the existing behavior, we will just redirect the hardware events only when editor is focused. 
Mr.Tim, Mr.Yuan, what is your opinion on this?
Flags: needinfo?(xyuan)
Flags: needinfo?(timdream)
(In reply to Yuan Xulei [:yxl] from comment #15)
> The usage of `mozInputMethod.inputcontext.addEventListener('keydown'...)`
> implies it will listen to all key events regardless of whether a hardware a
> keyboard is attached or whether the events are generated by
> mozInputMethod.inputcontext.sendKey(...).
> 
> When we don't have a physical keyboard and use software keyboard only,
> should `mozInputMethod.inputcontext.addEventListener('keydown'...)` catch
> the `keydown` events from sendKey()?
> 
> If so, just forget about all my comments :-)
> If not, I suggest change the API to make it clear that virtual key events
> won't be listened.

No, sendKey() has nothing directly related to keyEvent handling in the proposal. Calling sendKey() does not make inputcontext dispatch KeyboardEvent generated from hardware keyboard since the event is not generated from hardware keyboard.
(In reply to Jayachandra Yakasiri from comment #18)

I have no opinion on the implementation detail. Let's only worry about if the API fits all use cases for now.
Flags: needinfo?(timdream)
(In reply to Jayachandra Yakasiri from comment #18)
> With this approach we will not disturb the existing behavior, we will just
> redirect the hardware events only when editor is focused. 

PS I think the current spec also "disturb" the current behavior when the IM API is active. IM API is not active (thus will not redirect keyboard events) when the keyboard is not shown.
(In reply to Jayachandra Yakasiri from comment #18)
> Instead of adding a listener to Keydown in mozInputMethod.inputcontext, why
> can't we use another new interface in mozInputMethod[ex:inputHardwareEvent]
> which sends keycode/charcode to gaia for further processing.
> In this case we have to implement the interface and make a way via forms.js,
> Keyboard.jsm, MozKeyboard.js 
>  
> we will add a listener in forms.js for keypress, So we will be listening to
> only editor hardware keypress and sending them to gaia for further
> processing. 
> 
> 
> Currently we have a channel to editor via mozInputMethod from gaia
> [sendKey() soft touch]
> with this new approach we will be adding reverse channel for hardware keys
> which gives the event info to gaia and processes then sends back to
> mozInputMethod by existing path [sendKey()]
It is another user case that might be considered in the future. Since the data exchange between the editor and gaia keyboard app is async. This reservse channel can only be used as a way to post process hardware key events, while in this bug, we need a way to preprocess the events.
Flags: needinfo?(xyuan)
Whiteboard: [red-tai]
Luke, please update our offline discussion here to clarify the issue Xulei brought up. Thank you all.
Flags: needinfo?(lchang)
(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TPE) (please ni?) from comment #19)
> (In reply to Yuan Xulei [:yxl] from comment #15)
> > When we don't have a physical keyboard and use software keyboard only,
> > should `mozInputMethod.inputcontext.addEventListener('keydown'...)` catch
> > the `keydown` events from sendKey()?
> >
> > If so, just forget about all my comments :-)
> > If not, I suggest change the API to make it clear that virtual key events
> > won't be listened.
>
> No, sendKey() has nothing directly related to keyEvent handling in the
> proposal. Calling sendKey() does not make inputcontext dispatch
> KeyboardEvent generated from hardware keyboard since the event is not
> generated from hardware keyboard.

If I understand correctly, the concern Xulei mentioned is about the rationality of the API's naming. Since the "mozInputMethod.inputcontext" is regarded as an agent of the event target and "sendKey()" intends to send a standard DOM KeyboardEvent, it may confuse users whether "mozInputMethod.inputcontext.addEventListener('keydown')" can receive a "keydown" event generated by "sendKey()".

For now, a key event generated by "sendKey()" doesn't follow the "mozbrowserbeforekeyXXX" proposal [1] and contains only partial properties in its event object (e.g. "event.key" is absent). That means these software key events have already been considered to be a non-standard KeyboardEvent. So I second Tim's opinion that we don't dispatch these software key events to "inputcontext" at this moment.

For long-term, however, I think we do need to define a "hardwareinput" object to deal with some issues such as supporting multiple hardware input devices, listening whether devices are connected or not, distinguishing the events' source, detecting the type of devices, and so on. We may change the API accordingly until the "hardwareinput" is well defined. Fortunately, the current API proposed in this bug will be certified only. The future changes should not take much effort. Any thoughts?

[1] https://wiki.mozilla.org/WebAPI/BrowserAPI/KeyboardEvent
Flags: needinfo?(lchang)
Here i am attaching a document for Hardware key event rerouting to Keyboard app proposal.
Please review and let us know your comments.
Attachment #8589584 - Flags: review?(mchen)
Comment on attachment 8589584 [details]
This is proposal for Hardkey event[12 keys] routing to Keyboard app

Hi,

Thanks for sharing this diagram I will check whether it is align with what we discussed in last face to face meeting.

But according to some business travel next week, I might need some days to do it.
Attachment #8589584 - Flags: review?(mchen) → feedback?(mchen)
Comment on attachment 8589584 [details]
This is proposal for Hardkey event[12 keys] routing to Keyboard app

(for ppl with MS Word installed)

https://docs.google.com/viewer?url=https://bugzilla.mozilla.org/attachment.cgi?id=8589584
(In reply to Jayachandra Yakasiri from comment #25)
> Created attachment 8589584 [details]
> This is proposal for Hardkey event[12 keys] routing to Keyboard app
> 
Hi,

Thanks for taking the effort to come out this proposal. Great!
And I have some high level questions for understanding your proposal first.

Q1: In the diagram, does the arrow from Keyboard app to content app mean 
  a. will Keyboard event be sent to content app by keyboard app?
  b. or it just mean the logic sequence of apps to receive Keyboard event not a real event flow? (because the real flow now for touch case would be keyboard app process -> b2g process -> content process)

  I even don't know the arrow mean the original keyboard event pressed by user or the new key generated by IME of keyboard app. 

Suggestion: would suggest you have two diagrams, one is for logical flow for sequence of apps to receive keys. And the other is the real event flow between processes. Or I will confuse the meaning of what you want to show.

Q2: It seems that you only include one possible direction in you steps - "keyboard app consumes the key". 

Suggestion: would suggest you show up what other possible flows you considered as well. 
  ex: 1. system app default-prevent this event then no key will be sent to keyboard app.
      2. keyboard app consumes this key then no key will be sent to content process but will generate a new key for content process.
      3. keyboard app doesn't consume this key so it should be sent to content process by b2g process.

Q3: I think in your proposal, there will be a new API in MozInputContext for web app to retrieve keys. right?

Q4: What is the event queue you mean in the "steps"? could you show us where is it in the code?
Flags: needinfo?(yjc92385)
(In reply to Marco Chen [:mchen] from comment #28)
> (In reply to Jayachandra Yakasiri from comment #25)
> > Created attachment 8589584 [details]
> > This is proposal for Hardkey event[12 keys] routing to Keyboard app
> > 
> Hi,
> 
> Thanks for taking the effort to come out this proposal. Great!
> And I have some high level questions for understanding your proposal first.
> 
> Q1: In the diagram, does the arrow from Keyboard app to content app mean 
>   a. will Keyboard event be sent to content app by keyboard app?
>   b. or it just mean the logic sequence of apps to receive Keyboard event
> not a real event flow? (because the real flow now for touch case would be
> keyboard app process -> b2g process -> content process)
> 
>   I even don't know the arrow mean the original keyboard event pressed by
> user or the new key generated by IME of keyboard app. 
> 
> Suggestion: would suggest you have two diagrams, one is for logical flow for
> sequence of apps to receive keys. And the other is the real event flow
> between processes. Or I will confuse the meaning of what you want to show.
> 
> Q2: It seems that you only include one possible direction in you steps -
> "keyboard app consumes the key". 
> 
> Suggestion: would suggest you show up what other possible flows you
> considered as well. 
>   ex: 1. system app default-prevent this event then no key will be sent to
> keyboard app.
>       2. keyboard app consumes this key then no key will be sent to content
> process but will generate a new key for content process.
>       3. keyboard app doesn't consume this key so it should be sent to
> content process by b2g process.
> 
> Q3: I think in your proposal, there will be a new API in MozInputContext for
> web app to retrieve keys. right?
> 
> Q4: What is the event queue you mean in the "steps"? could you show us where
> is it in the code?

Please find my inline comments for the questions below

Q1: In the diagram, does the arrow from Keyboard app to content app mean 
  a. will Keyboard event be sent to content app by keyboard app?
  b. or it just mean the logic sequence of apps to receive Keyboard event not a real event flow? (because the real flow now for touch case would be keyboard app process -> b2g process -> content process)
Ans:---------------------
 Here the arrow indicates just giving the text to content process via b2g process [normal flow by calling inputContext.sendKey()which sends the typed text via domWindowUtils].For touch case also it is same, i.e after finding the target element it just sends it to editor of corresponding content process via domWindowUtils

https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/forms.js#511
---------------------
  
I even don't know the arrow mean the original keyboard event pressed by user or the new key generated by IME of keyboard app. 

Ans:---------------------
Once the charcode/keycode conversion is done by IME, we use inputContext.sendKey() to send the converted charcode to Editor[normal way of sending] via domWindowUtils where it creates a new WidgetKeyboardEvent and dispatches the event

https://dxr.mozilla.org/mozilla-central/source/dom/base/nsDOMWindowUtils.cpp#1177
---------------------

Suggestion: would suggest you have two diagrams, one is for logical flow for sequence of apps to receive keys. And the other is the real event flow between processes. Or I will confuse the meaning of what you want to show.

Ans:---------------------
Ok, I will attach flow diagrams soon.
---------------------
Q2: It seems that you only include one possible direction in you steps - "keyboard app consumes the key". 

Suggestion: would suggest you show up what other possible flows you considered as well. 
  ex: 1. system app default-prevent this event then no key will be sent to keyboard app.
Ans:---------------------
 Since we are rerouting the events NS_KEY_BEFORE_DOWN and NS_KEY_BEFORE_UP to Keyboard app, this scenario will not exists.
---------------------
      2. keyboard app consumes this key then no key will be sent to content process but will generate a new key for content process.
Ans:---------------------
 Yes, Once Keyboard consumes the events[NS_KEY_BEFORE_DOWN and NS_KEY_BEFORE_UP]we will not send next events in the series to content process, we will send a new event only via domWindowUtils where it creates a new WidgetKeyboardEvent and dispatches the event to corresponding editor of content process.
---------------------
      3. keyboard app doesn't consume this key so it should be sent to content process by b2g process.

Ans:---------------------
I have a question here

Is there any scenario where the keyboard app is not consuming the event even though the editor is focused and the content process want to use it for different purpose? 


 If Yes, then if Keyboard app is not consuming the event then we have to send it back to content process via b2g process, but there are technical problems involved in this scenario

Problem 1: Since IPC involved in this, All the events are asynchronous, so we will not come to know whether the event [NS_KEY_BEFORE_DOWN]is consumed by keyboard app or not before it sends the other events in the series[NS_KEY_DOWN, NS_KEY_AFTER_DOWN, NS_KEY_BEFORE_UP, NS_KEY_UP, NS_KEY_AFTER_UP]. 

Problem 2: Since IPC involved and we are firing a custom event from DOM input context, calling event.preventdefault() in keyboard app will not prevents the event series.

Currently we are assuming that Keyboard app will consume the event. We implemented in such a way that the event will be given to 3rd party keyboard app also if it is active. So I could not find any scenario where a 3rd party app used the event for other purpose event though the editor is focused and keyboard is active. 
---------------------

Q3: I think in your proposal, there will be a new API in MozInputContext for web app to retrieve keys. right?

Ans:---------------------
 No, once we get the event in Tab child of Keyboard Process, we used domWindowUtils to dispatch the event
utils->SendKeyEventToKeyboard(event.message,event.keyCode,event.charCode,event.location,&ignored);
and we added an EventListener for mozbrowserbeforekeydown ,sending this event to Keyboard app as a part of MozInputContext              this._window.addEventListener("mozbrowserbeforekeydown", this);

---------------------
Q4: What is the event queue you mean in the "steps"? could you show us where is it in the code?

Ans:---------------------

This is a proposal to overcome the problems mentioned in point3 in Question2. (if there is a scenario where the keyboard app is not consuming the event even though the editor is focused and keyboard is active).

Then the EventQueue proposal is for queueing the events for making the events synchronous due to the below problem when the keyboard app does not consumes the event.

Problem 1: Since IPC involved in this, All the events are asynchronous, so we will not come to know whether the event [NS_KEY_BEFORE_DOWN]is consumed by keyboard app or not before it sends the other events in the series[NS_KEY_DOWN, NS_KEY_AFTER_DOWN, NS_KEY_BEFORE_UP, NS_KEY_UP, NS_KEY_AFTER_UP]. 
 
 

This event queue will be linked to EventStateManager and makes the events synchronous. But there will be a delay in getting the event by content process.
Is there any other better solution for the problem mentioned above?
---------------------
Flags: needinfo?(yjc92385) → needinfo?(mchen)
(In reply to Jayachandra Yakasiri from comment #25)
> Created attachment 8589584 [details]
> This is proposal for Hardkey event[12 keys] routing to Keyboard app
> 
> Here i am attaching a document for Hardware key event rerouting to Keyboard
> app proposal.
> Please review and let us know your comments.

Hi, Jayachandra, 
I think your proposal has some differences with our proposal mentioned above[1]. In your diagram, it seems that the NS_KEY_BEFORE_DOWN and NS_KEY_BEFORE_UP will be dispatched to keyboard app. AFAIK, NS_KEY_BEFORE_DOWN/UP event should only be received by shell.html or system/index.html(The goal of this design is to enable b2g to handle the keyXXX event before or after dispatching keyXXX event to target app, i.e., some certain keys[Power, VolumeUp, ...] should be handled by b2g only). In our proposal[1], we will dispatch Keydown or Keyup event to keyboard app directly if it's active.


[1] https://wiki.mozilla.org/WebAPI/InputMethod_API_with_hardware_keyboard#Flowchart
Comment on attachment 8589584 [details]
This is proposal for Hardkey event[12 keys] routing to Keyboard app

(In reply to Chun-Min Chang[:chunmin] from comment #30)
> AFAIK,
> NS_KEY_BEFORE_DOWN/UP event should only be received by shell.html or
> system/index.html(The goal of this design is to enable b2g to handle the
> keyXXX event before or after dispatching keyXXX event to target app

More in general, only mozBrowser-embedder (and the target is a moz iframe embedded by it) can receive NS_KEY_BEFORE_DOWN and others. Please refer to [1].

In this case the keyboard app is not a mozBrowser-embedder and the target is not it's mozBrowser-embedded i-frame.

[1] https://wiki.mozilla.org/WebAPI/BrowserAPI/KeyboardEvent#Dispatch_KeyboardEvent_across_BrowserElements
Flags: needinfo?(mchen)
Attachment #8589584 - Flags: feedback?(mchen)
Flags: needinfo?(yjc92385)
> I think your proposal has some differences with our proposal mentioned
> above[1]. In your diagram, it seems that the NS_KEY_BEFORE_DOWN and
> NS_KEY_BEFORE_UP will be dispatched to keyboard app. AFAIK,
> NS_KEY_BEFORE_DOWN/UP event should only be received by shell.html or
> system/index.html(The goal of this design is to enable b2g to handle the
> keyXXX event before or after dispatching keyXXX event to target app, i.e.,
> some certain keys[Power, VolumeUp, ...] should be handled by b2g only). In
> our proposal[1], we will dispatch Keydown or Keyup event to keyboard app
> directly if it's active.

As Mr.Marco mentioned, in this case both target and keyboard iframes are embedded iframes with respect to system app.
In this case there is no use of giving NS_KEY_BEFORE_DOWN/UP events of input keys to embedder iframe[system]? In Power, VolumeUp case system may require those events for fixing the behaviour before it consumed by target but do we really require this behaviour for input keys?
Flags: needinfo?(yjc92385) → needinfo?(cchang)
(In reply to Jayachandra Yakasiri from comment #32)
> As Mr.Marco mentioned, in this case both target and keyboard iframes are
> embedded iframes with respect to system app.
> In this case there is no use of giving NS_KEY_BEFORE_DOWN/UP events of input
> keys to embedder iframe[system]? 

1. What I mentioned is "NS_KEY_BEFORE_DOWN/UP should be sent to EMBEDDER ONLY not embedded iframe". In your case, both of keyboard moz-iframe and target app moz-iframe should not receive these keys.

> In Power, VolumeUp case system may require those events for fixing the behaviour before it consumed > by target but do we really require this behaviour for input keys?

2. This mechanism is provided for all of keyboard events and doesn't have different behaviors for different kind of keyboard events.
Flags: needinfo?(yjc92385)
Component: Runtime → DOM: Events
Product: Firefox OS → Core
(In reply to Jayachandra Yakasiri from comment #32)
> As Mr.Marco mentioned, in this case both target and keyboard iframes are
> embedded iframes with respect to system app.
+--------------------------------+
| shell.html                     |
|                                |
|  +---------------------------+ |
|  | system/index.html         | |
|  |                           | |
|  | +----------+   +--------+ | |
|  | | keyboard |   | target | | |
|  | | app      |   | app    | | |
|  | +----------+   +--------+ | |
|  |                           | |
|  +---------------------------+ |
|                                |
+--------------------------------+

Yes, so both keyboard app and target app should not receive NS_KEY_BEFORE_DOWN or NS_KEY_BEFORE_UP events.

> In this case there is no use of giving NS_KEY_BEFORE_DOWN/UP events of input
> keys to embedder iframe[system]? In Power, VolumeUp case system may require
> those events for fixing the behaviour before it consumed by target but do we
> really require this behaviour for input keys?
I may not explain it well. Here[1] gives a good example to show why we need NS_KEY_BEFORE_DOWN/UP events.

As the mention from comment #33:
> This mechanism is provided for all of keyboard events and doesn't have
> different behaviors for different kind of keyboard events.
This is not designed to fix some special cases, but a general behaviors for all keyboard events. Please refer the bug 989198[2]

[1] https://groups.google.com/forum/#!topic/mozilla.dev.webapi/xPGXAHEiVmg
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=989198
Flags: needinfo?(cchang)
I updated the proposal as per the comments. Please review it.
Attachment #8589584 - Attachment is obsolete: true
Flags: needinfo?(yjc92385) → needinfo?(cchang)
Attachment #8595735 - Flags: review?(mchen)
(In reply to Jayachandra Yakasiri from comment #35)
> Created attachment 8595735 [details]
> This is updated proposal for Hardkey event[12 keys] routing to Keyboard app
> 
> I updated the proposal as per the comments. Please review it.
It looks better! However, these are some details you should notice:
1. Actually, beforeKey*, key*, afterKey* are all dispatched by EventDispatcher::Dispatch. In attachment 8595735 [details], only key* events are dispatched by EventDispatcher::Dispatch.
2. Does key* events will be dispatched from keyboard app to target app directly? I think the keyboard app should ask b2g to dispatch the key* event to target app
3. It will be better if you add the case that keyboard app call key*.preventDefault(). In this case, the key* event should not be dispatched to target app.
Flags: needinfo?(cchang)
(In reply to Chun-Min Chang[:chunmin] from comment #36)

> 1. Actually, beforeKey*, key*, afterKey* are all dispatched by
> EventDispatcher::Dispatch. In attachment 8595735 [details], only key* events
> are dispatched by EventDispatcher::Dispatch.
Yes... All key* events are dispatched via Event Dispatcher, but to make the diagram simple i did not included it for before and after key events.
> 2. Does key* events will be dispatched from keyboard app to target app
> directly? I think the keyboard app should ask b2g to dispatch the key* event
> to target app
Yes... I mentioned it in the diagram with dotted lines and with a text, which says it is not a direct dispatch.
> 3. It will be better if you add the case that keyboard app call
> key*.preventDefault(). In this case, the key* event should not be dispatched
> to target app.
I did not understood your point[3].
When Keyboard app consumes the event, it will call event.Preventdefault(). and creates an another event to give it to target app editor[not directly but via b2g], this is same behavior for touch case also.[it calls sendkey()]
https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/forms.js#511
Flags: needinfo?(cchang)
(In reply to Jayachandra Yakasiri from comment #37)
> I did not understood your point[3].
> When Keyboard app consumes the event, it will call event.Preventdefault().
> and creates an another event to give it to target app editor[not directly
> but via b2g], this is same behavior for touch case also.[it calls sendkey()]
> https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/forms.js#511
If keyboard app call key* event.preventDefault(), it means that it will handle this event. Yes, the target app will still receive a event after keyboard handle it, but it's not the original event. We should avoid dispatch the original event to target app, otherwise the target app will receive two events. One is the event synthesized by keyboard app. Another is the original event. For example, if we choose Japanese as the input method in keyboard app and character 'A' will be converted to 'ち'. When keyboard app is active and we press down the character 'A' on hardware keyboard: 
1) Before_Keydown_event(A) will be dispatched to system app 
2) Keydown_event(A) will be dispatched to system app
3) Keydown_event(A) will be dispatched to keyboard app
  - 1) 'A' is converted to 'ち'
  - 2) 'ち' will be sent to target app via b2g
4) Keydown_event(A) will be dispatched to target app
  ---> This event should be cancelled because the key 'A' is already converted to 'ち'
  ---> This event will be received by target app if we don't prevent the event dispatching in the original key event chain.
5) After_Keydown_event(A) will be dispatched to system app
Flags: needinfo?(cchang)
> If keyboard app call key* event.preventDefault(), it means that it will
> handle this event. Yes, the target app will still receive a event after
> keyboard handle it, but it's not the original event. We should avoid
> dispatch the original event to target app, otherwise the target app will
> receive two events. One is the event synthesized by keyboard app. Another is
> the original event. For example, if we choose Japanese as the input method
> in keyboard app and character 'A' will be converted to 'ち'. When keyboard
> app is active and we press down the character 'A' on hardware keyboard: 
> 1) Before_Keydown_event(A) will be dispatched to system app 
> 2) Keydown_event(A) will be dispatched to system app
> 3) Keydown_event(A) will be dispatched to keyboard app
>   - 1) 'A' is converted to 'ち'
>   - 2) 'ち' will be sent to target app via b2g
> 4) Keydown_event(A) will be dispatched to target app
>   ---> This event should be cancelled because the key 'A' is already
> converted to 'ち'
>   ---> This event will be received by target app if we don't prevent the
> event dispatching in the original key event chain.
> 5) After_Keydown_event(A) will be dispatched to system app

As per the proposal attached, The NS_KEY_DOWN event will be given to Keyboard based on Editor is focused or not. [it means we are rerouting the NS_KEY_DOWN event to keyboard via IPC, so the system app or target app will not get this event]
So Keydown_event(A) will not be dispatched to system app or target app if editor is focused, but after keyboard converts it to 'ち' it will create a custom event and dispatches it to target app via system app. 
So as per the proposal only one event will be dispatched to target app.
Flags: needinfo?(mchen)
Flags: needinfo?(cchang)
(In reply to Jayachandra Yakasiri from comment #39)
> So Keydown_event(A) will not be dispatched to system app or target app if
> editor is focused

1. Do you mean that system app or target app will not be added to event chain if keyboard app is active?
2. Is the following events order your expectation?

1) Before_Keydown_event(A) will be dispatched to system app 
2) Keydown_event(A) will be dispatched to keyboard app
   - 1) 'A' is converted to 'ち'
   - 2) 'ち' will be sent to target app via b2g
3) After_Keydown_event(A) will be dispatched to system app
Flags: needinfo?(cchang)
(In reply to Chun-Min Chang[:chunmin] from comment #40)
> 1. Do you mean that system app or target app will not be added to event
> chain if keyboard app is active?
> 2. Is the following events order your expectation?
> 
> 1) Before_Keydown_event(A) will be dispatched to system app 
> 2) Keydown_event(A) will be dispatched to keyboard app
>    - 1) 'A' is converted to 'ち'
>    - 2) 'ち' will be sent to target app via b2g
> 3) After_Keydown_event(A) will be dispatched to system app

Yes... This is the proposal mentioned in the attachment.
(In reply to Jayachandra Yakasiri from comment #41)
> (In reply to Chun-Min Chang[:chunmin] from comment #40)
> > 1. Do you mean that system app or target app will not be added to event
> > chain if keyboard app is active?
> > 2. Is the following events order your expectation?
> > 
> > 1) Before_Keydown_event(A) will be dispatched to system app 
> > 2) Keydown_event(A) will be dispatched to keyboard app
> >    - 1) 'A' is converted to 'ち'
> >    - 2) 'ち' will be sent to target app via b2g
> > 3) After_Keydown_event(A) will be dispatched to system app
> 
> Yes... This is the proposal mentioned in the attachment.

stating two opinions:

1. In order to keep event dispatch consistent with the behaviour when the pref["dom.beforeAfterKeyboardEvent.enabled", true] is disabled, key events should be still dispatched in the parent process. That is, the system app should receive NS_KEY_DOWN event. Therefore, system app should be added into event chain.

1) Before_Keydown_event(A) will be dispatched to system app
2) Keydown_event(A) will be dispatched to system app
3) Keydown_event(A) will be dispatched to keyboard app if it's active
...
...
n) After_Keydown_event(A) will be dispatched to system app


2. If target app is not in event chain, how will the following case be handled?
  - 1) keyboard app is active, so the NS_KEY_DOWN event will be dispatched to it.
  - 2) keyboard app does not want to handle this NS_KEY_DOWN event, so it doesn't call preventDefault()
    --> The NS_KEY_DOWN should be dispatched to target app if keyboard app does not process it.
    --> However, the target app is not in event chain, 
    --> so NS_KEY_DOWN event can not be dispatched to target app

Although the NS_KEY_DOWN event can be dispatched to target app through the path that keyboard app use, but personally I think its weird to do that. If keyboard app doesn't want to process this key event, it should not forward this event. I prefer remaining the target app in event chain and determine whether we should dispatch the keyboard event to it by checking the calling of event.preventDefault()
(In reply to Chun-Min Chang[:chunmin] from comment #42)

> stating two opinions:
> 
> 1. In order to keep event dispatch consistent with the behaviour when the
> pref["dom.beforeAfterKeyboardEvent.enabled", true] is disabled, key events
> should be still dispatched in the parent process. That is, the system app
> should receive NS_KEY_DOWN event. Therefore, system app should be added into
> event chain.
> 
> 1) Before_Keydown_event(A) will be dispatched to system app
> 2) Keydown_event(A) will be dispatched to system app
> 3) Keydown_event(A) will be dispatched to keyboard app if it's active
> ...
> ...
> n) After_Keydown_event(A) will be dispatched to system app
> 
'keydown' is intended for mozbrowser-embedded iframe not for mozbrowser-embedder iframe. So system app will not be present in Keydown_event event chain.  
https://wiki.mozilla.org/WebAPI/BrowserAPI/KeyboardEvent#Dispatch_KeyboardEvent_across_BrowserElements
> 
> 2. If target app is not in event chain, how will the following case be
> handled?
>   - 1) keyboard app is active, so the NS_KEY_DOWN event will be dispatched
> to it.
>   - 2) keyboard app does not want to handle this NS_KEY_DOWN event, so it
> doesn't call preventDefault()
>     --> The NS_KEY_DOWN should be dispatched to target app if keyboard app
> does not process it.
>     --> However, the target app is not in event chain, 
>     --> so NS_KEY_DOWN event can not be dispatched to target app
> 
> Although the NS_KEY_DOWN event can be dispatched to target app through the
> path that keyboard app use, but personally I think its weird to do that. If
> keyboard app doesn't want to process this key event, it should not forward
> this event. I prefer remaining the target app in event chain and determine
> whether we should dispatch the keyboard event to it by checking the calling
> of event.preventDefault()

May be we have to send the NS_KEY_DOWN event to target app if the keyboard app is not consuming[not calling prevent default] it, Is there any use case for this? 
Since IPC is involved in this what about synchronization of events in event chain? and what about delay between events?
Flags: needinfo?(cchang)
(In reply to Jayachandra Yakasiri from comment #43)
> 'keydown' is intended for mozbrowser-embedded iframe not for
> mozbrowser-embedder iframe. So system app will not be present in
> Keydown_event event chain.  
> https://wiki.mozilla.org/WebAPI/BrowserAPI/
> KeyboardEvent#Dispatch_KeyboardEvent_across_BrowserElements

Here is the test case of keyboard events: 
https://dxr.mozilla.org/mozilla-central/source/dom/events/test/test_dom_before_after_keyboard_event_remote.html#27
It specifies the expected keyboard events order and shows the parent process of an embedded iframe will still receive the NS_KEY_DOWN/NS_KEY_UP.

For understanding the complicated keyboard events, I organize the code flow into a slide a few days ago.
http://goo.gl/ccjdCt
Please feel free to use it and point out my errors. I hope it can be a helpful reference.

> May be we have to send the NS_KEY_DOWN event to target app if the keyboard
> app is not consuming[not calling prevent default] it, Is there any use case
> for this? 

Let's use the 'BrowserBack' key(http://goo.gl/R4iFgM) as the example use case. Suppose the target app want to use the 'BrowserBack' key for navigation no matter keyboard app is active or not(tt's the real behavior of facebook app). Assume the keyboard app is active in this case, NS_KEY_DOWN of 'BrowserBack' key will be dispatched to keyboard app first. Then, keyboard app doesn't want to handle 'BrowserBack' key, so it won't call event.preventDefault(). Next, the 'BrowserBack' key should be dispatched to target app for navigating the app's view.

In a broad sense, all the non-printable keys need to be dispatched to target app if we allow target app to handle them.

Does this use case make sense to you?

> Since IPC is involved in this what about synchronization of events in event chain? 

... Do you mean the events order may be wrong? 

Suppose keyboard app is active, and we send 'A', 'BrowserBack','B' in order:

case 1: if ['A', 'B'] and ['BrowserBack'] are dispatched by different path(domWindowUtils.sendKeyEvent in keyboard app and EventDispatcher::Dispatch in nsPresShell), target app might receive:
  a) 'A', 'BrowserBack', 'B'
  b) 'A', 'B', 'BrowserBack'

case 2: if ['A', 'B'] and ['BrowserBack'] are dispatched by same path(domWindowUtils.sendKeyEvent in keyboard app), the order will be kept constant(No matter IPC is sync or async, the delivery is "in-order": https://developer.mozilla.org/en-US/docs/IPDL/Tutorial#Message_Delivery_Order). That is, target app will receive: 'A', 'BrowserBack', 'B'.

However, I wonder if it's important to keep the keyboard events in order when keyboard app is active and the non-printable, printable keys are dispatched to target app through different path like case 1. 

> and what about delay between events?
Yes, performance is important issue. 

1. If some specified keys are forwarded to target app by keyboard app via b2g(domWindowUtils.sendKeyEvent in keyboard app), it needs two async ipc to dispatch them to target app: http://goo.gl/4e3s6j
One is https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/MozKeyboard.js#682
the other is https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/Keyboard.jsm#360 

2. If some specified keys are ignored by keyboard app, and they will be dispatched by the same path of NS_BEFORE_KEY_DOWN, it still needs two async ipc to dispatch them to target app:
One is the notification that keyboard app finish processing these keys(ignore them actually)
The other is https://dxr.mozilla.org/mozilla-central/source/dom/ipc/TabParent.cpp#1570

No matter which case will be accepted, I think the keyboard app can be tuned to achieve the expected performance because now it is certified only.
Flags: needinfo?(cchang)
I hope these slides can be helpful to illustrate my thoughts:
- Keyboard events order now: http://goo.gl/ccjdCt
- How does keyboard app forward keyboard events to target app via b2g: http://goo.gl/4e3s6j
- Expected events order after routing hardware key events to keyboard app: http://goo.gl/wt3HiB
- Proposal for this bug: http://goo.gl/D5yW6c
Thanks for your comment. I understood it completely.

Yes both the cases has advantages and disadvantages, Which one do you suggest in the below cases

case 1: forwarding only printable key events to Keyboard app, and non printable key events in normal flow[example :  ['A', 'B'] are dispatched via domWindowUtils.sendKeyEvent in keyboard app and ['BrowserBack'] is dispached via EventDispatcher::Dispatch in nsPresShell)] 

case 2: forwarding all keyboard events to Keyboard app, let it decide which one it want to consume and which one it has to give it back to target app in normal flow[example: ['A', 'B'] and ['BrowserBack'] are dispatched via same path to keybaord app, ['A', 'B'] will be consumed by keyboard app and for['BrowserBack'] event keyboard will not be calling prevent default. so let it dispatch in normal way from preshell again to Target app via EventDispatcher::Dispatch ]

or any other better case than this?
Flags: needinfo?(cchang)
Cancel the needinfo/review flag because Chun-Min will be the one to do discussion first.
Flags: needinfo?(mchen)
(In reply to Jayachandra Yakasiri from comment #46)
Sorry for coming back to you quite late. 

After a brief discussion with Luke, we suggest taking case 2. The reason is that this approach can provide more flexibility to keyboard app. 

Possible use case: input-method switch
Now, in keyboard app, we have some small buttons "En", "12&", ...etc to switch languages. If we want to allow using hardware keyboard shortcut to switch languages layout on keyboard app like PC(Ctrl-Space[Windows]/Cmd-Space[Mac]/Super-Space[Ubuntu]), the non-printable key should be sent to keyboard app first.
Flags: needinfo?(cchang)
Depends on: 1161424
This might not be related but I am working on bug 1137557 to route the internal call in forms.js to nsITextInputProcessor and away from nsIDOMWindowUtil.
There is still one concern in this proposal(http://goo.gl/D5yW6c): It may cause infinite key events if target app is **in-process**. 
The keyboard event sent by forms.js in target app will call PresShell::HandleKeyboardEvent in main process again and lead a infinite key events.

in-process target app
===============================
process:   [chrome]         [chrome]              [content]        [chrome]
           nsPressShell --> EventStateManager --> keyboard app --> form.js in target app
             ^                                                        |
             |                                                        |
             +--------------------------------------------------------+
(In reply to Chun-Min Chang[:chunmin] from comment #50)
One possible solution is to check the *state* of MozKeyboard.js in keyboard app.
If we can add a variable named 'state' in MozKeyboard.js and set it like "state = sending" when sendKey[1] is called, then PresShell can distinguish the keyboard event is fired from keyboard app or hardware by accessing the "state" in keyboard app's MozKeyboard.js. After the keyboard event triggered by keyboard app finish dispatching, then we can set "state = idle" when keyboard app receive the ack "keyboard:SendKey:Result:OK"[2]. However, this way will add two IPC overhead.

[1] https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/MozKeyboard.js#679
[2] https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/MozKeyboard.js#513
(In reply to Chun-Min Chang[:chunmin] from comment #50, comment #51)
comment #16 already discuss this issue
Why not add a state to the key event itself? (totally fine to add some [ChromeOnly] attribute to KeyboardEvent.webidl)
(In reply to Chun-Min Chang[:chunmin] from comment #52)
> (In reply to Chun-Min Chang[:chunmin] from comment #50, comment #51)
> comment #16 already discuss this issue
The keyboardEvent.location[1] does not work as expected, the return value of keyboard.location from hardware VolumeUp/VolumeDown and Power is still 0 on my Flame. It need to find out the reason. On the other hand, as smaug mention, add a state to the keyboard event can provide a solid solution.
[1] https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/location
(In reply to Olli Pettay [:smaug] from comment #53)
> Why not add a state to the key event itself? (totally fine to add some
> [ChromeOnly] attribute to KeyboardEvent.webidl)

Yes... I think this is the right way to handle it.
Comment on attachment 8595735 [details]
This is updated proposal for Hardkey event[12 keys] routing to Keyboard app

To clear the review flag and Chun-Min is the one now.
Attachment #8595735 - Flags: review?(marcofreda527)
(In reply to Olli Pettay [:smaug] from comment #53)
> Why not add a state to the key event itself? (totally fine to add some
> [ChromeOnly] attribute to KeyboardEvent.webidl)

Instead of adding [ChromeOnly] attribute to KeyboardEvent.webidl, why can not we add it to WidgetKeyboardEvent it self, 
Anyway the sendkey in forms.js will ask nsDOMWindowUtils to reconstruct the event as WidgetKeyboardEvent and dispatches to widget. At this time we will set the [ChromeOnly] attribute true, which will be compared to avoid infinite looping.
Flags: needinfo?(cchang)
Flags: needinfo?(bugs)
Well, you of course want to be able to initialize the [ChromeOnly] property somehow, so yes, similar thing is needed on the dictionary too.
Flags: needinfo?(bugs)
Hi all,

Here i am attaching an updated proposal for Hardware key event routing, please review it and let us know your opinions.
Flags: needinfo?(cchang)
Attachment #8625393 - Flags: review?(timdream)
Attachment #8625393 - Flags: review?(marcofreda527)
Attachment #8625393 - Flags: review?(cchang)
Comment on attachment 8625393 [details]
Updated Proposal of Hardware Key Events Routing

Thanks for updating the proposal. It's clearer now. 
I have two questions here, hope you can explain more about them.

1. Does it really need to add an event queue?
Honestly, I am not sure whether or not the events order should be kept when two keys are dispatched via two different path. The event queue is designed to keep the events order if one key 'a' will be used in inputmethod, but the following key 'ESC' won't be used in inputmethod and the keyboard app is inactive at this moment. In such case, I reckon there is no need to keep the events order. If user want to exit the app by clicking 'Esc', it's not important to make sure that the exit will be fired after the app receive 'a'.

2. Why don't we call KeyboardAppProxy in both two cases?
In case 2, ESM will decide whether or not it should dispatch the keyboard event to target app based on the response of keyboard.jsm. If keyboard app doesn't consumes the keyboard event, then ESM will dispatch the keyboard event to target app. On the other hand, if keyboard app consumes the keyboard event, then ESM has no need to dispatch the event to target app. Is this case different from the case 1? If they're same, why don't we call KeyboardAppProxy to handle the case "when keyboard app consumes the event"(your case 1) in ESM?
Attachment #8625393 - Flags: review?(cchang)
Comment on attachment 8625393 [details]
Updated Proposal of Hardware Key Events Routing

Hi,

1. Does the event queue you mentioned mean the mEventQueue in nsAppShell [1]? Which is the queue to collect the physical key event from Event Hub.

2. A minor issue on case 2 - The final AfterKeyboardEvent is fired by Preshell not Tab Parent.

Personally I think it is good to go first then seeing the result.

[1] https://dxr.mozilla.org/mozilla-central/source/widget/gonk/nsAppShell.cpp#554
Attachment #8625393 - Flags: review?(marcofreda527)
(In reply to Chun-Min Chang[:chunmin] from comment #60)
> 2. Why don't we call KeyboardAppProxy in both two cases?
To Correct the question: Why don't KeyboardAppProxy response to ESM in case 1 and let ESM to call nsPresShell? Is it possible to use same interface(ESM<-->KeyboardAppProxy) in both cases?
> 1. Does it really need to add an event queue?
> Honestly, I am not sure whether or not the events order should be kept when
> two keys are dispatched via two different path. The event queue is designed
> to keep the events order if one key 'a' will be used in inputmethod, but the
> following key 'ESC' won't be used in inputmethod and the keyboard app is
> inactive at this moment. In such case, I reckon there is no need to keep the
> events order. If user want to exit the app by clicking 'Esc', it's not
> important to make sure that the exit will be fired after the app receive 'a'.

Yes,the proposed events queue is for making the events in order, I think we have to see the problems caused by events order after implementation and then decide whether it is really required or not...

> 2. Why don't we call KeyboardAppProxy in both two cases?
> In case 2, ESM will decide whether or not it should dispatch the keyboard
> event to target app based on the response of keyboard.jsm. If keyboard app
> doesn't consumes the keyboard event, then ESM will dispatch the keyboard
> event to target app. On the other hand, if keyboard app consumes the
> keyboard event, then ESM has no need to dispatch the event to target app. Is
> this case different from the case 1? If they're same, why don't we call
> KeyboardAppProxy to handle the case "when keyboard app consumes the
> event"(your case 1) in ESM?

Ok, let ESM dispatch both the crossprocess event and afterkey event[afterkey event via preshell]based on keyboardAppProxy decision. I agree with you.
> 1. Does the event queue you mentioned mean the mEventQueue in nsAppShell
> [1]? Which is the queue to collect the physical key event from Event Hub.
No, its for making the events in order, bcz keyboard consumes few events others will be dispatched to target app if keyboard app is not prevented them. so there will be a timing problem.
The same issue is mentioned in comment 44
> 2. A minor issue on case 2 - The final AfterKeyboardEvent is fired by
> Preshell not Tab Parent.
Yes, it is via presShell, but initiated by Tabparent
https://dxr.mozilla.org/mozilla-central/source/dom/ipc/TabParent.cpp#2204
> Personally I think it is good to go first then seeing the result.
Yes, I agree.
(In reply to Jayachandra Yakasiri from comment #64)
> No, its for making the events in order, bcz keyboard consumes few events
> others will be dispatched to target app if keyboard app is not prevented
> them. so there will be a timing problem.
> The same issue is mentioned in comment 44

It seems that the conclusion is to go option 2 then printable and non-printalbe keys will go to the same path. As a result what is the problem about order?

By the way, the order is important even they are dispatched as two different way.
If I understood it correctly, the [1] mentioned the sync event like keyboard event should be dispatched in order. 

From [1]
"Each event in this virtual queue MUST be delayed until the previous event has completed its propagation behavior, or been canceled."

[1] http://www.w3.org/TR/DOM-Level-3-Events/#sync-async
OS: Mac OS X → Gonk (Firefox OS)
Hardware: x86 → ARM
Comment on attachment 8625393 [details]
After brief discussion with smaug, there might be another easier solution: Is it possible to dispatch the key* event to target app upon sending the key* event to system app? It can avoid modifying the EventStateManager, although I think it still need to modify nsPresShell.
(In reply to Chun-Min Chang[:chunmin] from comment #66)
Further explain what I have mentioned: Maybe we can simply add the listener in system app to get the key* event. Once system app receives the key* event, it can forward this keyboard event to keyboard app. If keyboard app consumes the keyboard event, then system app should notify nsPresShell to fire the afterKey* event directly. On the other hand, if keyboard app doesn't consumes the keyboard event, then system app should notify nsPresShell to dispatch the keyboard event to target app and then fire the afterKey* event.
(In reply to Chun-Min Chang[:chunmin] from comment #67)
This proposal will prevent keyboard app from being privileged app because currently there is no Web API for communication between app to app in privileged permission level.

But if this is for achieving the feature first then it is the way to go.
And there was other partner doing this similar approach already.
(In reply to Marco Chen [:mchen] from comment #68)
The system app can communicate with keyboard app via MessageManager[1]. Thus, another solution mentioned in comment #66 can be achieved by hacking shell.js, which loads the Keyboard.jsm in [1]. The keyboard event can be intercepted in [2] and forward to keyboard app by calling Keyboard.sendToKeyboard('XXXX', { 'keyCode': evt.keyCode, 'charCode': evt.charCode, ..... });

[1] How does keyboard app dispatch keyboard event to web app now: http://goo.gl/4e3s6j
[2] https://dxr.mozilla.org/mozilla-central/source/b2g/chrome/content/shell.js#449
Hi Chun-min,

Got your point and note that shell.js is not part of system app.
The former one is on the chrome window and the letter one is on the content window.
(In reply to Marco Chen [:mchen] from comment #70)
Hi, Marco,
Thx for your reminder. Is it possible to use shell.js(and keyboardProxy xpcom) as a middle agent between system app and nsPresShell? If shell.js has a way to know when the key* events dispatch in system app is finished, then it still has a chance to achieve the new idea.
(In reply to Chun-Min Chang[:chunmin] from comment #71)
> (In reply to Marco Chen [:mchen] from comment #70)
> Is it possible to use shell.js(and keyboardProxy
> xpcom) as a middle agent between system app and nsPresShell?

Yes, it is possible. Either you can invent a new Web API or leverage the mozChrome/ContentEvent to do communication between System app and shell.js (shell.html)

> If shell.js has a way to know when the key* events dispatch in system app is finished

According to system app is an i-frame embeded by shell.html, you can add a event listener for key on the capturing phase. Once the listener is called, the key traversal in the system app was done.
(In reply to Marco Chen [:mchen] from comment #72)
From my test, the system/js/hardware_buttons.js[1] is able to catch the last keyboard event in system app. The last keyboard event is dispatched to system app's window in bubbling phase. After catching the last keyboard event, like you've said, we can send a mozContentEvent to shell.js. Then shell.js will know the keyboard event dispatch is finished in system app.

sample code
===============================
hardware_buttons.js
--------------------------
...
HardwareButtons.prototype.handleEvent = function hb_handleEvent(evt) {
  ...
  var cusEvt = new CustomEvent('mozContentEvent', { detail : { type: 'XXXX' } });
  window.dispatchEvent(cusEvt);
  ...
}
...


shell.js[2]
--------------------------
...
var CustomEventManager = {
  ...
  handleEvent: function custevt_handleEvent(evt) {
    ...
    switch(detail.type) {
      ...
      case 'XXXX':
        // Receive the notification that keyboard event dispatch is finished in system app!
        break;
    }
  }
}


[1] https://github.com/mozilla-b2g/gaia/blob/master/apps/system/js/hardware_buttons.js#L220
[2] https://dxr.mozilla.org/mozilla-central/source/b2g/chrome/content/shell.js#741
According to event dispatch is asynchronous, there will be a unknown time gap between normal key traverse in chrome process and sending key to content process because you can't guarantee how many jobs in the queue before you dispatch the event. Basically you additionally add one more asynchronous procedures into this work. 

Maybe you can alternatively add event listener in the window of shell.html in the capturing phase and call any IPC to send key.
(In reply to Marco Chen [:mchen] from comment #74)
If you add an event listener in the window of shell.html in the capturing phase, you will receive the key* event before dispatching to system app. At that time, if you forward the key* event to keyboard app in the event listener, then the order of key* events[1] may be wrong. We need to make sure that the key* event in keyboard app will be fired after key* events dispatch in system app is finished because keyboard app is an mozbrowser iframe embedded in system app.

Did I misunderstand what you were trying to say?
[1] https://goo.gl/JlWskF
Comment on attachment 8625393 [details]
Updated Proposal of Hardware Key Events Routing

I have no strong opinion on internal Gecko impl, and look like others have already handling it.
Attachment #8625393 - Flags: review?(timdream)
Blocks: red-square
(In reply to Chun-Min Chang[:chunmin] from comment #75)
Right, right, right.

I argued about event order and I gave the wrong comment.
What I want to say would be on the bobbling phase of shell.html's document.
feature-b2g: --- → 2.2r+
(In reply to Marco Chen [:mchen] from comment #77)
> (In reply to Chun-Min Chang[:chunmin] from comment #75)
> Right, right, right.
> 
> I argued about event order and I gave the wrong comment.
> What I want to say would be on the bobbling phase of shell.html's document.
Yes, I think it would be easier to add an event listener at bubbling phase in shell.html to forward the keyboard event without hacking the EventStateManager.
(In reply to Chun-Min Chang[:chunmin] from comment #78)
> > I argued about event order and I gave the wrong comment.
> > What I want to say would be on the bobbling phase of shell.html's document.
> Yes, I think it would be easier to add an event listener at bubbling phase
> in shell.html to forward the keyboard event without hacking the
> EventStateManager.
Hi, all.
If shell.html forwards a keyboard event, how can we block the dispatch to target app in EventStateManager?
I think nsPresShell should check if a keyboard event was forwarded to keyboard app.
If then, nsPresShell does not call EventStateManager.PostHandleEvent() nor mozbrowserafterkey* event.
And then, when the response is received from Keyboard app, it decide whether the dispatch to target app continues or it fires a mozbrowserafterkey* event according to the status of preventDefault.
Right?

By the way, currently we have almost prepared the patch implementing the original architecture containing the modification in EventStateManager.
I expect the patch can be uploaded around the next week.
I hope to discuss these things based on the patch.
(In reply to Youngwoo Jo from comment #79)
> Hi, all.
> If shell.html forwards a keyboard event, how can we block the dispatch to
> target app in EventStateManager?
> I think nsPresShell should check if a keyboard event was forwarded to
> keyboard app.
> If then, nsPresShell does not call EventStateManager.PostHandleEvent() nor
> mozbrowserafterkey* event.
> And then, when the response is received from Keyboard app, it decide whether
> the dispatch to target app continues or it fires a mozbrowserafterkey* event
> according to the status of preventDefault.
> Right?
> 
> By the way, currently we have almost prepared the patch implementing the
> original architecture containing the modification in EventStateManager.
> I expect the patch can be uploaded around the next week.
> I hope to discuss these things based on the patch.
It doesn't mean that we don't need to modify any parts of gecko code by the new approach discussed.
Yes, We still need to modify the nsPresShell to check whether or not a keyboard event should be forwarded to keyboard app(so it still need a keyboardProxy xpcom). The advantage of the new approach is to avoid modifying the complex EventStateManager.

Ok, it's good to hear that! If you already work on the patch that based on the original architecture, then we can discuss which approach is better then.
I uploaded the 1st patch.
And I hope this patch becomes the base of our discussion about architecture.

And this patch includes the modification in EventStateManager. (only one part)
However, I expect the modification in EventStateManager can be removed if the architecture is decided.
Flags: needinfo?(marcofreda527)
Flags: needinfo?(cchang)
I'm sorry to make a wrong patch without KeyboardAppProxy.
I uploaded the new patch.
Attachment #8633902 - Attachment is obsolete: true
In the attached patch i have not taken care of the issue mentioned in the below comment
https://bugzilla.mozilla.org/show_bug.cgi?id=1110030#c50
so the infinite loop problem still occurs when the editor is in b2g[i.e Inprocess Target app]

I propose a solution for this by adding a new flag in the structure "BaseEventFlags"
https://dxr.mozilla.org/mozilla-central/source/widget/BasicEvents.h#496
Initialize the flag in WidgetEvent constructor as false,
and then making the flag true when the event is recreated in nsDOMWindowUtils.cpp
This flag can be checked before sending the event to proxy in ESM[in the newly added method HandleKeyboardEvent] to stop the recursion.
Let me know your opinion on this
Comment on attachment 8633917 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.1.patch

Review of attachment 8633917 [details] [diff] [review]:
-----------------------------------------------------------------

Overall looks good to me. 
There is two main issues needed to be solved:
1. Infinite loop to send keyboard events if the target app is in-process
2. Events order
   I agree with your earlier concern(comment #65): the events order is important. 
   The events fired from the same source(e.g. 12 keys) should be in order[1,2].

[1] http://www.w3.org/TR/DOM-Level-3-Events/#glossary-event-order
[2] http://www.w3.org/TR/DOM-Level-3-Events/#sync-async

::: dom/events/EventStateManager.cpp
@@ +1216,5 @@
> +  WidgetKeyboardEvent* keyboardEvent = aEvent->AsKeyboardEvent();
> +  if ((aEvent->mClass == eKeyboardEventClass) && (hasActiveIME) && (aEvent->message != NS_KEY_PRESS) && (keyboardEvent->keyCode != NS_EVENT_NULL)) {
> +    keyboardAppProxy->SetESMContext(this, *(aEvent->AsKeyboardEvent()));
> +    rv = keyboardAppProxy->SendEventToActiveKeyboard(aEvent->message, keyboardEvent->keyCode, keyboardEvent->charCode, keyboardEvent->time);
> +    if (rv == NS_OK) {

[Personal opinion] Is it possible to call keyboardAppProxy::SetESMContext inside keyboardAppProxy::SendEventToActiveKeyboard? it might looks cleanr

::: dom/events/KeyboardAppProxy.cpp
@@ +64,5 @@
> +  }
> +  else if (aType == NS_KEY_UP) {
> +    mKeyboardEventForwardCb->OnKeyboardEventReceived("keyup", aKeyCode, aCharCode, aTimeStamp);
> +  }
> +  else {

mozilla coding style:
if {
  ...
} else if {
  ...
}

@@ +90,5 @@
> +  uint32_t EventType;
> +  if (!strcmp(aEventType,"keydown")) {
> +    EventType = NS_KEY_DOWN;
> +  }
> +  else if (!strcmp(aEventType,"keyup")) {

Remember to change the coding style here too.

::: layout/base/nsPresShell.cpp
@@ +8276,5 @@
> +                keyboardAppProxy = do_GetService("@mozilla.org/keyboardAppProxy;1", &rv);
> +                if (!keyboardAppProxy) {
> +                   return NS_ERROR_FAILURE;
> +                }
> +                rv = keyboardAppProxy->SetEventTarget(eventTarget);

Could you explain why we need to set eventTarget here? Is the eventTarget different from WidgetEvent* aEvent->target in EventStateManager? If the answer is no, why not set the eventTarget in EventStateManager? Is it possible that the nsPresShell has no relationship with keyboardAppProxy?
Flags: needinfo?(cchang)
(In reply to Chun-Min Chang[:chunmin] from comment #84)
> Comment on attachment 8633917 [details] [diff] [review]
> IME_new_hardware_keyboard_event_routing_v0.1.patch
> 
> Review of attachment 8633917 [details] [diff] [review]:
> -----------------------------------------------------------------

I've already almost changed the patch as per your comments.
And also the new patch will resolve the infinite loop problem by using the legacy WidgetEvent.mFlags.mFlagNotCrossProcessBoundaryFowarding.
Soon today I will update the patch.
Flags: needinfo?(marcofreda527)
This patch has been modified as per Chunmin's review.
And also this includes the following.
 - KeyboardAppProxy is changed to store the only current events but not an ESM nor a target.
 - KeyboardAppProxy is changed to use a queue for pending events.
 - Modification in nsPresShell is removed.
 - Added a new flag to nsDOMWindowUtils for resolving the infinite loop issue.
 - And minor changes
Attachment #8633917 - Attachment is obsolete: true
Comment on attachment 8636941 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.1.1.patch

Review of attachment 8636941 [details] [diff] [review]:
-----------------------------------------------------------------

It seems that if the target app is in the b2g process (not remote=true), then the keyboard event will be dispatched by nsPreshell to target app directly.
And the hook point of KeyboardAppProxy is too late?

::: dom/inputmethod/KeyboardAppProxy.cpp
@@ +67,5 @@
> +NS_IMETHODIMP
> +KeyboardAppProxy::SetHasActiveIME(bool aHasActiveIME)
> +{
> +  mHasActiveIME = aHasActiveIME;
> +  return NS_OK;

One question:
  Is there a potential issue of wrong key events order? 
  while 
    1. there are pending events in the queue 
    2. and IME is became inactive
    3. then a new key event is coming
Whiteboard: [red-tai] → [red-tai] [ETA=9/25]
Hi, smaug,
I was wondering if you could take a look on the following issues. I'll be grateful if you could give me some suggestions.


Based on the old proposal, there are some errors when target app is in-process
---------------------------------------------------------------------------------
1. Infinite loop to send keyboard events:
   - comment #50
   - Add a flag: keyboardevent.mFlags.mGeneratedFromIME to stop the infinite loop
2. Duplicated keydown, keypress, key up events: 
   - One is fired from hardware, the other is fired from forms.js
3. Duplicated beforeKeydown, afterKeydown, beforeKeyup, afterKeyup
   - The cause is same as the above one.
   - The beforeKey* and afterKey* should follow the original key* events
     - so the key* events fired from forms.js shouldn't trigger them


Possible solution:
---------------------------------------------------------------------------------
We have three use cases: 
a) If keyboard app isn’t active
b) If keyboard app is active and keyboard IME want to handle this keyboard event
   (event.preventDefault() will be called in keyboard IME)
c) If keyboard app is active but keyboard IME doesn’t want to handle this keyboard event
   (event.preventDefault() won't be called in keyboard IME)

To produce appropriate results in all cases, we could use additional conditions to decide the code flow

1) To decide whether or not beforekey* and afterKey* will be dispatched
if (processIsB2G && IMEIsActive && fromIME) {
  // Cancel beforekey* event
  // Cancel afterkey* event
} else {
  // dispatch beforekeykey* event
  // dispatch afterkey* event
}


2) To decide whether or not key* will be fired
if (processIsB2G && IMEIsActive && !fromIME && !targetIsRemote) {
  // block key* event
} else {
  // dispatch key*
}


However, the main concern for this method is that we need to use "targetIsRemote"(target app is in remote process) to decide the code flow. It will be hard to trace code if we use this condition. We need to figure out the use case is in-process or not first. Could we have better solution to get rid of the “targetIsRemote”?

I draw some slides to demonstrate these issues, please go https://goo.gl/1IigKD for detail information.
Flags: needinfo?(bugs)
Random thought:
I think comment #66 might be an another solution for comment #88. If we forward key* events via shell.js(or system app) instead of nsPresShell(or EventStateManager), then nsPresShell(or EventStateManager) may not need to know the target app is in-process or out-of-process because shell.js can receive keydown events in both case. However, we still need the keyboardevent.mFlags.mGeneratedFromIME to stop the the infinite loop.
Flags: needinfo?(hiro7998)
This is a patch to resolve the in-process target related issues.
However, this patch is not regarding comment #89 and a temporary patch to verify the event flow.

This patch includes the following changes.
- Added a new flag, WidgetEvent.mFlags.mGeneratedFromIME, to check the case.
- forms.js and nsDOMWindowUtils are changed to set the above flag.
- Removed modification in EventStateManager.
- nsPresShell::HandleKeyboardEvent is modified to use KeyboardAppProxy and to control the routing.
- KeyboardAppProxy is modified to follow the event flow as discussed with Chunmin.
- Minor changes
Attachment #8636941 - Attachment is obsolete: true
IME_new_hardware_keyboard_event_routing_v0.2.patch follows this event flow as explained in this diagram.
I hope this is useful to you.
No longer blocks: 1142877
Blocks: 1142877
Sorry about delay, I'm way behind my needinfo queue (since I tend to process review queue first).

Adding a new flag to event.flags is fine, and if you want to expose it to JS, just
add [ChromeOnly] readonly attribute to the relevant .webidl interface.

Why do we get the duplicates? Couldn't forms.js figure out that the event is dispatched by the hardware, and in such case not dispatch event. I thought that was the idea.
Flags: needinfo?(bugs)
Hey Vance, 
This is essential features we'll need for 2.2R.
Could you please confirm the ETA (9/25 or earlier) with partner?
Flags: needinfo?(vchen)
(In reply to Olli Pettay [:smaug] from comment #92)
Hi, Smaug,
Thx for your reply. Yes, we use a flag event.mFlags.fromIME to distinguish whether the event is dispatched by the hardware, or not. On the other hand, we need to use another condition: "eventTargetIsRemoteProcess" to control the proper event flow. However, it will become harder to trace/maintain code. Is that ok?

Besides, is it necessary to dispatch event to system app first when target app is remote(yellow area of this fig:https://goo.gl/34OiSC)? If the purpose of this behavior is same as mozbrowserbeforeKey*, can we drop it?
Flags: needinfo?(bugs)
Why kind of condition is eventTargetIsRemoteProcess?
Is it enough to just check
event.originalTarget == <xul:browser remote="true">/mozbrowser ?
Something like
http://mxr.mozilla.org/mozilla-central/source/dom/events/EventStateManager.cpp?rev=18911ac13934&mark=1151-1151,1156-1160,1163-1166#1151
We could expose that method to JS, if it is not exposed yet.

Currently we do dispatch events first to system process and then EventStateManager forwards to
child process if system process hasn't consumed (preventDefault()) the event - by default at least.
Would it help if the dispatch in system process wouldn't happen?
If the event is cancelable you could of course add event listener somewhere in the system process and call preventDefault(), then manually forward the event to child process.
Flags: needinfo?(bugs)
Blocks: TV_FxOS2.5
blocking-b2g: --- → 2.5+
Whiteboard: [red-tai] [ETA=9/25] → [red-tai] [ETA=9/25][ft:conndevices]
Duplicate of this bug: 1132380
(In reply to Olli Pettay [:smaug] from comment #95)
> Why kind of condition is eventTargetIsRemoteProcess?
> Is it enough to just check
> event.originalTarget == <xul:browser remote="true">/mozbrowser ?
> Something like
> http://mxr.mozilla.org/mozilla-central/source/dom/events/EventStateManager.
> cpp?rev=18911ac13934&mark=1151-1151,1156-1160,1163-1166#1151
> We could expose that method to JS, if it is not exposed yet.

counter-example for this condition: target element is an iframe and it's in in-process app.

If I understand right, the event target for keyboard event will be set on the focus element. AFAIK, every window has a focus. When A.html embeds an iframe that is sourced from B.html, then we will have two window(A's window and B's window) and two focus(one is in A.html, the other is in B.html). 

If A.html is an in-process app, then the keyboard event will be dispatched to focus element in B.html only. Calling A.iframe.focus() indeed can snatch the focus on A.html, but it will also set the focus element of B.html to B.document.body. Thus, the event target will be set to B.document.body in this case.
(If A.html is an oop app, then we will dispatch keyboard events to A's process with event target: A.iframe and B's process with event target:B's focus element)


Summary
-------------------
1) If we use iframe, then we will have two focus.
2) If the app is in-process, then the keyboard event will be dispatched to the inner focus element in iframe's source page.
3) => if the app is in-process, then the event target is not iframe.
   => if event target is iframe, then app is not in-process (oop)
blocking-b2g: 2.5+ → ---
Assignee: nobody → hiro7998
(In reply to Chun-Min Chang[:chunmin] from comment #89)
> Random thought:
> I think comment #66 might be an another solution for comment #88. If we
> forward key* events via shell.js(or system app) instead of nsPresShell(or
> EventStateManager), then nsPresShell(or EventStateManager) may not need to
> know the target app is in-process or out-of-process because shell.js can
> receive keydown events in both case. However, we still need the
> keyboardevent.mFlags.mGeneratedFromIME to stop the the infinite loop.

Hi Chun-Min, 

If we forward the events from shell.js instead of nsPresShell, if we are not calling preventDefault() after forwarding the event it will still propagate to target in in-process case.
If we call preventDefault()after forwarding, then nsAppShell will not generate Keypress event which is required for keyboard app not consumed case to send it to target. 
https://dxr.mozilla.org/mozilla-central/source/widget/gonk/nsAppShell.cpp#346

we can not stop the event using stopPropagation() since it can stop propagation only in its own Document.
Is there any way to solve this problem?
Flags: needinfo?(cchang)
(In reply to Jayachandra Yakasiri from comment #98)
> Hi Chun-Min, 
> 
> If we forward the events from shell.js instead of nsPresShell, if we are not
> calling preventDefault() after forwarding the event it will still propagate
> to target in in-process case.
> If we call preventDefault()after forwarding, then nsAppShell will not
> generate Keypress event which is required for keyboard app not consumed case
> to send it to target. 
> https://dxr.mozilla.org/mozilla-central/source/widget/gonk/nsAppShell.cpp#346
> 
> we can not stop the event using stopPropagation() since it can stop
> propagation only in its own Document.
> Is there any way to solve this problem?

Hi, Jayachandra

Maybe you can try calling Event::StopCrossProcessForwarding()[1] to make flag mNoCrossProcessBoundaryForwarding = true after forwarding event to IME, then the event dispatch will be stopped in EventStateManager[2].

Another question for this alternative method(forward keyboard event via shell.js) is that: 
Is there any event listener that will be called before shell's window? (Assume we add an event listener in shell's window to intercept/forward the key events) If the answer is yes, then is it possible to cancel it?

AFAIK, if several listeners are attached to the same element for the same event type, they are called in order in which they have been added. If during one such call, event.stopImmediatePropagation()[3] is called, no remaining listeners will be called. Thus, if we can make sure that our event listener will be added first, then we have a chance to stop whole the event dispatch.

[1]https://dxr.mozilla.org/mozilla-central/source/dom/events/Event.cpp#485
[2]https://dxr.mozilla.org/mozilla-central/source/dom/events/EventStateManager.cpp#1207
[3]https://dxr.mozilla.org/mozilla-central/source/dom/events/Event.cpp#477
Flags: needinfo?(cchang)
Hi Jinyoon, 

Would you please kindly provide your confirmation and feedback on this request ?
Will this work be able to be done(landed on 2.2R) by the end of Sep ?

Thank you very much !!
Flags: needinfo?(ellio.chang)
Rachelle, this work will be done on the given schedule.
Flags: needinfo?(ellio.chang)
This is a patch for minimizing the modification in nsPresShell, as per Chunmin's proposal.
So I moved almost codes to shell.js from nsPresShell.
However, the flow is almost same as the previous patch.

Chunmin, could you review this patch?
Attachment #8641387 - Attachment is obsolete: true
Attachment #8641388 - Attachment is obsolete: true
Flags: needinfo?(hiro7998) → needinfo?(cchang)
Comment on attachment 8647457 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.3.patch

Review of attachment 8647457 [details] [diff] [review]:
-----------------------------------------------------------------

Overall looks good to me.

::: b2g/chrome/content/shell.js
@@ +352,1 @@
>      window.removeEventListener('keyup', this, true);

In the case that IME is active and IME wan to handle this keyboard event, shell.js will forward the keyboard event once it receive it in capturing phase ans stop the event propagation. If the target app is oop, then we will lose the keyboard event to system app here because the keyboard event will be dispatched to system app and then dispatched to oop target app. I am not sure whether we can intercept the keyboard event here. It might need more tests to verify whether it's ok or not.

::: widget/BasicEvents.h
@@ +492,5 @@
>    // the default action has been performed.
>    bool    mDefaultPrevented : 1;
> +  // mIsGeneratedFromIME is used to check whether the event is generated from
> +  //  nsDOMWindowUtils or actual event from nsAppShell.
> +  bool    mFromIME : 1;

Instead of using "nsDOMWindowUtils", maybe we could just say "Input Method Editor" here because keyboard events will be dispatched by nsITextInputProcessor in the future. See more on Bug 1137557.

::: widget/gonk/nsAppShell.cpp
@@ +317,5 @@
> +       event.mIMEState == WidgetKeyboardEvent::eIMEState::ePending &&
> +       aEventMessage == NS_KEY_DOWN) {
> +      return nsEventStatus_eIgnore;
> +    }
> +    return status;

hmm...It's too hacky to me. This might break the rule of preventDefault(). Is it possible to stop the event propagation by .stop(Immediate)Propagation and .mNoCrossProcessBoundaryForwarding? Then we could avoid hack nsAppShell here.
Flags: needinfo?(cchang)
(In reply to Chun-Min Chang[:chunmin] from comment #103)
> ::: widget/gonk/nsAppShell.cpp
> @@ +317,5 @@
> > +       event.mIMEState == WidgetKeyboardEvent::eIMEState::ePending &&
> > +       aEventMessage == NS_KEY_DOWN) {
> > +      return nsEventStatus_eIgnore;
> > +    }
> > +    return status;
> 
> hmm...It's too hacky to me. This might break the rule of preventDefault().
> Is it possible to stop the event propagation by .stop(Immediate)Propagation
> and .mNoCrossProcessBoundaryForwarding? Then we could avoid hack nsAppShell
> here.

I agree with your opinion, that looks too hacky. however, I do not have any better way.
We have to do the followings.
 - We call only |EventDispatcher::Dispatch| in nsPresShell.
 - This method dispatches a keyboard event only to shell.js(in-process)
   not dispatching to system app nor the target.
I think there is no normal way to make them.

If you think this is too hacky and we can not avoid this, we can adopt the previous patch, IME_new_hardware_keyboard_event_routing_v0.2.patch.

What do you think about that?
Target Milestone: --- → FxOS-S7 (18Sep)
(In reply to Youngwoo Jo from comment #104)
From my tests on b2g master and v2.2, in-process app won't receive any keyboard event if we call evt.stopImmediatePropagation() in handleEvent of shell.js[1], so we might not need to hack nsAppShell. On the other hand, if we still receive keyboard events when we call evt.stopImmediatePropagation() in shell.js, then it implies that there are some problems in gecko code that will break the rule of event dispatch. We must check our current patch again.
(In reply to Chun-Min Chang[:chunmin] from comment #105)
I confirmed your point is right. my previous verification was wrong.
However, nsEditorEventListener is still checking if defaultPrevented is called to decide handling.
https://dxr.mozilla.org/mozilla-central/source/editor/libeditor/nsEditorEventListener.cpp#624

As a result, so as not to modify nsEditorEventListener, evt.preventDefault() is still required.
Or we have to add the new condition check into nsEditorEventListener.

I think adding some checks into nsEditorEventListener is better than a hack in nsAppShell.
So now I'm trying to find out the new condition check in nsEditorEventListener.
(In reply to Youngwoo Jo from comment #106)
Could we check the evt.mFlags.mPropagationStopped? (I'm not sure about this)
(In reply to Chun-Min Chang[:chunmin] from comment #107)
> Could we check the evt.mFlags.mPropagationStopped? (I'm not sure about this)
nsEditorEventListener adds the event listener as system group.
https://dxr.mozilla.org/mozilla-central/source/editor/libeditor/nsEditorEventListener.cpp#164

And system group is not affected by evt.stopPropagation and evt.stopImmediatePropagation called by non-system group, because the flags are reset before dispatch to system group.
https://dxr.mozilla.org/mozilla-central/source/dom/events/EventDispatcher.cpp#347

So we can not use evt.mFlags.mPropagationStopped to check if shell.js calls stopPropagation or not.
We can add another condition check to nsEditorEventListener.
Each event handler of keydown, keypress and keyup can check the condition like the below.
if (KeyboardAppProxy &&
    KeyboardAppProxy.IsEventPending(aEvent)) // Check if aEvent is in queue.
  return NS_OK;
So it can ignore the event handling.

How about that?
I removed a hack from nsAppShell. Instead of calling evt.preventDefault in shell.js, I used only evt.stopImmediatePropagation. And nsEditorEventListener can use a new state variable of WidgetKeyboardEvent to check its ignorance.

if (keyEvent.mIMEState == WidgetKeyboardEvent::eIMEState::ePending) {
  return NS_OK;  // Ignored in KeyPress() in nsEditorEventListener
}
Attachment #8647457 - Attachment is obsolete: true
My previous patches are not sending a keypress to keyboard app, because I thought keyboard app does not need a keypress and it's just a redundant communication.

Keypress is generated from nsAppShell using a kcm(keycode character mapping) file, if the kcm file defines the mapping for the keycode.
I think a kcm file for qwerty will define the mapping like the below cases.
 - shift + 'a' => 'A'
 - shift + '1' => '!'
So, if we do not send a keypress to keyboard app, keyboard app always should convert the keycode to the proper character code like the above cases.
However, it's duplicated with kcm file's role in case of qwerty.

So, I will change the patch to send a keypress to keyboard app.
If it's not correct, please let me know.
If then, I will stop implementing that.
This patch contains the below fixes.
- Sending a keypress to keyboard app.
- Fixed the event order problem caused by the quick multiple inputs.
=> To ensure the event order, I made all the following events pending until KeyboardAppProxy receives keydown's reply.

I think the almost known issues have been resolved.
Chun-min and smaug, could you review this patch?
Attachment #8650917 - Attachment is obsolete: true
Flags: needinfo?(cchang)
Flags: needinfo?(bugs)
This is possibly something masayuki should look at too, but please add it to my review queue.
(random comment, better to not leave printfs to patches ;) )
Flags: needinfo?(bugs)
This patch fixed the infinite waiting queue problem.

I found the infinite waiting queue problem when I tested it with spatial navigation(Bug 1008862, it changes the focus very many times.). KeyboardAppProxy manages the queue for sending events and pending events to make them in-order in the communication between KeyboardAppProxy and keyboard app. However, if the communication has some problem, the queue becomes broken. As a result, the pending events become waiting infinitely and no inputs are handled. The problem is caused by not handling the failure of |KeyboardAppProxy.sendKey| or by no responding from keyboard app with 'Keyboard:SendHardwareKeyEvent'. So, I fixed them in this patch.

Masayuki, could you review this patch please?
Attachment #8653271 - Attachment is obsolete: true
Attachment #8654730 - Flags: review?(masayuki)
Comment on attachment 8654730 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.6.patch

I'm sorry that I've changed the reviewer to smaug.
Masayuki, could you check my patch before the review?
Attachment #8654730 - Flags: review?(masayuki) → review?(bugs)
Comment on attachment 8654730 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.6.patch

Although, I've not read the above discussion, why do we need to handle keypress too? I feel that keydown/keyup, which are the physical key events, are enough for keyboard apps. And note that we need to change keypress event for conforming UI Events spec. So, keypress events will become not fired for non-printable keydown events. In that time, sending keypress events to keyboard apps may cause compatibility issue.

And looks like that this still uses nsIDOMWindowUtils::SendKey() in forms.js, however, Tim already landed the patch which makes forms.js use nsITextInputProcessor. So, I don't think that you need to change nsIDOMWindowUtils... And also NS_KEY_(DOWN|PRESS|UP) are changed to eKey(Down|Press|Up) and WidgetEvent::message is now WidgetEvent::mMessage. So, you must write this patch with old code. 

And this is a nit, looks like that you forget to change the UUID of nsIPresShell.

Finally, I think that you shouldn't add new flag to BaseEventFlags in this case since it's necessary only in WidgetKeyboardEvent. So, it should be added to WidgetKeyboardEvent.

Anyway, I think that you should separate this patch to smaller pieces. This includes a lot of ideas and steps to change.
Comment on attachment 8654730 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.6.patch

Based on masayuki's comment, this patch will need some changes.
(and I wouldn't mind if masayuki reviewed this. I'm in general rather overloaded with reviews, and masayuki knows this code at least as well as I do ;) )
Attachment #8654730 - Flags: review?(bugs)
Comment on attachment 8654730 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.6.patch

Review of attachment 8654730 [details] [diff] [review]:
-----------------------------------------------------------------

(Just a drive-by comment, not a review of all files.)

::: dom/inputmethod/Keyboard.jsm
@@ +22,5 @@
>  XPCOMUtils.defineLazyGetter(this, "appsService", function() {
>    return Cc["@mozilla.org/AppsService;1"].getService(Ci.nsIAppsService);
>  });
>  
> +XPCOMUtils.defineLazyGetter(this, "KeyboardAppProxy", function() {

This should be lower case "keyboardAppProxy".
BTW, code in shell.js is not testable -- you would need to find another way to invoke the same call in your XPCOM instance, and left the shell.js code untested.

(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #118)
> This should be lower case "keyboardAppProxy".

Actually, if this is not a trouble for you, the interface name should be some thing like MozInputMethodHardwareKeyboardEventProxy and it's instance named hardwareKeyboardEventProxy, just to avoid the word "keyboard app" from creeping into Gecko.

Thanks!
(In reply to Masayuki Nakano (:masayuki) (Mozilla Japan) from comment #116)
> And looks like that this still uses nsIDOMWindowUtils::SendKey() in
> forms.js, however, Tim already landed the patch which makes forms.js use
> nsITextInputProcessor. 
They're based on v2.2r, so they still use nsIDOMWindowUtils::SendKey() to generate a keyboard event. 

Hi, Youngwoo,
Could you move the modified part of nsIDOMWindowUtils::SendKey() to nsITextInputProcessor when you're available to rebase this patch to master?

and...Do we still need to set KEY_FLAG_NOT_CROSS_PROCESS_BOUNDARY_FORWARDING if we use evt.preventDefault() to stop dispatching keyboard events to other process?
Flags: needinfo?(cchang)
I have a question about type of keyboard event which keyboard app receives via InputMethod.
The previous patch uses it as just a CustomEvent as the below.
  {
    keyCode : value
    charCode : value
    timeStamp : value
  }
timestamp is not defined in KeyboardEvent(standard). Without timestamp, keyboard app can be also implemented. However, it might be required in order that keyboard app can handle the events more correctly.
I think it should be defined newly as a dictionary or it should be a KeyboardEvent without timestamp.

Masayuki, which one is better? Or could you suggest any other solution?
Flags: needinfo?(masayuki)
(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #119)
> BTW, code in shell.js is not testable -- you would need to find another way
> to invoke the same call in your XPCOM instance, and left the shell.js code
> untested.
Thank you for your review, Guan-tin.
Right, the code in shell.js is not testable. nsPresShell is triggering the dispatch to shell.js.
I think, only for test, it's better to move the code in shell.js to nsPresShell. However, there is a reason to change shell.js. Its history is presented in the above comments which had been started from Comment 66 and Comment 69. For minimizing the modification in common gecko source code, shell.js is more proper. However, as you commented, there is no way to test the changed code in shell.js even if we make a new method to the proxy component. In my opinion, it's not bad to move the code in shell.js to nsPresShell.
Chunmin, what do you think about that?

> Actually, if this is not a trouble for you, the interface name should be
> some thing like MozInputMethodHardwareKeyboardEventProxy and it's instance
> named hardwareKeyboardEventProxy, just to avoid the word "keyboard app" from
> creeping into Gecko.
I understood that. I'm changing the name to your suggestion, "MozInputMethodHardwareKeyboardEventProxy", However I think "Moz" prefix is not required because this is just an internal xpcom. So I will use the name, "InputMethodHardwareKeyboardEventProxy" without "Moz".
Flags: needinfo?(cchang)
(In reply to Chun-Min Chang[:chunmin] from comment #120)
> They're based on v2.2r, so they still use nsIDOMWindowUtils::SendKey() to
> generate a keyboard event. 
Right, this patch is based on v2.2r. I want the code fix in v2.2r first because of my schedule. 

> Could you move the modified part of nsIDOMWindowUtils::SendKey() to
> nsITextInputProcessor when you're available to rebase this patch to master?
Sure, I will do that. When I tested it on master, I confirmed it works fine with the several minor changes such as the variable name changes. So that's no problem.

> and...Do we still need to set KEY_FLAG_NOT_CROSS_PROCESS_BOUNDARY_FORWARDING
> if we use evt.preventDefault() to stop dispatching keyboard events to other
> process?
I also thought about that. So I will remove KEY_FLAG_NOT_CROSS_PROCESS_BOUNDARY_FORWARDING after I test it.
(In reply to Youngwoo Jo from comment #121)
I'm sorry that Event.webidl has a property, timestamp.
If then, I think we do not need a new type definition for this case.
I will change the code to use KeyboardEvent defined in KeyboardEvent.webidl.

Could you let me know that is incorrect, please.
Flags: needinfo?(masayuki)
(In reply to Youngwoo Jo from comment #124)
> (In reply to Youngwoo Jo from comment #121)
> I'm sorry that Event.webidl has a property, timestamp.
> If then, I think we do not need a new type definition for this case.

You're right.
I make the patch to the small pieces.
Part 1 prevents dispatching mozbrowserbeforeafterkey* events for IME generating keyboard events.
Attachment #8654730 - Attachment is obsolete: true
Attachment #8655830 - Flags: review?(masayuki)
Part2 adds mIMEState to WidgetKeyboardEvent in order to indicate the state of handling a hardware keyboard event by keyboard app.
Attachment #8655831 - Flags: review?(masayuki)
Part 3 adds a new method to nsPresShell as a xpcom interface's method in order to dispatch a keyboard event directly to the DOM tree.
It's used by InputMethodHardwareKeyboardEventProxy to dispatch a pending keyboard event after keyboard app replies.
Attachment #8655835 - Flags: review?(masayuki)
Part 4 is a new xpcom component named, nsIInputMethodHardwareKeyboardEventProxy.
It sends a hardware keyboard event to keyboard app and keyboard app handles the keyboard event. Once keyboard app replies, it runs the remaining flow according to the reply from keyboard app. And, in order to make them, it has a queue to manage the events in order.
Attachment #8655841 - Flags: review?(masayuki)
Part 5 adds the communication between Keyboard.jsm and an InputMethod in keyboard app via MessageManager. And, once InputMethod receives a keyboard event, InputMethod fires a custom event for keyboard event to MozInputContext.
This implementation is dependent on Bug 1161424.
Attachment #8655849 - Flags: review?(masayuki)
Part 6 hooks a keyboard event from shell.js in capturing phase by calling InputMethodHardwareKeyboardEventProxy.shouldEventPending() and sendKey(). And if it calls sendKey(), it stops the propagation by calling evt.stopImmediatePropagation().
This implementation has two constraints that it must be the first line of the handlers of shell.js and it can not be tested according to Comment 122.
Attachment #8655852 - Flags: review?(masayuki)
Part7 fixed a problem which prints a character two times in the editor for one keyboard event in case of an in-process target app.
You can find the history from comment #98 to comment #110.
Attachment #8655862 - Flags: review?(masayuki)
Comment on attachment 8655830 [details] [diff] [review]
Part1_IME_prevent_dispatch_mozbrowserbeforeafterkeyevent_for_ime_generating_keyevents.patch

>diff --git a/dom/inputmethod/forms.js b/dom/inputmethod/forms.js
>index 4f8ba47..eb01caf 100644
>--- a/dom/inputmethod/forms.js
>+++ b/dom/inputmethod/forms.js
>@@ -505,19 +505,20 @@ let FormAssistant = {
>         let event = target.ownerDocument.createEvent('HTMLEvents');
>         event.initEvent('input', true, false);
>         target.dispatchEvent(event);
>         break;
>       }
> 
>       case "Forms:Input:SendKey":
>         CompositionManager.endComposition('');
> 
>-        let flags = domWindowUtils.KEY_FLAG_NOT_SYNTHESIZED_FOR_TESTS;
>+        let flags = domWindowUtils.KEY_FLAG_NOT_SYNTHESIZED_FOR_TESTS |
>+                    domWindowUtils.KEY_FLAG_GENERATED_FROM_IME;
>         this._editing = true;
>         let doKeypress = domWindowUtils.sendKeyEvent('keydown', json.keyCode,
>                                                      json.charCode, json.modifiers, flags);
>         if (doKeypress) {
>           domWindowUtils.sendKeyEvent('keypress', json.keyCode,
>                                       json.charCode, json.modifiers, flags);
>         }
> 
>         if(!json.repeat) {

Hey, why don't you write the patches based on the latest code? We've already stopped using nsIDOMWindowUtils.sendKeyEvent() already:
http://mxr.mozilla.org/mozilla-central/source/dom/inputmethod/forms.js?rev=478dd6079573&mark=727-727,797-797,801-801,806-806,812-812#723
Attachment #8655830 - Flags: review?(masayuki) → review-
Comment on attachment 8655831 [details] [diff] [review]
Part2_IME_added_ime_state_to_WidgetKeyboardEvent.patch

Hmm, this file is based on too old tree... Please rebase to the latest m-c.

>diff --git a/widget/TextEvents.h b/widget/TextEvents.h
>index fc967e9..240e77f 100644
>--- a/widget/TextEvents.h
>+++ b/widget/TextEvents.h
>@@ -96,18 +96,19 @@ public:
>     , mKeyNameIndex(mozilla::KEY_NAME_INDEX_Unidentified)
>     , mCodeNameIndex(CODE_NAME_INDEX_UNKNOWN)
>     , mNativeKeyEvent(nullptr)
>     , mUniqueId(0)
> #ifdef XP_MACOSX
>     , mNativeKeyCode(0)
>     , mNativeModifierFlags(0)
> #endif
>     , mIsFromIME(false)
>+    , mIMEState(eIMEState::eNotHandled)
>   {
>   }

I think that you should initialize mIMEState in the default constructor too if you don't copy the value at IPC.

>+  // Indicates that the event is being handled by keyboard app
>+  enum eIMEState {

nit: "{" should be the next line (below "e" of "enum").

And please remove "e" from the type name and IMEState sounds odd because this indicates the state of keyboard apps. So, perhaps, "KeyboardAppState"? And also it should be uint8_t for reducing the class size. So, please define this enum as:

typedef uint8_t KeyboardAppStateType;
enum KeyboardAppState : KeyboardAppStateType
{
  eNotHandled,...

>+    eNotHandled,  // not yet handled by keyboard app
>+    ePending,     // being handled by keyboard app
>+    eHandled      // handled by keyboard app
>+  };
>+  eIMEState mIMEState;

So, mKeyboardAppState?
Attachment #8655831 - Flags: review?(masayuki) → review-
Comment on attachment 8655841 [details] [diff] [review]
Part4_IME_added_InputMethodHardwareKeyboardEventProxy_component.patch

>diff --git a/dom/inputmethod/InputMethodHardwareKeyboardEventProxy.cpp b/dom/inputmethod/InputMethodHardwareKeyboardEventProxy.cpp
>new file mode 100644
>index 0000000..a2f7957
>--- /dev/null
>+++ b/dom/inputmethod/InputMethodHardwareKeyboardEventProxy.cpp
>@@ -0,0 +1,356 @@

Please add these two lines before the license text:

/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* vim: set ts=8 sts=2 et sw=2 tw=80: */

See https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Coding_Style#Mode_Line

>+/* This Source Code Form is subject to the terms of the Mozilla Public
>+ * License, v. 2.0. If a copy of the MPL was not distributed with this file,
>+ * You can obtain one at http://mozilla.org/MPL/2.0/. */
>+
>+#include "mozilla/ClearOnShutdown.h"
>+#include "nsFocusManager.h"
>+#include "nsIContent.h"
>+#include "nsPIDOMWindow.h"
>+#include "mozilla/BasicEvents.h"
>+#include "nsFrameLoader.h"
>+#include "nsPresShell.h"
>+#include "InputMethodHardwareKeyboardEventProxy.h"
>+#include "mozilla/dom/TabParent.h"
>+#include "mozilla/TextEvents.h"
>+#include "nsDeque.h"
>+#include "nsIDOMKeyEvent.h"

Could you sort out the order from "a" to "z"?

>+
>+namespace mozilla {
>+
>+using namespace dom;
>+
>+NS_IMPL_ISUPPORTS(InputMethodHardwareKeyboardEventProxy, nsIInputMethodHardwareKeyboardEventProxy)

Over 80 character, perhaps:

NS_IMPL_ISUPPORTS(InputMethodHardwareKeyboardEventProxy,
                  nsIInputMethodHardwareKeyboardEventProxy)

>+StaticRefPtr<InputMethodHardwareKeyboardEventProxy> InputMethodHardwareKeyboardEventProxy::sSingleton;

Same. Perhaps:

StaticRefPtr<InputMethodHardwareKeyboardEventProxy>
  InputMethodHardwareKeyboardEventProxy::sSingleton;

And I like sInstance because a lot of singleton classes use the name.

>+InputMethodHardwareKeyboardEventProxy::InputMethodHardwareKeyboardEventProxy()
>+  : mHasActiveIME(false)
>+  , mEventQueue(new nsDeque)

Looks odd. Why don't you define bool members at last? Member variables less than 4 byte should be defined at last for reducing the memory.

And why do you need to define mEventQueue as nsDeque* rather than just nsDeque?

>+InputMethodHardwareKeyboardEventProxy::~InputMethodHardwareKeyboardEventProxy()
>+{
>+  if (mEventQueue) {

if (!mEventQueue) {
  return;
}

Then, following while statement will be less than 80 characters.

>+    WidgetKeyboardEvent* event = nullptr;
>+    while ((event = static_cast<WidgetKeyboardEvent*>(mEventQueue->PopFront()))) {
>+      delete event;
>+    }
>+    delete mEventQueue;
>+    mEventQueue = nullptr;
>+  }
>+}
>+
>+NS_IMETHODIMP
>+InputMethodHardwareKeyboardEventProxy::ShouldHookEvent(nsIDOMKeyEvent* aEvent,
>+                                  bool *aShouldHook)

Odd indent. "b" of bool should be below the "n" of nsIDOMKeyEvent. And please add * to the type, not before the argument name.

>+{
>+  WidgetKeyboardEvent* keyEvent =
>+    aEvent->GetInternalNSEvent()->AsKeyboardEvent();
>+  MOZ_ASSERT(keyEvent,
>+             "DOM key event's internal event must be WidgetKeyboardEvent");
>+
>+  if (!mHasActiveIME ||
>+      keyEvent->mIsFromIME ||
>+      keyEvent->mIMEState != WidgetKeyboardEvent::eIMEState::eNotHandled) {
>+    *aShouldHook = false;
>+    return NS_OK;
>+  }
>+
>+  *aShouldHook = true;

Looks like that this can be:

*aShouldHook = mHasActiveIME && !keyEvent->mIsFromIME &&
               keyEvent->mIMEState == WidgetKeyboardEvent::eNotHandled;


>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+InputMethodHardwareKeyboardEventProxy::RegisterKeyboardEventForwarder(nsIKeyboardEventForwardCb* aCb)

Too long, perhaps:

InputMethodHardwareKeyboardEventProxy::RegisterKeyboardEventForwarder(
                                         nsIKeyboardEventForwardCb* aCb)


>+{
>+  mKeyboardEventForwardCb = aCb;

And I don't like "Cb". They should be aCallback and mKeyboardEventForwardCallback or something.

>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+InputMethodHardwareKeyboardEventProxy::SendKey(nsIDOMKeyEvent* aEvent, bool* aResult)

Too long, perhaps:

InputMethodHardwareKeyboardEventProxy::SendKey(nsIDOMKeyEvent* aEvent,
                                               bool* aResult)

>+{
>+  MOZ_ASSERT(mHasActiveIME, "Keyboard is not activated!");
>+  MOZ_ASSERT(mKeyboardEventForwardCb,
>+               "KeyboardEventForwardCb should be set before calling SendKey");

nit: you can reduce two spaces of this line.

>+
>+  WidgetKeyboardEvent* originalEvent =
>+    aEvent->GetInternalNSEvent()->AsKeyboardEvent();
>+  MOZ_ASSERT(keyEvent,
>+             "DOM key event's internal event must be WidgetKeyboardEvent");
>+
>+  // Copy and store a keyboard event to reuse it when its reply arrives.
>+  nsAutoPtr<WidgetKeyboardEvent> copiedEvent(
>+    new WidgetKeyboardEvent(*originalEvent));
>+
>+  if (copiedEvent->message == NS_KEY_PRESS) {
>+    mEventQueue->Push(copiedEvent);
>+  } else {
>+    nsString eventType;
>+    switch (copiedEvent->message) {
>+      case NS_KEY_DOWN:
>+        eventType = NS_LITERAL_STRING("keydown");
>+        break;
>+      case NS_KEY_UP:
>+        eventType = NS_LITERAL_STRING("keyup");
>+        break;
>+    }
>+    bool isSent = false;
>+    if (mKeyboardEventForwardCb) {
>+      mKeyboardEventForwardCb->OnKeyboardEventReceived(eventType,
>+                                                       copiedEvent->keyCode,
>+                                                       copiedEvent->charCode,
>+                                                       copiedEvent->time,
>+                                                       &isSent);

Hmm, too poor information... Why don't you send nsIDOMKeyEvent? Then, whole information available at the receiver.

>+    }
>+
>+    if (isSent) {
>+      mEventQueue->Push(copiedEvent);
>+    } else {
>+      *aResult = false;
>+      return NS_OK;
>+    }

if (!isSent) {
  *aResult = false;
  return NS_OK;
}
mEventQueue->Push(copiedEvent);

>+NS_IMETHODIMP
>+InputMethodHardwareKeyboardEventProxy::ReplyKey(const nsAString& aEventType,
>+                           bool aDefaultPreventedStatus)

Odd indent...

InputMethodHardwareKeyboardEventProxy::ReplyKey(const nsAString& aEventType,
                                                bool aDefaultPreventedStatus)

And aDefaultPrevented is better name.

>+{
>+  MOZ_ASSERT((aEventType.EqualsLiteral("keydown") ||
>+              aEventType.EqualsLiteral("keyup")),
>+             "Invalid event type");
>+
>+  // We can not handle this reply because the pending event had been already
>+  // removed from the sending queue before this reply arrives.
>+  if (mEventQueue->GetSize() == 0) {

nit: if (!mEventQueue->GetSize()) {

# I wonder why nsDeque doesn't have IsEmpty()...

>+    return NS_OK;
>+  }
>+
>+  nsAutoPtr<WidgetKeyboardEvent> keyEvent(
>+    static_cast<WidgetKeyboardEvent*>(mEventQueue->PopFront()));
>+
>+  MOZ_ASSERT(((aEventType.EqualsLiteral("keydown") && keyEvent->message == NS_KEY_DOWN) ||
>+              (aEventType.EqualsLiteral("keyup") && keyEvent->message == NS_KEY_UP)),

should be:

  MOZ_ASSERT(((aEventType.EqualsLiteral("keydown") &&
               keyEvent->message == NS_KEY_DOWN) ||
              (aEventType.EqualsLiteral("keyup") &&
               keyEvent->message == NS_KEY_UP)),



>+             "Replied key event does not match with the pending event");
>+
>+  // Set the flag to specify the reply phase.
>+  keyEvent->mIMEState = WidgetKeyboardEvent::eIMEState::eHandled;

You didn't define the enum as enum class. So, you don't need to write the enum name. Please remove that for reducing each line length.

>+  // Reset the flags to dispatch this event again.
>+  keyEvent->mFlags.mDefaultPrevented = false;
>+  keyEvent->mFlags.mIsBeingDispatched = false;

I wonder, could you add WidgetEvent::ResetForReuse()? Resetting here could be broken by other developers (including me!).

>+
>+  if (aDefaultPreventedStatus) {
>+    keyEvent->mFlags.mDefaultPrevented = true;

Although, I've not understood who calls this, you need to tread mDefaultPreventedByContent and mDefaultPreventedByChrome too.


I'll check following code in this patch, later, but I hope you will rebase the patches and conform to 80 characters per line rule...
Attachment #8655841 - Flags: review?(masayuki) → review-
Hi, Youngwoo,
I think the goal of the test is to verify the event dispatch flow. If there's a way to activate/deactivate the IME(e.g. focus on a input text), can we test whether KeyboardAppProxy::ShouldHookEvent works fine or not from behavior instead of testing shell.js directly? [1,2] may be good test examples. However, we need to find a way to simulate the IME if we use mochitest.

[1] https://dxr.mozilla.org/mozilla-central/source/dom/events/test/test_dom_before_after_keyboard_event_remote.html
[2] https://dxr.mozilla.org/mozilla-central/source/dom/events/test/test_dom_before_after_keyboard_event.html
Flags: needinfo?(cchang)
Sorry, I couldn't review the remaining part today. Could you rebase the patches first and fix the lines which breaks 80 char par line rule? Then, I can review them easier.
(In reply to Masayuki Nakano (:masayuki) (Mozilla Japan) from comment #134)
> Comment on attachment 8655831 [details] [diff] [review]
> Part2_IME_added_ime_state_to_WidgetKeyboardEvent.patch
> 
> Hmm, this file is based on too old tree... Please rebase to the latest m-c.
> 
Just to clarify the expectation:
* we appreciate partner's commitment on contribution this work
* we agreed in the workshop in June that if necessary, partner can focus on landing to 2.2R firstly and the m-c.
(In reply to Chun-Min Chang[:chunmin] from comment #136)
> Hi, Youngwoo,
> I think the goal of the test is to verify the event dispatch flow. If
> there's a way to activate/deactivate the IME(e.g. focus on a input text),
> can we test whether KeyboardAppProxy::ShouldHookEvent works fine or not from
> behavior instead of testing shell.js directly? [1,2] may be good test
> examples. However, we need to find a way to simulate the IME if we use
> mochitest.
Chunmin, Thank you for your opinion.
As you mentioned, IME can be simulated in the test. However, shell.js is not loaded in mochitest. If then, which one should call |KeyboardAppProxy::ShouldHookEvent| and |KeyboardAppProxy::SendKey|? According to your opinion, should the test script call them to simulate shell.js? I'm not sure if the test is correct or not.
Could you let me know that is a correct way or not?
Comment on attachment 8655835 [details] [diff] [review]
Part3_IME_added_DispatchKeyboardEvent_to_PresShell.patch

Sorry, I don't have much time to review them. For reducing the review time, please rebase your patches with the latest m-c and conform to the 80 chars per line rule. Then, I can focus to check the main purpose of the changes.
Attachment #8655835 - Flags: review?(masayuki)
(In reply to Youngwoo Jo from comment #139)
Hi, Youngwoo,
I think Marionette maybe an alternative solution. However, I can't find any API to simulate the hardware keyboard event(it has a way to send key event from nsIDOMWindowUtils but it's not what we want). I am trying to contact the testing guys and let you know if I have further information.
I'm sorry for the late new patch.
I've rebased it to m-c and modified it as your review.
Masayuki, could you review this again, please?
Attachment #8655830 - Attachment is obsolete: true
Attachment #8662924 - Flags: review?(masayuki)
I've changed |WidgetKeyboardEvent.imeState| to |WidgetKeyboardEvent.keyboardAppState| and changed the more things as your review.
Attachment #8655831 - Attachment is obsolete: true
Attachment #8662926 - Flags: review?(masayuki)
It's rebased to m-c.
Attachment #8655835 - Attachment is obsolete: true
Attachment #8662928 - Flags: review?(masayuki)
It's rebased to m-c and changed as Masayuki's review.
Attachment #8655841 - Attachment is obsolete: true
Attachment #8662930 - Flags: review?(masayuki)
It's rebased to m-c.
Attachment #8655849 - Attachment is obsolete: true
Attachment #8662931 - Flags: review?(masayuki)
It's rebased to m-c.
Masayuki, thanks for your kindly review.
Could you review my new patches?
Attachment #8655852 - Attachment is obsolete: true
Attachment #8655862 - Attachment is obsolete: true
Attachment #8662932 - Flags: review?(masayuki)
Target Milestone: FxOS-S7 (18Sep) → FxOS-S8 (02Oct)
(In reply to Chun-Min Chang[:chunmin] from comment #141)
> (In reply to Youngwoo Jo from comment #139)
> Hi, Youngwoo,
> I think Marionette maybe an alternative solution. However, I can't find any
> API to simulate the hardware keyboard event(it has a way to send key event
> from nsIDOMWindowUtils but it's not what we want). I am trying to contact
> the testing guys and let you know if I have further information.
I find a way to simulate the hardware keyboard event in python version of Marionette.


import mozdevice
dm = mozdevice.DeviceManagerADB()
'''simulate to press b by adb shell'''
print '==> press physical key: b'
dm.shellCheckOutput(['sendevent', 'dev/input/event0','1', '48', '1'])
dm.shellCheckOutput(['sendevent', 'dev/input/event0','0', '0', '0'])
dm.shellCheckOutput(['sendevent', 'dev/input/event0','1', '48', '0'])
dm.shellCheckOutput(['sendevent', 'dev/input/event0','0', '0', '0'])

adb shell command can simulate not only physical keyboard events but also touch events, sensors events, ...etc, so I hope that we can expend this idea into a library and make it reusable by other test cases.

However, the bad news is that python version of Marionette will be phased out on try-server years later. 
I will open another bug to follow up this staff.
Depends on: 1207502
Comment on attachment 8662924 [details] [diff] [review]
Part1_IME_prevent_dispatch_mozbrowserbeforeafterkeyevent_for_ime_generating_keyevents.patch

Perhaps, mIsFromIME should be called mIsSynthesizedByTIP because mIsFromIME sounds like that the keyboard event was caused by IME including native IME of other platforms.

(TIP is also another name of IME in TSF terms on Windows, however, it's not major to other developers who don't touch TSF specific part.)
Attachment #8662924 - Flags: review?(masayuki) → review+
Comment on attachment 8662926 [details] [diff] [review]
Part2_IME_added_ime_state_to_WidgetKeyboardEvent.patch

>@@ -380,26 +379,29 @@ struct ParamTraits<mozilla::WidgetKeyboardEvent>
>     WriteParam(aMsg, aParam.location);
>     WriteParam(aMsg, aParam.mUniqueId);
> #ifdef XP_MACOSX
>     WriteParam(aMsg, aParam.mNativeKeyCode);
>     WriteParam(aMsg, aParam.mNativeModifierFlags);
>     WriteParam(aMsg, aParam.mNativeCharacters);
>     WriteParam(aMsg, aParam.mNativeCharactersIgnoringModifiers);
>     WriteParam(aMsg, aParam.mPluginTextEventString);
> #endif
>+    WriteParam(aMsg, static_cast<uint8_t>(aParam.mKeyboardAppState));

Use KeyboardAppStateType instead of uint8_t.

>     WriteParam(aMsg, aParam.mIsFromIME);
>+
>     // An OS-specific native event might be attached in |mNativeKeyEvent|,  but
>     // that cannot be copied across process boundaries.
>   }
> 
>   static bool Read(const Message* aMsg, void** aIter, paramType* aResult)
>   {
>     uint32_t keyNameIndex = 0, codeNameIndex = 0;
>+    uint8_t keyboardAppState = 0;

Same here.

>@@ -408,25 +410,28 @@ struct ParamTraits<mozilla::WidgetKeyboardEvent>
>         ReadParam(aMsg, aIter, &aResult->location) &&
>         ReadParam(aMsg, aIter, &aResult->mUniqueId)
> #ifdef XP_MACOSX
>         && ReadParam(aMsg, aIter, &aResult->mNativeKeyCode)
>         && ReadParam(aMsg, aIter, &aResult->mNativeModifierFlags)
>         && ReadParam(aMsg, aIter, &aResult->mNativeCharacters)
>         && ReadParam(aMsg, aIter, &aResult->mNativeCharactersIgnoringModifiers)
>         && ReadParam(aMsg, aIter, &aResult->mPluginTextEventString)
> #endif
>+        && ReadParam(aMsg, aIter, &keyboardAppState)
>         && ReadParam(aMsg, aIter, &aResult->mIsFromIME)
>         )
>     {
>       aResult->mKeyNameIndex = static_cast<mozilla::KeyNameIndex>(keyNameIndex);
>       aResult->mCodeNameIndex =
>         static_cast<mozilla::CodeNameIndex>(codeNameIndex);
>       aResult->mNativeKeyEvent = nullptr;
>+      aResult->mKeyboardAppState =
>+        static_cast<mozilla::WidgetKeyboardEvent::KeyboardAppState>(keyboardAppState);

Then, this is always safe even in the future.

# Although, mKeyNameIndex and mCodeNameIndex are unsafe, it should be fixed in another bug.
Attachment #8662926 - Flags: review?(masayuki) → review+
Comment on attachment 8662930 [details] [diff] [review]
Part4_IME_added_InputMethodKeyEventProxy_component.patch

>diff --git a/dom/inputmethod/nsIInputMethodKeyEventProxy.idl b/dom/inputmethod/nsIInputMethodKeyEventProxy.idl
>new file mode 100644
>index 0000000..3ed53ba
>--- /dev/null
>+++ b/dom/inputmethod/nsIInputMethodKeyEventProxy.idl
>@@ -0,0 +1,63 @@
>+/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
>+/* vim: set ts=8 sts=2 et sw=2 tw=80: */
>+/* This Source Code Form is subject to the terms of the Mozilla Public
>+ * License, v. 2.0. If a copy of the MPL was not distributed with this
>+ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
>+
>+#include "nsISupports.idl"
>+
>+%{C++
>+#define NS_INPUTMETHOD_KEY_EVENT_PROXY_CID \
>+  { 0x8b86dacb, 0xc0ad, 0x42c7, { 0x95, 0x11, 0x49, 0x8f, 0x1e, 0xed, 0x4e, 0x80 } }

nit: over 80 characters. Perhaps:

>+#define NS_INPUTMETHOD_KEY_EVENT_PROXY_CID \
>+  { 0x8b86dacb, 0xc0ad, 0x42c7, \
>+    { 0x95, 0x11, 0x49, 0x8f, 0x1e, 0xed, 0x4e, 0x80 } }

>+#define NS_INPUTMETHOD_KEY_EVENT_PROXY_CONTRACTID \
>+  "@mozilla.org/InputMethodKeyEventProxy;1"
>+%}
>+
>+interface nsIDOMKeyEvent;
>+
>+[scriptable, function, uuid(de699629-fc2a-4fff-b268-69e50a8f1f08)]
>+interface nsIKeyEventForwardCallback : nsISupports

How about nsIHardwareKeyEventListener?

Could you add the summary of this interface above uuid definition? (example: http://mxr.mozilla.org/mozilla-central/source/dom/interfaces/base/nsITextInputProcessorCallback.idl)

>+{
>+  /**
>+   * Forwards a keyboard event to the module having the communication
>+   * channel with the active InputMethod.
>+   */
>+  bool onKeyEventReceived(in nsIDOMKeyEvent aEvent);

How about OnHardwareKey()?

>+};
>+
>+[scriptable, uuid(f468a3ea-6d58-405c-ba63-1dcec3944d03)]

I guess this should be "builtinclass".

>+interface nsIInputMethodKeyEventProxy : nsISupports

"Proxy" is also used by a name of design pattern. How about nsIHardwareKeyHandler or something?

And also I'd like you to add the summary of this interface as I mentioned above.

>+{
>+  void registerKeyEventForwarder(in nsIKeyEventForwardCallback aForwarder);

Please explain about that with a comment what happens if another nsIKeyEventForwardCallback has already been registered.

>+  /**
>+   * Returns true if a keyboard event should be hooked by a content
>+   * having a InputMethod. Otherwise, returns false.
>+   */
>+  bool shouldHookEvent(in nsIDOMKeyEvent aEvent);
>+
>+  /**
>+   * Sends a hardware keyboard event to the active InputMethod to
>+   * give the first chance to handle a keyboard event in InputMethod.
>+   * Content having an InputMethod can receive a keyboard event via
>+   * this method.
>+   */
>+  bool sendKey(in nsIDOMKeyEvent aEvent);

According to the part6, I think that you can merge these methods to |bool maybeSendKeyToInputMethodApp()| or |bool maybeSendKeyToKeyboardApp()|. If it should return true when the sending keyboard event is handled by the keyboard app.

BTW, we should sort out the terms here. I think that we should call the app to input text as one of IME, InputMethodApp or KeyboardApp. IME is shorter but not clear in XP level code. So, I like InputMethodApp or KeyboardApp. In the future, if Firefox OS will support speech input or something, InputMethodApp might be better.

Please choose one from them or suggest better name. And use it for part1 too.

>+  /**
>+   * A Reply paired with |sendKey| from the active InputMethod.
>+   * Content having an InputMethod hanndls a keyboard event and it can
>+   * call |evt.preventDefault|. So this method informs that the default
>+   * prevented or not.
>+   */
>+  void replyKey(in DOMString aEventType,
>+                in bool aDefaultPrevented,
>+                in bool aDefaultPreventedByContent,
>+                in bool aDefaultPreventedByChrome);

How about |OnHandledByInputMethodApp()| or something?

>+  /**
>+   * Indicates that InputMethod is activated or not. In other words,
>+   * this implies an InputContext exists or not.
>+   */
>+  attribute bool hasActiveIME;

I have a question, this is an interface for singleton class, so, I guess that this is created per application's process, right? I worry about that an instance of this interface can be used two or more IME at once. If it's possible, you should check if unexpected situation occurs.

Anyway, I think that OnInputMethodAppConnected() and OnInputMethodAppDisconnected() are better to me.

>+};


I'd like to discuss the names and their structure first. My suggestions are not orders, please tell me your idea.
Flags: needinfo?(hiro7998)
Comment on attachment 8662928 [details] [diff] [review]
Part3_IME_added_DispatchKeyboardEvent_to_PresShell.patch

Now I'm lost with this. Why would we ever want to bypass most of the
default key event handling happening in EventStateManager. Please explain.

And looks wrong to add the method to presshell, given that we don't really use presshell itself for anything in that method.
Attachment #8662928 - Flags: superreview?(bugs) → superreview-
No longer blocks: TV_FxOS2.5
Blocks: TV_dev_board
Blocks: 1209776
(In reply to Youngwoo Jo from comment #139)
Hi, Youngwoo,
After applying this patch, you could simulate to fire hardware keyboard events via ADB
https://github.com/ChunMinChang/gaia/commit/bd991a794c62fe2e8e96e227f3d4fca0f83e0ed9

This patch use Marionette as testing framework, so you need to setup the following environment first
https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Automated_testing/gaia-ui-tests

The test file is run on emulator, so remember to open the hardware keyboard setting in <B2G>/external/qemu/android/avd/hardware-properties.ini
…
…
# Keyboard support (qwerty/azerty)
name        = hw.keyboard                                                                      
type        = boolean
#default     = no
default     = yes
abstract    = Keyboard support
description = Whether the device has a QWERTY keyboard.
…
…
(In reply to Chun-Min Chang[:chunmin] from comment #153)
To use this test on red-tai reference device, the following line:
https://github.com/ChunMinChang/gaia/commit/bd991a794c62fe2e8e96e227f3d4fca0f83e0ed9#diff-ffe3a2c997dc0d592b72f922cb77f791R39
should be modified as:
kbevt = deviceEventInputer.KeyboardEvents(keys, devieName = 'keypad')
and |keys| should also be changed into the keys that used on device keypad(e.g. 0~9, #, *,..)
(In reply to Masayuki Nakano (:masayuki) (Mozilla Japan) from comment #151)
> Comment on attachment 8662930 [details] [diff] [review]
> Part4_IME_added_InputMethodKeyEventProxy_component.patch
First of all, I appreciate your review.

> "Proxy" is also used by a name of design pattern. How about
> nsIHardwareKeyHandler or something?
How about nsIHardwareKeyEventHandler instead of nsIHardKeyHandler? However, the name is getting logger. It makes me too diffcult to make the lines shorter than 80 characters, because I'm not familiar with mozilla coding style. :(

> BTW, we should sort out the terms here. I think that we should call the app
> to input text as one of IME, InputMethodApp or KeyboardApp. IME is shorter
> but not clear in XP level code. So, I like InputMethodApp or KeyboardApp. In
> the future, if Firefox OS will support speech input or something,
> InputMethodApp might be better.
> 
> Please choose one from them or suggest better name. And use it for part1 too.
In my opinion, InputMethodApp* looks better than KeyboardApp* And, if the name is decided, I will change all other names related with this concept.

> I have a question, this is an interface for singleton class, so, I guess
> that this is created per application's process, right? I worry about that an
> instance of this interface can be used two or more IME at once. If it's
> possible, you should check if unexpected situation occurs.
I added the below check to ensure it's created only in b2g process. So the other processes can not create a new instance.

+/* static */ already_AddRefed<InputMethodKeyEventProxy>
+InputMethodKeyEventProxy::GetInstance()
+{
+  if (!XRE_IsParentProcess()) {
+    return nullptr;
+  }

> Anyway, I think that OnInputMethodAppConnected() and
> OnInputMethodAppDisconnected() are better to me.
I agree. It looks better to me.

> I'd like to discuss the names and their structure first. My suggestions are
> not orders, please tell me your idea.
About the other things, totally I agree with your idea. I will change all things by following your opinion. Until now, my update has been delayed because I had many works interrupting me. ASAP I will upload the new patch. :)
Flags: needinfo?(hiro7998)
(In reply to Olli Pettay [:smaug] from comment #152)
> Comment on attachment 8662928 [details] [diff] [review]
> Part3_IME_added_DispatchKeyboardEvent_to_PresShell.patch
Thanks for your review, Olli.

> Now I'm lost with this. Why would we ever want to bypass most of the
> default key event handling happening in EventStateManager. Please explain.
The discussion to remove the changes in EventStateManager had started since Comment 66. At that time, I understood that is to make debug easily to reduce the code complexity caused by multi process and to minimize the modifications in legay codes and in legacy event flows.

> And looks wrong to add the method to presshell, given that we don't really
> use presshell itself for anything in that method.
You're right. It's not using nsPresShell itself. Therefor I can remove the this patch.
(In reply to Youngwoo Jo from comment #130)
> Created attachment 8655849 [details] [diff] [review]
> Part5_IME_communication_between_InputMethodHardwareKeyboardEventProxy_and_key
> board_app.patch
handleFocusChange has already been removed from Keyboard.jsm after bug 1197700 was landed. Remember to rebase the code to the latest one when you're available.
Posted patch [WIP] test cases (obsolete) — Splinter Review
(In reply to Chun-Min Chang[:chunmin] from comment #153)
Now three cases are all testable:
1) Send keys when IME is inactive
2) Send Keys when IME is active, and IME will consume these keys
3) Send Keys when IME is active, but IME won't consume these keys
Depends on: 1211811
(In reply to Youngwoo Jo from comment #155)
> (In reply to Masayuki Nakano (:masayuki) (Mozilla Japan) from comment #151)
> > Comment on attachment 8662930 [details] [diff] [review]
> > Part4_IME_added_InputMethodKeyEventProxy_component.patch
> First of all, I appreciate your review.

np, and sorry for the delay to reply.

> > "Proxy" is also used by a name of design pattern. How about
> > nsIHardwareKeyHandler or something?
> How about nsIHardwareKeyEventHandler instead of nsIHardKeyHandler? However,
> the name is getting logger. It makes me too diffcult to make the lines
> shorter than 80 characters, because I'm not familiar with mozilla coding
> style. :(

How about nsIHardwareKeyHandler? Indeed, "Hard" can mean "Hardware", but not explicitly.

I like clearer name better even if it's long (but not *too* long).

> > BTW, we should sort out the terms here. I think that we should call the app
> > to input text as one of IME, InputMethodApp or KeyboardApp. IME is shorter
> > but not clear in XP level code. So, I like InputMethodApp or KeyboardApp. In
> > the future, if Firefox OS will support speech input or something,
> > InputMethodApp might be better.
> > 
> > Please choose one from them or suggest better name. And use it for part1 too.
> In my opinion, InputMethodApp* looks better than KeyboardApp* And, if the
> name is decided, I will change all other names related with this concept.

Okay, let's use it.

> > I have a question, this is an interface for singleton class, so, I guess
> > that this is created per application's process, right? I worry about that an
> > instance of this interface can be used two or more IME at once. If it's
> > possible, you should check if unexpected situation occurs.
> I added the below check to ensure it's created only in b2g process. So the
> other processes can not create a new instance.

Okay.

> > I'd like to discuss the names and their structure first. My suggestions are
> > not orders, please tell me your idea.
> About the other things, totally I agree with your idea. I will change all
> things by following your opinion. Until now, my update has been delayed
> because I had many works interrupting me. ASAP I will upload the new patch.
> :)

I see, thanks. And sorry for the delay again. I'll be back in normal work style at late this week.
(In reply to Chun-Min Chang[:chunmin] from comment #158)
> Created attachment 8670129 [details] [diff] [review]
> [WIP] test cases
> 
> (In reply to Chun-Min Chang[:chunmin] from comment #153)
> Now three cases are all testable:
> 1) Send keys when IME is inactive
> 2) Send Keys when IME is active, and IME will consume these keys
> 3) Send Keys when IME is active, but IME won't consume these keys

Thanks for your test cases. With your help, I've finished setting up the test environment for emulator gaia-ui-test. And now I'm trying to test my patches. And then I will upload the unit test cases using mochitest asap.
I changed mIsFromIME to mIsSynthesizedByTIP as per your suggestion.
Attachment #8662924 - Attachment is obsolete: true
Attachment #8672493 - Flags: review?(masayuki)
I changed the prefix names, KeyboardApp* to InputMethodApp as per masayuki's suggestion.
Attachment #8662926 - Attachment is obsolete: true
Attachment #8672494 - Flags: review?(masayuki)
As per smaug's review, I removed |nsPresShell::DispatchKeyboardEvent| added by me. Instead of that, I implemented it directly in HardwareKeyHandler.
As a result, the previous part3 patch and part4 patch are merged into this patch.
And this patch contains the modifications as per masayuki's review in Comment 151.
Attachment #8662928 - Attachment is obsolete: true
Attachment #8662930 - Attachment is obsolete: true
Attachment #8662930 - Flags: review?(masayuki)
Attachment #8672495 - Flags: review?(masayuki)
This patch replaces the previous Part5 patch which forwards a hardware keyboard event to the active input method app and receive a reply from the active input method app.
Attachment #8662931 - Attachment is obsolete: true
Attachment #8662931 - Flags: review?(masayuki)
Attachment #8672501 - Flags: review?(masayuki)
This patch replaces the previous part6 patch which hooks a hardware keyboard event in shell.js. With this patch, HardwareKeyHandler is actually hooking a hardware keyboard event.
Attachment #8662932 - Attachment is obsolete: true
Attachment #8662932 - Flags: review?(masayuki)
Attachment #8672502 - Flags: review?(masayuki)
Sorry for the delay to review this. I have some urgent bugs to fix ASAP and your patches are big. I'll be back ASAP.
Comment on attachment 8672495 [details] [diff] [review]
Part3_IME_added_HardwareKeyHandler_component.patch

>+StaticRefPtr<HardwareKeyHandler> HardwareKeyHandler::sSingleton;

nit: I like sInstance since a lot of singleton classes use this name.

>+HardwareKeyHandler::HardwareKeyHandler()
>+  : mInputMethodAppConnected(false)
>+{
>+}
>+
>+HardwareKeyHandler::~HardwareKeyHandler()
>+{
>+  while (mEventQueue.GetSize()) {
>+    delete static_cast<WidgetKeyboardEvent*>(mEventQueue.PopFront());
>+  }
>+}
>+
>+NS_IMETHODIMP
>+HardwareKeyHandler::RegisterListener(nsIHardwareKeyEventListener* aListener)
>+{
>+  mHardwareKeyEventListener = aListener;
>+  return NS_OK;

When mHardwareKeyEventListener is not nullptr, the new key event listener will override it. I.e., the old listener won't receive hardware keyboard events.

Is this your intentional behavior?

If such case never occurs, it's okay. In this case, I'd like you to add:

if (NS_WARN_IF(mHardwareKeyEventListener && aListener)) {
  return NS_ERROR_NOT_AVAILABLE;
}

If this may be occurred and the old listener shouldn't need new events anymore, we should let the old listener know that a new listener steals new key events via nsIHardwareKeyEventListener.onHardwareKey() with additional argument. For example, make the method as:

nsIHardwareKeyEventListener.onNotify(DOMString aMessage, nsIDOMKeyEvent aEvent).

Then, we can notify the listener of other things with aEvent = nullptr.

>+NS_IMETHODIMP
>+HardwareKeyHandler::MaybeSendKeyToInputMethodApp(nsIDOMKeyEvent* aEvent,
>+                                                 bool* aResult)
>+{
>+  MOZ_ASSERT(mInputMethodAppConnected, "No InputMethod app is connected");
>+  MOZ_ASSERT(mHardwareKeyEventListener,
>+             "HardwareKeyEventListener should be set");
>+
>+  WidgetKeyboardEvent* originalEvent =
>+    aEvent->GetInternalNSEvent()->AsKeyboardEvent();
>+  MOZ_ASSERT(keyEvent,
>+             "DOM key event's internal event must be a WidgetKeyboardEvent");
>+
>+  if (!mInputMethodAppConnected || originalEvent->mIsSynthesizedByTIP ||
>+      originalEvent->mInputMethodAppState != WidgetKeyboardEvent::eNotHandled) {
>+    *aResult = false;
>+    return NS_OK;
>+  }

Hmm, these are unexpected cases, aren't them? If so, I'd like this to return NS_ERROR_NOT_AVAILABLE for !mInputMethodAppConnected case and NS_ERROR_INVALID_ARG for the other cases. (i.e., should cause exception) At least I feel the latter case are unexpected cases.

>+
>+  // Copy and store a keyboard event to reuse it when its reply arrives.
>+  nsAutoPtr<WidgetKeyboardEvent> copiedEvent(
>+    new WidgetKeyboardEvent(*originalEvent));
>+
>+  if (copiedEvent->mMessage == eKeyPress) {
>+    mEventQueue.Push(copiedEvent);
>+  } else {
>+    bool isSent = false;
>+    if (mHardwareKeyEventListener) {
>+      mHardwareKeyEventListener->OnHardwareKey(aEvent, &isSent);
>+    }
>+    if (!isSent) {
>+      *aResult = false;
>+      return NS_OK;
>+    }
>+    mEventQueue.Push(copiedEvent);
>+  }
>+
>+  // Set the flags to specify the sending phase.
>+  originalEvent->mInputMethodAppState = WidgetKeyboardEvent::eHandling;
>+
>+  copiedEvent.forget();
>+  *aResult = true;
>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+HardwareKeyHandler::OnInputMethodAppConnected()
>+{

Don't you need to check mInputMethodAppConnected here? Like:

if (NS_WARN_IF(mInputMethodAppConnected)) {
  return NS_ERROR_UNEXPECTED;
}

>+  mInputMethodAppConnected = true;
>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+HardwareKeyHandler::OnInputMethodAppDisconnected()
>+{

Similar, how about:

if (NS_WARN_IF(!mInputMethodAppConnected)) {
  return NS_ERROR_UNEXPECTED;
}

>+  mInputMethodAppConnected = false;
>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+HardwareKeyHandler::OnHandledByInputMethodApp(const nsAString& aEventType,
>+                                              bool aDefaultPrevented,
>+                                              bool aDefaultPreventedByContent,
>+                                              bool aDefaultPreventedByChrome)
>+{
>+  MOZ_ASSERT((aEventType.EqualsLiteral("keydown") ||
>+              aEventType.EqualsLiteral("keyup")),
>+             "Invalid event type");
>+
>+  // We can not handle this reply because the pending event had been already
>+  // removed from the sending queue before this reply arrives.
>+  if (mEventQueue.GetSize() == 0) {

nit: if (!mEventQueue.GetSize()) {

>+    return NS_OK;
>+  }
>+
>+  nsAutoPtr<WidgetKeyboardEvent> keyEvent(
>+    static_cast<WidgetKeyboardEvent*>(mEventQueue.PopFront()));
>+
>+  MOZ_ASSERT(((aEventType.EqualsLiteral("keydown") &&
>+               keyEvent->mMessage == eKeyDown) ||
>+              (aEventType.EqualsLiteral("keyup") &&
>+               keyEvent->mMessage == eKeyUp)),
>+             "Handled key event does not match with the pending event");

I'm afraid that isn't this is available a way to crash the process intentionally (even though only on debug build)?

I like you to use:

if (NS_WARN_IF(aEventType.EqualsLiteral("keydown") &&
                 keyEvent->mMessage != eKeyDown) ||
    NS_WARN_IF(aEventType.EqualsLiteral("keyup") &&
                 keyEvent->mMessage != eKeyUp)) {
  return NS_ERROR_INVALID_ARG;
}

>+void
>+HardwareKeyHandler::DispatchAfterKeyEvent(WidgetKeyboardEvent* aEvent)
>+{
>+  if (!PresShell::BeforeAfterKeyboardEventEnabled() ||
>+      aEvent->mMessage == eKeyPress) {
>+    return;
>+  }
>+
>+  nsCOMPtr<nsIContent> target = GetEventTarget();

I feel this is odd. If a keyboard event handler in the process has changed the focus, this is different from the target of normal key events. Is that okay? Looks like that PresShell doesn't refer new focused target at dispatching after key events.

>+  if(!target) {
>+    return;
>+  }
>+  nsCOMPtr<nsIPresShell> presShell = GetPresShell(target);

So, I think that presShell and target should be passed with arguments.

>+bool
>+HardwareKeyHandler::DispatchToTargetApp(WidgetKeyboardEvent* aEvent)
>+{
>+  MOZ_ASSERT(aEvent, "Event is null");
>+
>+  nsCOMPtr<nsIContent> target = GetEventTarget();
>+  if (!target) {
>+    return false;
>+  }
>+
>+  nsCOMPtr<nsIPresShell> presShell = GetPresShell(target);
>+  if (!presShell) {
>+    return false;
>+  }
>+
>+  // 1. Dispatch keydown or keyup to in-process
>+  DispatchToCurrentProcess(presShell, target, aEvent);

Dispatching an event may cause destroying the widget. So, please check if the widget is available and the presShell is still available. You should expose PresShell::CanDispatchEvent() and use it.

>+  if (IsTargetIframe(target)) { // If target app is OOP,
>+    // 1-1. If the default prevented, it stops dispatching keydown or keyup.
>+    //      And then it dispatches mozbrowserafterkey* to in-process.
>+    if (aEvent->mFlags.mDefaultPrevented) {
>+      DispatchAfterKeyEvent(aEvent);
>+      return true;
>+    }
>+
>+    // 2. Dispatch keydown or keyup to oop target.
>+    if (!DispatchToCrossProcess(target, aEvent)) {
>+      return false;
>+    }
>+  } else { // If target app is in-process
>+    // 2. Dispatch mozbrowserafterkey* to in-process target.
>+    DispatchAfterKeyEvent(aEvent);
>+  }
>+  return true;
>+}

>+bool
>+HardwareKeyHandler::IsTargetIframe(nsIContent* aTarget)
>+{
>+  return aTarget && aTarget->IsHTMLElement(nsGkAtoms::iframe);

Don't you need to check additional attributes if the iframe is actually a target? (I'm not sure, though)

>+/* static */ already_AddRefed<HardwareKeyHandler>
>+HardwareKeyHandler::GetInstance()
>+{
>+  if (!XRE_IsParentProcess()) {
>+    return nullptr;
>+  }
>+
>+  if (!sSingleton) {
>+    sSingleton = new HardwareKeyHandler();
>+    ClearOnShutdown(&sSingleton);
>+  }
>+
>+  nsRefPtr<HardwareKeyHandler> service = sSingleton.get();

FIY: nsRefPtr is renamed to RefPtr

>+class HardwareKeyHandler : public nsIHardwareKeyHandler
>+{
>+  public:
>+    HardwareKeyHandler();
>+
>+    NS_DECL_ISUPPORTS
>+    NS_DECL_NSIHARDWAREKEYHANDLER
>+
>+    static already_AddRefed<HardwareKeyHandler> GetInstance();
>+
>+  private:
>+    virtual ~HardwareKeyHandler();
>+
>+    void DispatchAfterKeyEvent(WidgetKeyboardEvent* aEvent);
>+    bool DispatchToTargetApp(WidgetKeyboardEvent* aEvent);
>+    void DispatchToCurrentProcess(nsIPresShell* presShell,
>+                                  nsIContent* aTarget,
>+                                  WidgetKeyboardEvent* aEvent);
>+    bool DispatchToCrossProcess(nsINode* aTarget, WidgetKeyboardEvent* aEvent);
>+
>+    bool IsTargetIframe(nsIContent* aTarget);
>+    already_AddRefed<nsIContent> GetEventTarget();
>+    already_AddRefed<nsIPresShell> GetPresShell(nsINode* aTarget);
>+
>+    nsDeque mEventQueue;
>+    nsCOMPtr<nsIHardwareKeyEventListener> mHardwareKeyEventListener;
>+    bool mInputMethodAppConnected;
>+
>+    static StaticRefPtr<HardwareKeyHandler> sSingleton;

You can reduce 2 white spaces of each line. The scope should be started at start of each line.

>+/**
>+ * This interface should be registered to nsIHardwareKeyHandler through
>+ * |nsIHardwareKeyHandler.registerListener|. When nsIHardwareKeyHandler needs
>+ * to forward a hardware keyboard event to the active input method, this
>+ * listener is invoked to communicate with the active input method. Therefor,

Therefore?


>+  /**
>+   * Reply paired with |maybeSendKeyToInputMethodApp| from the active input
>+   * method. When the processing in content finishes, the result is notified
>+   * through this interface. Therefor, this interface should be called by a

Therefore?

>+   * module having the communication channel with the active input method.
>+   * Probably it might be a listener registered through |registerListener|.
>+   *
>+   * The result should contain the original event type and the info whether
>+   * the default is prevented or not. So the below parameters should be passed.
>+   *
>+   * @param aEventType                 The type of an original event.
>+   * @param aDefaultPrevented          |evt.preventDefault| is called or not
>+   * @param aDefaultPreventedByContent True, if prevented by input method app.

This sounds odd. defaultPreventedByContent means that it's consumed by web apps. For example, if a keydown event shouldn't be blocked by content, mDefaultPrevented will be ignored. E.g., Ctrl+Tab on Windows and Linux.

>+   * @param aDefaultPreventedByChrome  True, if prevented by chrome.
>+   */
>+  void onHandledByInputMethodApp(in DOMString aEventType,
>+                                 in bool aDefaultPrevented,
>+                                 in bool aDefaultPreventedByContent,
>+                                 in bool aDefaultPreventedByChrome);

3 bool arguments are ugly... e.g., |true, true false| isn't clear for other users. Although, I know you use this method clearer than that since you specify variables whose name indicates them. However, I'd like you to use flags like:

NOT_DEFAULT_PREVENTED        = 0x00;
DEFAULT_PREVENTED            = 0x01;
DEFAULT_PREVENTED_BY_CONTENT = 0x02;
DEFAULT_PREVENTED_BY_CHROME  = 0x04;

>+  /**
>+   * Notifies to nsIHardwareKeyHandler that the communication channel between

Notifies nsIHardwareKeyHandler of the communcation channel...

>+   * b2g process and the active input method app has been estabilshed. This
>+   * notification is required to check if sending a hardware keyboard event to
>+   * the active keyboard app is needed or not.
>+   */
>+  void onInputMethodAppConnected();
>+
>+  /**
>+   * Notifies to nsIHardwareKeyHandler that the communication channel between

Notifies nsIHardwareKeyHandler of the communication channel...

>+  /**
>+   * Reset the flags related to event flow control for redispatch
>+   */
>+  void ResetForReuse() {

nit: |{| should be below the 'v' of void. (i.e., put it to the start of next line.
Attachment #8672495 - Flags: review?(masayuki)
Comment on attachment 8672501 [details] [diff] [review]
Part4_IME_communication_between_HardwareKeyHandler_and_inputmethod_app.patch

>+  bool DefaultPreventedByContent() const
>+  {
>+    return mEvent->mFlags.mDefaultPreventedByContent;

I think that should check mEvent->mFlags.mDefaultPrevented too.

>+  }
>+
>+  bool DefaultPreventedByChrome() const
>+  {
>+    return mEvent->mFlags.mDefaultPreventedByChrome;
>+  }

Ditto.

>+  onHardwareKey: function onHardwareKeyReceived(event) {
>+    let keyboardEventDict = {
>+      // dictionary KeyboardEvetInit
>+      key: event.key,
>+      code: event.code,
>+      location: event.location,
>+      repeat: event.repeat,
>+      isComposing: event.isComposing,
>+      charCode: event.charCode,
>+      keyCode: event.keyCode,
>+      which: event.which,
>+      // dictionary EventModifierInit
>+      ctrlKey: event.getModifierState('Control'),
>+      shiftKey: event.getModifierState('Shift'),
>+      altKey: event.getModifierState('Alt'),
>+      metaKey: event.getModifierState('Meta'),
>+      modifierAltGraph: event.getModifierState('AltGraph'),
>+      modifierCapsLock: event.getModifierState('CapsLock'),
>+      modifierFn: event.getModifierState('Fn'),
>+      modifierFnLock: event.getModifierState('FnLock'),
>+      modifierNumLock: event.getModifierState('NumLock'),
>+      modifierOS: event.getModifierState('OS'),
>+      modifierScrollLock: event.getModifierState('ScrollLock'),
>+      modifierSymbol: event.getModifierState('Symbol'),
>+      modifierSymbolLock: event.getModifierState('SymbolLock'),

Hmm, this is not good for maintenance. I guess that here is always run as chrome, right? If so, we can use chrome only method. Then, cannot we create dictionary object from dom/KeyboardEvent.cpp? If it's possible, we can support new modifier flags automatically if it's coded with macro and all members of dictionary is defined in EventForwards.h.

>+      case 'Keyboard:ReplyHardwareKeyEvent':
>+        hardwareKeyHandler
>+        .onHandledByInputMethodApp(msg.data.eventType,
>+                                   msg.data.defaultPrevented,
>+                                   msg.data.defaultPreventedByContent,
>+                                   msg.data.defaultPreventedByChrome);

So, I'm happy if you'd use flags as I mentioned at reviewing the previous patch.

>+    // Notify the communication channel with the active input method app has
>+    // been destroyed.
>+    hardwareKeyHandler.onInputMethodAppDisconnected();

I wonder is this enough? If this object is deleted by GC or something, blur event handler might not run before that...

>           this._inputcontext.updateSelectionContext(data, false);
>         }
>         break;
>       case 'Keyboard:GetContext:Result:OK':
>         this.setInputContext(data);
>         break;
>       case 'Keyboard:SupportsSwitchingTypesChange':
>         this._supportsSwitchingTypes = data.types;
>         break;
>+      case 'Keyboard:SendHardwareKeyEvent':
>+        let res;
>+        if (this.inputcontext) {
>+          res = this._inputcontext.dispatchKeyboardEvent(data);
>+        }
>+        cpmmSendAsyncMessageWithKbID(this, 'Keyboard:ReplyHardwareKeyEvent', {
>+          eventType: data.eventType,
>+          defaultPrevented: !!(res && res.defaultPrevented),
>+          defaultPreventedByContent: !!(res && res.defaultPreventedByContent),
>+          defaultPreventedByChrome: !!(res && res.defaultPreventedByChrome)

So, I'm happy if you use flags for representing this.


I'm really not familiar with around this module, so, if there is somebody better to review here, I'd like you to ask review additionally to hem/her after my next review.
Attachment #8672501 - Flags: review?(masayuki)
Comment on attachment 8672502 [details] [diff] [review]
Part5_IME_hooking_by_HardwareKeyHandler_in_shell.js.patch

>@@ -621,35 +621,41 @@ nsEditorEventListener::KeyPress(nsIDOMKeyEvent* aKeyEvent)
>   // If the client pass cancelled the event, defaultPrevented will be true
>   // below.
> 
>   bool defaultPrevented;
>   aKeyEvent->GetDefaultPrevented(&defaultPrevented);
>   if (defaultPrevented) {
>     return NS_OK;
>   }
> 
>+  WidgetKeyboardEvent* keyEvent =
>+    aKeyEvent->GetInternalNSEvent()->AsKeyboardEvent();
>+  MOZ_ASSERT(keyEvent,
>+             "DOM key event's internal event must be WidgetKeyboardEvent");
>+  // We have to ignore the event being handled by input method app.
>+  // If it's not ignored, the duplicated keypresses will be handled.
>+  if (keyEvent->mInputMethodAppState == WidgetKeyboardEvent::eHandling) {
>+    return NS_OK;
>+  }

Oh, this is too bad. If somebody handles keypress event before editor, they also need to check this state. Hmm, cannot we stop the propagation at EventDispatcher if the content shouldn't handle the keypress event?
Attachment #8672502 - Flags: review?(masayuki) → review-
(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #169)
> Comment on attachment 8672502 [details] [diff] [review]
> Part5_IME_hooking_by_HardwareKeyHandler_in_shell.js.patch
> 
> Oh, this is too bad. If somebody handles keypress event before editor, they
> also need to check this state. Hmm, cannot we stop the propagation at
> EventDispatcher if the content shouldn't handle the keypress event?

I'm sorry for the late reply.
I reviewed all my patches again. However, I could not find a good solution for your above comment. And I had thought about the more problems.

We dispatches the event to shell.js and it decides to forward the event to input method app or not.
The above dispatch to shell.js makes the problems as the below.
(1) EventDispatcher can not prevent the event handling in system group event handlers.
(2) EventDispatcher will call PreHandleEvent() of all target event chain elements.
(3) EventDispatcher will call PostHandleEvent() of all target event chain elements.
(4) We can not ensure that our new event handler in shell.js is the first event handler of shell.js.
    i.e. What if the handler prior to our new handler consumes the hardware keyboard event?

The above problems are caused by our calling only evt.stopPropagation() in shell.js and by our calling EventDispatcher::Dispatch() to target element.

If then, how about calling also evt.preventDefault() in shell.js? It will resolve only (1) problem. However, it still has the several differnt problems including the above problems.
(1) System dependent keypress can not be generated from nsAppShell since the default prevented.
(2) nsAppShell's modification will require the modifications in all platforms(window, gtk, cocoa,..).
(3) And mozbrowserafterkey* events will be generated even though the event processing does not finish.
(4) When nsHardwareKeyHandler dispatches the event to target app after the input method app handles the event, the event will be regarded as a totally new event. Is that OK?

To fix the all above problems, they need the many hacky codes for each case. I could not find out the good solution without the hacky codes.

So, I suggest that the code to forward the keyboard event to input method app should be moved prior to calling EventDispatcher::Dispatch(target).
New location to forward the event to input method app,
 - https://dxr.mozilla.org/mozilla-central/source/layout/base/nsPresShell.cpp#6933

However, this change will make that input method app become the first event handler prior to shell.html and system app. This is commented already in https://wiki.mozilla.org/WebAPI/InputMethod_API_with_hardware_keyboard#Limitation. According to the above post, we have to trust the built-in keyboard app works correctly and it assumes that system supports mozborwserbefore/afterkey* events. So I think this patch should be binded with the feature preference of mozborwserbefore/afterkey*.

Masayuki, how about this change?
(In reply to Youngwoo Jo from comment #170)
Hi, Youngwoo, 
I was just wondering if these problems could be solved by original architecture used in attachment 8641387 [details] [diff] [review](comment #90).
Summary of above
-------------------------
- mozbrowserafterkeyXX/mozbrowserbeforekeyXXX:
  only mozBrowser-embedder (and the target is a moz iframe embedded by it) can receive them(comment #31)
- Using "state" to avoid the infinite loop(comment #50, comment #53, comment #57)
- Event queue: It's used for synchronous.
  The events fired from the same source(e.g. physical keyboards) should be in order[1,2].
- Keypress: It's used for decoding "keycode"(comment #111).
  - If we call |evt.preventDefault()| after receiving keydown,
  then keypress will NOT be generated from nsAppShell[3](comment #98).
  - However, the "keycode" is a legacy attribute. Can't we just ignore it?
- The |mozbrowserafterkeyXX| or |mozbrowserbeforekeyXXX| can be fired once
  IME(keyboard app) call |preventDefault()|.
  It dosen't need to wait for the response from IME(keyboard app) because
  the keyboard event generated from IME(keyboard app) is another "independent" event.

1. http://www.w3.org/TR/DOM-Level-3-Events/#glossary-event-order
2. http://www.w3.org/TR/DOM-Level-3-Events/#sync-async
3. https://dxr.mozilla.org/mozilla-central/source/widget/gonk/nsAppShell.cpp#346
Comment on attachment 8670129 [details] [diff] [review]
[WIP] test cases

Review of attachment 8670129 [details] [diff] [review]:
-----------------------------------------------------------------

::: tests/python/gaia-ui-tests/gaiatest/apps/hw_keyevent/DeviceEvent/deviceEventInputer.py
@@ +366,5 @@
> +            keyupEvent = keyup.toSendEvtCmdList()
> +
> +            cmdList = keydownEvent + [';'] + synEvent + [';'] + keyupEvent + [';'] + synEvent
> +            # print cmdList
> +            self.dmADB.shellCheckOutput(cmdList)

Since we move tests from GIP to GIJ, I'd like to replace this tests. However, I can't find what is the equivalent GIJ api to the GIP api "mozdevice.DeviceManagerADB().shellCheckOutput(...)".
Hi, Ghislain,
I was wondering if you could tell me what is the equivalent GIJ api to the GIP api "mozdevice.DeviceManagerADB().shellCheckOutput(...)".
Flags: needinfo?(aus)
It's no longer as a part of red-tai project. It's included in TV project now.
Whiteboard: [red-tai] [ETA=9/25][ft:conndevices] → [ft:conndevices]
(In reply to Chun-Min Chang[:chunmin] from comment #174)
> Hi, Ghislain,
> I was wondering if you could tell me what is the equivalent GIJ api to the
> GIP api "mozdevice.DeviceManagerADB().shellCheckOutput(...)".

Unfortunately we do not have direct support for this at this time. It will, however, be part of the fxos-device-service which will replace the current bits we use to startup Mulet / Devices when executing tests.

In the meantime, I believe you can call adb shell yourself directly using nodejs apis for child processes. It would probably look something like this:

var spawnSync = require('child_process').spawnSync;

...

var childProcess = spawnSync('adb', ['shell', '_YOUR_SHELL_COMMAND_']);

childProcess will be one of these objects as described here -- https://nodejs.org/dist/latest-v4.x/docs/api/child_process.html#child_process_child_process_spawnsync_command_args_options

Hope that helps :)
Flags: needinfo?(aus)
Three use cases for this bug
------------------------------------
1) If keyboard app isn’t active
2) If keyboard app is active and keyboard IME want to handle the keyboard event
3) If keyboard app is active but keyboard IME doesn’t want to handle the keyboard event

In our current architecture, we have several possible code flows:
case 2 >> 
|event.preventDefault()| and |event.stopImmediatePropagation()| should be called by shell.js or keyboard IME to stop dispatching event, then IME will generate a new keyboard event with IME's values.

case 3 >> we have 2 possible ways:
- |event.preventDefault()| and |event.stopImmediatePropagation()| won't be called, and the events will be dispatched as original once IME notify shell.js that it doesn't want to handle this event.
- |event.preventDefault()| and |event.stopImmediatePropagation()| still be called, but IME then will generate a new keyboard with original values.

If we take approach "a", then we need to use a synchronous call from shell.js to Keyboard.jsm to get to know whether or not the IME app want to handle the keyboard events.

If we take approach "b", then we can always use an asynchronous from shell.js to Keyboard.jsm. IME app always can generate a new keyboard events with appropriate values, depending on whether or not IME app want to handle the keyboard events. If IME want to handle the key, then IME can generate a new keyboard events with IME's values. If IME doesn't want to handle the key, then IME can just generate a new keyboard events with all original values.

However, these two approaches will fire the |mozbrowserafterkey| in different time. 
- approach "a"
If IME is active, |mozbrowserafterkey| will be fired after receiving result from IME
("result" = IME want to handle the key or not)
- approach "b"
If IME is active, |mozbrowserafterkey| will be fired upon |event.preventDefault()| is called.

Hi Masayuki,
I was wondering if you could share your opinions about this.
Flags: needinfo?(masayuki)
(In reply to Chun-Min Chang[:chunmin] from comment #177)
It occurred to me that one case might be affected if we fire |mozbrowserafterkey| in different time.
In APP-First scenario[0]:
  One key can be handled by app first. If app doesn't want to handle the key, 
  system will handle this key. To achieve this scenario, system listen the |mozbrowserafterkey|
  to handle the key. If we take approach "b", |mozbrowserafterkey| will always be fired 
  when IME is active, upon |event.preventDefault()| is called. 
  Therefore, system might able to handle the key before target app handle it.


The code flow for these two different approaches
-----------------------------------------------------
approach "a":
1) IME is active. shell.js asked whether or not IME want to handle this key. 
   The event dispatch will suspend to wait the response from IME.
2) If IME doesn't want to handle this key, |event.preventDefault()| won't be called. 
   The event dispatch will follow the original path.
   ---> event will be dispatched to target app
   ------> If target app want to handle this key, it can handle the key at this time.
3) The timing of |mozbrowserafterkey| will be fired as original
   ---> System can check whether the key is handled by target app or not
   ------> If target app doesn't handle the key yet, system can handle the key at this time.

approach "b":
1) IME is active, so shell.js will call |event.preventDefault()| 
   no matter whether IME want to handle this key or not
2) |mozbrowserafterkey| will be fired once |event.preventDefault()| is called
   ---> The target app doesn't handle this key yet, but system might be able to handle this key
        before target app!
3) If IME doesn't want to handle this key, IME will generate a new keyboard event 
   with same values as original one.
   ---> The new keyboard event is generated by nsITextInputProcessor, and this synthesized event
        won't fire any |mozbrowserafterkey| or |mozbrowserbeforekey|.
        Therefore, there is no chance for system to handle the key *after* target app handle it.

However, if APP-First scenario doesn't need to exist when IME is active, then approach "b" can be an available solution.


[0] https://wiki.mozilla.org/WebAPI/BrowserAPI/KeyboardEvent#Dispatch_KeyboardEvent_across_BrowserElements
(In reply to Chun-Min Chang[:chunmin] from comment #178)
Hi, Tim,
May I ask your suggestions about this issue?
Flags: needinfo?(timdream)
(In reply to Chun-Min Chang[:chunmin] from comment #177)
> case 3 >> we have 2 possible ways:
> - |event.preventDefault()| and |event.stopImmediatePropagation()| won't be
> called, and the events will be dispatched as original once IME notify
> shell.js that it doesn't want to handle this event.
> - |event.preventDefault()| and |event.stopImmediatePropagation()| still be
> called, but IME then will generate a new keyboard with original values.
> 
> If we take approach "a", then we need to use a synchronous call from
> shell.js to Keyboard.jsm to get to know whether or not the IME app want to
> handle the keyboard events.
> 
> If we take approach "b", then we can always use an asynchronous from
> shell.js to Keyboard.jsm. IME app always can generate a new keyboard events
> with appropriate values, depending on whether or not IME app want to handle
> the keyboard events. If IME want to handle the key, then IME can generate a
> new keyboard events with IME's values. If IME doesn't want to handle the
> key, then IME can just generate a new keyboard events with all original
> values.
> 
> However, these two approaches will fire the |mozbrowserafterkey| in
> different time. 
> - approach "a"
> If IME is active, |mozbrowserafterkey| will be fired after receiving result
> from IME
> ("result" = IME want to handle the key or not)
> - approach "b"
> If IME is active, |mozbrowserafterkey| will be fired upon
> |event.preventDefault()| is called.
> 

This is a hard question. And it's possible we have cornered ourselves to this due to bad API design/planning (that does not consider the async nature of the cross-process communications). We should discuss this in person and maybe find an alternative (plz. grab Luke too).
Flags: needinfo?(timdream)
Well, I'm still being confused, though...

> However, if APP-First scenario doesn't need to exist when IME is active, then approach "b" can be
> an available solution.

Note that on desktop OSes, IME handles (native) key events *before* Gecko and some key inputs which are registered by system, they may not be notified to applications (e.g., Win+Tab key on Windows).

So, I think that this order isn't problem:

1. handlable system first
2. handlable IME next
3. handlable app/content
4. handlable system finally

With approach A and B, how this order will be changed?
Flags: needinfo?(masayuki)
(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) (offline until 1/6) from comment #181)
> So, I think that this order isn't problem:
> 
> 1. handlable system first
> 2. handlable IME next
> 3. handlable app/content
> 4. handlable system finally
> 
> With approach A and B, how this order will be changed?

With approach A and B, the order of |keydown| or |keyup| won't be changed. 
However, the |afterkeydown| or |afterkeyup| will be changed. 
Let's use |keydown| for example.

Usecase: IME is active, but IME doesn't want to handle this key

approach A
-----------------
1) system receive 'beforekeydown' 
2) system receive 'keydown' fired from hardware
   --> we will ask IME want to handle or not here. 
   -----> wait for IME's response:
          IME doesn't want to handle this key, so we will dispatch key event as original.
3) app(the event target) receive 'keydown' fired from hardware
4) system receive 'afterkeydown'


approach B
-----------------
1) system receive 'beforekeydown'
2) system receive 'keydown' fired from hardware
   --> we don't need to ask IME here.
       we will forward the key event's information to IME if it's active.
   -----> system will call event.preventDefault() directly 
          because we assume IME will handle it.
3) system receive 'afterkeydown' 
   ('afterkeydown' will be fired upon system call event.preventDefault())
4) app(the event target) receive 'keydown' generated by IME
   --> IME will generate a same key event as received one if IME doesn't want to handle this key.
(In reply to Chun-Min Chang[:chunmin] from comment #184)
To make sure that the 'afterkeydown' is always fired after 'keydown', it seems that approach A is better than B. Nevertheless, even we take approach A, there are some issues needed to be concerned.

# forward key event via shell.js
--------------------------------------
To ask whether or not the IME want to handle the key, system need to send a request to IME and wait for the response. There are two ways:

1. If the request is asynchronous, then we won't forward key event to IME until receiving a message indicating that IME will handle this key(that is, we will dispatch key event directly to it's target app before IME tell system that it want to handle the key). Suppose that 'A', 'B' on physical keyboard will be converted respectively to 'ち' and 'こ', then we will expect the result is 'ちちここ' if we type 'AABB' on hardware keyboard. However, the result might be 'AちBこ' because system don't receive the response from IME yet when the first 'A' and first 'B' are pressed.

2. If the request is synchronous, then the above situation can be solved. System will wait for the response, and decide how to dispatch the key event depending on the response. However, this means that IME(content process) will block system(chrome process). That's not good.

To keep request asynchronous(content process shouldn't block chrome process) but the result won't be 'AちBこ', we can intercept keyboard event in nsPresShell instead of shell.js. 

# forward key event in nsPresShell
--------------------------------------
We actually can determine how to dispatch the key event before it's dispatched to system. We can intercept keyboard event and use asynchronous IPC to ask IME whether or not it want to handle the key(if IME call event.preventDefault(), it means that IME already handle the key). After receiving the feedback from IME, the key event routing path can be set. That is, we might need to back to old method, which will be similar to those reviewed patches.
(In reply to Chun-Min Chang[:chunmin] from comment #185)
> # forward key event in nsPresShell
> --------------------------------------
The main concern of this approach is performance. In usecase: IME is active, but IME doesn't want to handle this key, we need to use one async IPC round trip to know that IME won't handle it. One simple method is to use a blacklist-cache to remember which key doesn't need to be send to IME. Once system get refuse-to-handle of key 'X' from IME, then we can add key 'X' into blacklist-cache. After then, if key is in this blacklist-cache, then we can dispatch the key to its target directly.

On the other handle, whitelist is an alternative solution. If we could get a whitelist that contains all the key needed to be handled by IME, then we have no need to wait for the result from IME. However, every input-method has its own unique layout. It means that every input-method has its own unique whitelist. It will be a big work to build all whitelist we need.
(In reply to Chun-Min Chang[:chunmin] from comment #186)
> (In reply to Chun-Min Chang[:chunmin] from comment #185)
> > # forward key event in nsPresShell
> > --------------------------------------
> The main concern of this approach is performance. In usecase: IME is active,
> but IME doesn't want to handle this key, we need to use one async IPC round
> trip to know that IME won't handle it. One simple method is to use a
> blacklist-cache to remember which key doesn't need to be send to IME. Once
> system get refuse-to-handle of key 'X' from IME, then we can add key 'X'
> into blacklist-cache. After then, if key is in this blacklist-cache, then we
> can dispatch the key to its target directly.

That unfortunately won't work; IMEs might consider whether or not to consume a key based on it's own current state, e.g. having a candidate menu shown, has pending compositions etc.

> On the other handle, whitelist is an alternative solution. If we could get a
> whitelist that contains all the key needed to be handled by IME, then we
> have no need to wait for the result from IME. However, every input-method
> has its own unique layout. It means that every input-method has its own
> unique whitelist. It will be a big work to build all whitelist we need.

That's what we talked about last week; my only concern is that this API is quite unique compare to other text input frameworks/services in other OSes thus sounds like a workaround to our IPC-caused latency. I would rather get the interfaces right and revert the architectures, if possible; but that's separate conversation though.
(In reply to Chun-Min Chang[:chunmin] from comment #184)
> (In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) (offline until 1/6)
> from comment #181)
> > So, I think that this order isn't problem:
> > 
> > 1. handlable system first
> > 2. handlable IME next
> > 3. handlable app/content
> > 4. handlable system finally
> > 
> > With approach A and B, how this order will be changed?
> 
> With approach A and B, the order of |keydown| or |keyup| won't be changed. 
> However, the |afterkeydown| or |afterkeyup| will be changed. 
> Let's use |keydown| for example.
> 
> Usecase: IME is active, but IME doesn't want to handle this key
> 
> approach A
> -----------------
> 1) system receive 'beforekeydown' 
> 2) system receive 'keydown' fired from hardware
>    --> we will ask IME want to handle or not here. 
>    -----> wait for IME's response:
>           IME doesn't want to handle this key, so we will dispatch key event
> as original.
> 3) app(the event target) receive 'keydown' fired from hardware
> 4) system receive 'afterkeydown'
> 
> 
> approach B
> -----------------
> 1) system receive 'beforekeydown'
> 2) system receive 'keydown' fired from hardware
>    --> we don't need to ask IME here.
>        we will forward the key event's information to IME if it's active.
>    -----> system will call event.preventDefault() directly 
>           because we assume IME will handle it.
> 3) system receive 'afterkeydown' 
>    ('afterkeydown' will be fired upon system call event.preventDefault())
> 4) app(the event target) receive 'keydown' generated by IME
>    --> IME will generate a same key event as received one if IME doesn't
> want to handle this key.

Sounds like A is really better.

(In reply to Chun-Min Chang[:chunmin] from comment #185)
> # forward key event via shell.js
> --------------------------------------
> To ask whether or not the IME want to handle the key, system need to send a
> request to IME and wait for the response. There are two ways:
> 
> 1. If the request is asynchronous, then we won't forward key event to IME
> until receiving a message indicating that IME will handle this key(that is,
> we will dispatch key event directly to it's target app before IME tell
> system that it want to handle the key). Suppose that 'A', 'B' on physical
> keyboard will be converted respectively to 'ち' and 'こ', then we will expect
> the result is 'ちちここ' if we type 'AABB' on hardware keyboard. However, the
> result might be 'AちBこ' because system don't receive the response from IME
> yet when the first 'A' and first 'B' are pressed.
> 
> 2. If the request is synchronous, then the above situation can be solved.
> System will wait for the response, and decide how to dispatch the key event
> depending on the response. However, this means that IME(content process)
> will block system(chrome process). That's not good.
> 
> To keep request asynchronous(content process shouldn't block chrome process)
> but the result won't be 'AちBこ', we can intercept keyboard event in
> nsPresShell instead of shell.js.

Yeah, we shouldn't take synchronous IPC approach. We need to send physical keyboard events to another process asynchronously and wait the result from IPC.
 
> # forward key event in nsPresShell
> --------------------------------------
> We actually can determine how to dispatch the key event before it's
> dispatched to system. We can intercept keyboard event and use asynchronous
> IPC to ask IME whether or not it want to handle the key(if IME call
> event.preventDefault(), it means that IME already handle the key). After
> receiving the feedback from IME, the key event routing path can be set. That
> is, we might need to back to old method, which will be similar to those
> reviewed patches.

Yeah, it must be reasonable you to update the patch.

(In reply to Chun-Min Chang[:chunmin] from comment #186)
> The main concern of this approach is performance. In usecase: IME is active,
> but IME doesn't want to handle this key, we need to use one async IPC round
> trip to know that IME won't handle it.

I don't worry about the performance because other OSes must do same thing, i.e., sending key events from system process or something to focused process. If system (desktop or something) needs to intercept some key combination like Alt+Tab on Windows, it consumes key events and focused process won't receive such key events.

> One simple method is to use a
> blacklist-cache to remember which key doesn't need to be send to IME. Once
> system get refuse-to-handle of key 'X' from IME, then we can add key 'X'
> into blacklist-cache. After then, if key is in this blacklist-cache, then we
> can dispatch the key to its target directly.
>
> On the other handle, whitelist is an alternative solution. If we could get a
> whitelist that contains all the key needed to be handled by IME, then we
> have no need to wait for the result from IME. However, every input-method
> has its own unique layout. It means that every input-method has its own
> unique whitelist. It will be a big work to build all whitelist we need.

Yeah, such API becomes complicated and breaks compatibility with existing IMEs. So, I don't think that you need to worry about optimizing the keyboard event's performance at least now.

(In reply to Tim Guan-tin Chien [:timdream] (please needinfo) from comment #187)
> (In reply to Chun-Min Chang[:chunmin] from comment #186)
> > (In reply to Chun-Min Chang[:chunmin] from comment #185)
> > > # forward key event in nsPresShell
> > > --------------------------------------
> > The main concern of this approach is performance. In usecase: IME is active,
> > but IME doesn't want to handle this key, we need to use one async IPC round
> > trip to know that IME won't handle it. One simple method is to use a
> > blacklist-cache to remember which key doesn't need to be send to IME. Once
> > system get refuse-to-handle of key 'X' from IME, then we can add key 'X'
> > into blacklist-cache. After then, if key is in this blacklist-cache, then we
> > can dispatch the key to its target directly.
> 
> That unfortunately won't work; IMEs might consider whether or not to consume
> a key based on it's own current state, e.g. having a candidate menu shown,
> has pending compositions etc.

Absolutely.

> > On the other handle, whitelist is an alternative solution. If we could get a
> > whitelist that contains all the key needed to be handled by IME, then we
> > have no need to wait for the result from IME. However, every input-method
> > has its own unique layout. It means that every input-method has its own
> > unique whitelist. It will be a big work to build all whitelist we need.
> 
> That's what we talked about last week; my only concern is that this API is
> quite unique compare to other text input frameworks/services in other OSes
> thus sounds like a workaround to our IPC-caused latency. I would rather get
> the interfaces right and revert the architectures, if possible; but that's
> separate conversation though.

As I mentioned above, I don't worry about the performance since other OSes must do same thing (i.e., sending key events across process boundary). Or, is our IPC really slow than other OSes?
Attachment #8707271 - Attachment description: [WI] → [WIP] Simulate hardware keystroke in mochitest
Attachment #8670129 - Attachment is obsolete: true
Comment on attachment 8707271 [details] [diff] [review]
[WIP] Simulate hardware keystroke in mochitest

Review of attachment 8707271 [details] [diff] [review]:
-----------------------------------------------------------------

Hi Masayuki-san,
We need to use a new method to simulate native keyboard event after adding a flag |IsSynthesizedByTIP| in |KeyboardEvent|. The |mozbowserbeforekeyXX| and |mozbowserafterkeyXX| won't be fired when |KeyboardEvent.IsSynthesizedByTIP| is true, so some old tests like |test_dom_before_after_keyboard_event_remote.html| should not use |synthesizeKey| to send key event.

To simulate hardware keyboard event like real user's input, I use a shellscript to generate it and launch this shellscript via nsIFile and nsIProcess. I was wondering if you could share your opinions about this method.
Attachment #8707271 - Flags: feedback?(masayuki)
(In reply to Chun-Min Chang[:chunmin] from comment #190)
> To simulate hardware keyboard event like real user's input, I use a
> shellscript to generate it and launch this shellscript via nsIFile and
> nsIProcess. I was wondering if you could share your opinions about this
> method.
To simulate hardware key input on Linux, we might need third-party's library(e.g., xdotool[0], python-uinput[1]) or write one by ourselves.

For Android, we can simply generate any native hardware events by |sendevent| via adb shell

[0] https://github.com/jordansissel/xdotool
[1] http://tjjr.fi/sw/python-uinput/
(In reply to Chun-Min Chang[:chunmin] from comment #190)
> Comment on attachment 8707271 [details] [diff] [review]
> [WIP] Simulate hardware keystroke in mochitest
> 
> To simulate hardware keyboard event like real user's input, I use a
> shellscript to generate it and launch this shellscript via nsIFile and
> nsIProcess.

(In reply to Chun-Min Chang[:chunmin] from comment #191)
> To simulate hardware key input on Linux, we might need third-party's
> library(e.g., xdotool[0], python-uinput[1]) or write one by ourselves.
> 
> For Android, we can simply generate any native hardware events by
> |sendevent| via adb shell
> 
> [0] https://github.com/jordansissel/xdotool
> [1] http://tjjr.fi/sw/python-uinput/

Hi Andrew,
Could you give me some suggestions about this approach?
Flags: needinfo?(ahalberstadt)
I'm don't think I can provide much input here. I looked into this for mochitest about 5 years ago, but we abandoned the project after realizing it would be too much work. Maybe things have changed since then though.

I know that webdriver (the backend to selenium) has a native events module that works with Firefox. Maybe it would be possible to re-use their implementation with Gaia. Here's the best article I could find on it, though it hasn't been updated since 2013:
https://code.google.com/p/selenium/wiki/NativeEventsOnLinux
Flags: needinfo?(ahalberstadt)
Sorry for the delay. I'll try to check the patch during business trip from tomorrow.
(In reply to Andrew Halberstadt [:ahal] from comment #193)
Thank you for your information. 
Well, I'd like to know how to make a bash-script-file executable on try-server?
I might need to use command "$ chmod +x filename" on our testing environment to make this script executable.
Is it possible to package this bash-script-file into our testing environment?
Yeah you could either bake it into the docker image, or have the test machines download it from hg.mozilla.org. See either one of the images in testing/docker or testing/taskcluster/scripts for examples of other jobs that do this.
(In reply to Chun-Min Chang[:chunmin] from comment #190)
> Comment on attachment 8707271 [details] [diff] [review]
> [WIP] Simulate hardware keystroke in mochitest
> 
> Review of attachment 8707271 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Hi Masayuki-san,
> We need to use a new method to simulate native keyboard event after adding a
> flag |IsSynthesizedByTIP| in |KeyboardEvent|. The |mozbowserbeforekeyXX| and
> |mozbowserafterkeyXX| won't be fired when |KeyboardEvent.IsSynthesizedByTIP|
> is true, so some old tests like
> |test_dom_before_after_keyboard_event_remote.html| should not use
> |synthesizeKey| to send key event.

If the input transaction of TIP is for tests, shouldn't before and after key events be fired?
http://mxr.mozilla.org/mozilla-central/source/widget/TextEventDispatcher.cpp?rev=63101b99fa95&mark=136-136#131
(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #197)
> http://mxr.mozilla.org/mozilla-central/source/widget/TextEventDispatcher.
> cpp?rev=63101b99fa95&mark=136-136#131
Hi Masayuki-san,
I may not understand your point right. Do you mean that we can route the event dispatching path depends on |event.mFlags.mIsSynthesizedForTests| in our code? I personally think the testing part should be split from the executing code. Instead of adding a testing-specific flag into code, I prefer to find a way to simulate the hardware input event. If I misunderstand what you mean, please feel free to correct me.
(In reply to Chun-Min Chang[:chunmin] from comment #198)
> (In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #197)
> > http://mxr.mozilla.org/mozilla-central/source/widget/TextEventDispatcher.
> > cpp?rev=63101b99fa95&mark=136-136#131
> Hi Masayuki-san,
> I may not understand your point right. Do you mean that we can route the
> event dispatching path depends on |event.mFlags.mIsSynthesizedForTests| in
> our code?

Yes, I guess that you are trying to change PresShell to prevent to fire before/after key events if key events came from TIP. However, I don't think that the change should be applied when mIsSynthesizedForTests is true because automated tests are testing current code path. And if automated tests need to emulate VKB's keyboard events, they can use TIP with beginInputTransaction() instead of beginInputTransactionForTests().

> I personally think the testing part should be split from the
> executing code. Instead of adding a testing-specific flag into code, I
> prefer to find a way to simulate the hardware input event. If I
> misunderstand what you mean, please feel free to correct me.

If you need to dispatch hardware keyboard event, you should implement nsIDOMWindowUtils::SendNativeKeyEvent() in Gonk widget. However, I'm not sure if it's available with multi-process.
This patch is mostly copied from the old one.
Attachment #8672493 - Attachment is obsolete: true
Attachment #8672494 - Attachment is obsolete: true
Attachment #8672495 - Attachment is obsolete: true
Attachment #8672501 - Attachment is obsolete: true
Attachment #8672502 - Attachment is obsolete: true
Attachment #8702481 - Attachment is obsolete: true
Attachment #8702482 - Attachment is obsolete: true
Attachment #8712074 - Flags: review?(masayuki)
Attachment #8712075 - Flags: review?(masayuki)
Attachment #8712076 - Flags: review?(masayuki)
Attachment #8712078 - Flags: review?(masayuki)
Hi Masayuki-san,
Could you review these patches when you're available?
Assignee: hiro7998 → cchang
Attachment #8712081 - Flags: review?(masayuki)
Comment on attachment 8712081 [details] [diff] [review]
part6 - Interface between HardwareKeyHandler and Keyboard App

Review of attachment 8712081 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/inputmethod/Keyboard.jsm
@@ +156,5 @@
> +                               type: 'hardwareinput',
> +                               keyDict: keyboardEventDict
> +                              });
> +  },
> +

In the previous review, to keep good maintenance of the code, you'd like to see that event's information can be passed by flags.

If I understange corretly, |onHardwareKey| in |nsIHardwareKeyHandler.idl| should be redefined, like:

bool onHardwareKey(in string aType, /* keydown, keypress, keyup */
                   in unsigned long keyPacket,
                   in boolean aRepeat);

the |keyPacket| is an unsigned 32 bits value, its format is:

| codeNameIndex | keyNameIndex  | 
(16 bits + 16 bits)

We can use |keyNameIndex| to get the DOMKeyName and keyCode and |codeNameIndex| to get the DOMCodeName, then we can get |event.key|, |event.keyCode| and |event.code|.

To create a keyboard event with flags in javascript like:

new KeyboardEvent(type, keyPacket, repeat)

we need to add a new function in dom/keyboardevent.h/cpp

For me, the advantage to use bit-flag here is that we can pass other information(e.g., isComposing) without changing the interface. We only need to adjust the format of |keyPacket|. For example, 

| repeat  | isComposing | codeNameIndex | keyNameIndex  |
(1 bit + 1 bit + 15 bits + 15 bits)

However, the drawback of using flag is un-readable. Thus, I think some information like |type| should not be compressed into the flags
Comment on attachment 8712074 [details] [diff] [review]
part1 - add flags:IsSynthesizedByTIP and InputMethodAppState in WidgetKeyboardEvent

>@@ -845,17 +847,17 @@ TextInputProcessor::KeydownInternal(const WidgetKeyboardEvent& aKeyboardEvent,
>     return NS_OK;
>   }
>
>   aConsumedFlags |=
>     (status == nsEventStatus_eConsumeNoDefault) ? KEYDOWN_IS_CONSUMED :
>                                                   KEYEVENT_NOT_CONSUMED;
>
>   if (aAllowToDispatchKeypress &&
>-      mDispatcher->MaybeDispatchKeypressEvents(keyEvent, status,
>+      mDispatcher->MaybeDispatchKeypressEvents(keyEvent, status,

What's changing this line??

>   nsString mCodeValue;
>   // OS-specific native event can optionally be preserved
>   void* mNativeKeyEvent;
>   // Unique id associated with a keydown / keypress event. Used in identifing
>   // keypress events for removal from async event dispatch queue in metrofx
>   // after preventDefault is called on keydown events. It's ok if this wraps
>   // over long periods.
>   uint32_t mUniqueId;
>+  // Indicates whether the event is synthesized from Text Input Processor
>+  // or an actual event from nsAppShell.
>+  bool mIsSynthesizedByTIP;
>+
>+  // Indicates that the event is being handled by input method app
>+  typedef uint8_t InputMethodAppStateType;
>+  enum InputMethodAppState : InputMethodAppStateType
>+  {
>+    eNotHandled, // not yet handled by intput method app
>+    eHandling,   // being handled by intput method app
>+    eHandled     // handled by input method app
>+  };
>+  InputMethodAppState mInputMethodAppState;

nit: For memory alignment, all bool members should be defined at the end of each class/struct. However, WidgetWheelEvent is already messy. So, I'll sort it out later. Don't you mind this issue for now.

>+  /**
>+   * Reset the flags related to event flow control for redispatch
>+   */
>+  void ResetForReuse()
>+  {
>+    mFlags.mPropagationStopped = false;
>+    mFlags.mImmediatePropagationStopped = false;
>+    mFlags.mDefaultPrevented = false;
>+    mFlags.mDefaultPreventedByContent = false;
>+    mFlags.mDefaultPreventedByChrome = false;
>+    mFlags.mIsBeingDispatched = false;
>+    mFlags.mDispatchedAtLeastOnce = false;
>+  }

I don't think that it's good idea to reuse events because we haven't done that. And I think that this method should be in BaseEventFlags.


Let me check following patches for deciding if we should add ResetForReuse().
Comment on attachment 8712075 [details] [diff] [review]
part2 - Expose IsSynthesizedByTIP to KeyboardEvent

Looks okay to me, but I'd like smaug to review this too.
Attachment #8712075 - Flags: review?(masayuki)
Attachment #8712075 - Flags: review?(bugs)
Attachment #8712075 - Flags: review+
Comment on attachment 8712076 [details] [diff] [review]
part3 - HardwareKeyHandler component

>+++ b/dom/inputmethod/HardwareKeyHandler.cpp
>@@ -0,0 +1,434 @@
>+/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
>+/* vim: set ts=8 sts=2 et sw=2 tw=80: */
>+/* This Source Code Form is subject to the terms of the Mozilla Public
>+ * License, v. 2.0. If a copy of the MPL was not distributed with this
>+ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
>+
>+#include "HardwareKeyHandler.h"
>+#include "mozilla/BasicEvents.h"
>+#include "mozilla/ClearOnShutdown.h"
>+#include "mozilla/dom/TabParent.h"
>+#include "mozilla/EventDispatcher.h"
>+#include "mozilla/TextEvents.h"
>+#include "nsDeque.h"
>+#include "nsFocusManager.h"
>+#include "nsFrameLoader.h"
>+#include "nsIContent.h"
>+#include "nsPIDOMWindow.h"
>+#include "nsPresShell.h"
>+#include "mozilla/EventStateManager.h"

nit: please move this line after EventDispatcher.h?

>+NS_IMETHODIMP
>+HardwareKeyHandler::RegisterListener(nsIHardwareKeyEventListener* aListener)
>+{
>+  // Make sure that listenr is not nullptr and there is no availabe

s/listenr/listener
s/availabe/available

>+  // hardwareKeyEventListener now
>+  if (!aListener) {

NS_WARN_IF()?

>+    return NS_ERROR_NULL_POINTER;
>+  }
>+
>+  if (mHardwareKeyEventListener) {

NS_WARN_IF()?

>+    return NS_ERROR_ALREADY_INITIALIZED;
>+  }
>+
>+  mHardwareKeyEventListener = aListener;
>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+HardwareKeyHandler::ForwardKeyToInputMethodApp(nsINode* aTarget,
>+                                               WidgetKeyboardEvent* aEvent,
>+                                               nsEventStatus* aEventStatus,
>+                                               bool* aResult)
>+{
>+  MOZ_ASSERT(aTarget, "No target provided");
>+  MOZ_ASSERT(aEvent, "No event provided");
>+
>+  // No need to forward hardware key event to IME if:
>+  // 1. IME is inactive

If you mean, no editor has focus,
s/inactive/disabled

>+  // 2. There is no nsIHardwareKeyEventListener in use
>+  // 3. This key event is genereated IME(by nsITextInputProcessor)

s/genereated/generated by

>+  // 4. The key event is already handling
>+  if (!mInputMethodAppConnected ||
>+      !mHardwareKeyEventListener ||
>+      aEvent->mIsSynthesizedByTIP ||
>+      aEvent->mInputMethodAppState != WidgetKeyboardEvent::eNotHandled) {
>+    *aResult = false;
>+    return NS_OK;
>+  }
>+
>+  // We only forward keydown and keyup event to input-method-app
>+  // because input-method-app will generate keypress by itself.
>+  // The keypress event will be pushed into queue without forwarding.
>+  if (aEvent->mMessage != eKeyPress) {
>+    // Retrieve key for the event
>+    aEvent->GetDOMKeyName(aEvent->mKeyValue);
>+
>+    // Retrieve code for the event
>+    aEvent->GetDOMCodeName(aEvent->mCodeValue);
>+
>+    // Try forwarding this key event to input-method-app
>+    bool isSent = false;
>+    mHardwareKeyEventListener->OnHardwareKey(((aEvent->mMessage == eKeyDown)?
>+                                               "keydown" : "keyup"),
>+                                             NS_ConvertUTF16toUTF8(aEvent->mKeyValue).get(), //keyValue,
>+                                             NS_ConvertUTF16toUTF8(aEvent->mCodeValue).get(),//codeValue,
>+                                             aEvent->mIsRepeat,
>+                                             aEvent->keyCode,
>+                                             aEvent->charCode,
>+                                             aEvent->time,
>+                                             &isSent);

Looks like creating dom::KeyboardEvent instance use it as the argument of OnHardwareKey. See also my comment at the definition of this.

>+NS_IMETHODIMP
>+HardwareKeyHandler::OnHandledByInputMethodApp(const nsAString& aType,
>+                                              bool aDefaultPrevented)
>+{
>+  // We can not handle this reply because the pending event had been already
>+  // removed from the forwarding queue before this reply arrives.
>+  if (!mEventQueue.GetSize()) {
>+    return NS_OK;
>+  }
>+
>+  nsAutoPtr<KeyboardInfo>
>+    keyInfo(static_cast<KeyboardInfo*>(mEventQueue.PopFront()));
>+
>+  // Only allow keydown and keyup to call this method
>+  if (NS_WARN_IF(aType.EqualsLiteral("keydown") &&
>+                 keyInfo->event->mMessage != eKeyDown) ||
>+      NS_WARN_IF(aType.EqualsLiteral("keyup") &&
>+                 keyInfo->event->mMessage != eKeyUp)) {
>+    return NS_ERROR_INVALID_ARG;
>+  }
>+
>+  // Reset the flags to dispatch this event again.
>+  keyInfo->event->ResetForReuse();

I wonder, why don't you just create new WidgetKeyboardEvent instance in the stack? As I said, like ResetForReuse() approach hasn't been used and hard to maintain when new flags are added. So, I strongly recommend that you should create WidgetKeyboardEvent again.

>+  keyInfo->event->mFlags.mDefaultPrevented = aDefaultPrevented;

I wonder, how about mDefaultPrevetnedBy*?

>+  // Set the flag to specify the reply phase.
>+  keyInfo->event->mInputMethodAppState = WidgetKeyboardEvent::eHandled;
>+
>+  // If |aDefaultPrevented| is true, it means that input-method-app has already
>+  // consumed this key, so we can dispatch |mozbrowserafterkeyXXX| now if
>+  // preference "dom.beforeAfterKeyboardEvent.enabled" is enabled.
>+  if (aDefaultPrevented) {
>+    DispatchAfterKeyEvent(keyInfo->target, keyInfo->event);
>+  // Otherwise, it means that input-method-app doesn't handle this key,
>+  // so we need to dispatch it to its original event target.
>+  } else {
>+    DispatchToTargetApp(keyInfo->target, keyInfo->event, keyInfo->status);
>+  }

Don't we need to check if the event target is valid (e.g., still in the DOM tree) if IME app can cause firing some events in the content?

>+  // Handle the pending keypress event once keydown's reply arrives:
>+  // If the event is keydown and it's consumed by input-method-app,
>+  // then there is no need to dispatch the following keypress to its
>+  // original event target. On the other hand, if the keydown event
>+  // doesn't be consumed by input-method-app, then the following
>+  // keypress need to be dispatched to its original event target.
>+  if (keyInfo->event->mMessage != eKeyDown) {
>+    return NS_OK;
>+  }

You've dispatched keyboard down event above, so, we don't know the document and event target state. Please check it first before dispatching next event.

>+
>+  nsAutoPtr<KeyboardInfo>
>+    keypressInfo(static_cast<KeyboardInfo*>(mEventQueue.PeekFront()));
>+
>+  // Make sure that the next of keydown is keypress
>+  MOZ_ASSERT(keypressInfo->event->mMessage == eKeyPress,
>+             "the keydown's next event is not keypress.");

If the keydown event is modifier key's, keypress event shouldn't be fired in Gecko. And also, for conforming UI Events spec, we shouldn't dispatch keypress events for non-printable keys in the future. So, if there is no keypress events, you can just do nothing.

And I guess that Gonk won't dispatch two or more keypress events per keydown event. But it's possible in other platforms. So, please use loop here even though this is used only on B2G now.

>+bool
>+HardwareKeyHandler::DispatchToTargetApp(nsINode* aTarget,
>+                                        WidgetKeyboardEvent* aEvent,
>+                                        nsEventStatus* aStatus)
>+{
>+  MOZ_ASSERT(aTarget, "No target provided");
>+  MOZ_ASSERT(aEvent, "No event provided");
>+
>+  // Get current focus content as the event target
>+  nsCOMPtr<nsIContent> currentFocusTarget = GetFocusContent();
>+  if(!currentFocusTarget) {
>+    return false;
>+  }
>+
>+  nsIContent* originalTarget = aTarget->AsContent();
>+
>+  // Check the event target is same as the current focus element or not.
>+  // The event target will change if focus is changed.
>+  // If the event target is changed, then what should we do?
>+  if (originalTarget != currentFocusTarget) {
>+    NS_WARNING("The current focus element is different from the origial event target.");
>+  }

We should fire on new focused element. That's valid behavior for UI Events spec.

>+  // After the oop target recevices the event from TabChild::RecvRealKeyEvent

s/recevices/receives

>+  // and retrun the result through TabChild::SendDispatchAfterKeyboardEvent,

s/retrun/return

>+bool
>+HardwareKeyHandler::DispatchToCrossProcess(nsINode* aTarget,
>+                                           WidgetKeyboardEvent* aEvent)
>+{
>+  nsCOMPtr<nsIFrameLoaderOwner> remoteLoaderOwner = do_QueryInterface(aTarget);
>+  if (!remoteLoaderOwner) {

NS_WARN_IF()?

>+    return false;
>+  }
>+
>+  RefPtr<nsFrameLoader> remoteFrameLoader =
>+    remoteLoaderOwner->GetFrameLoader();
>+  if (!remoteFrameLoader) {

NS_WARN_IF()?

>+    return false;
>+  }
>+
>+  uint32_t eventMode;
>+  remoteFrameLoader->GetEventMode(&eventMode);
>+  if (eventMode == nsIFrameLoader::EVENT_MODE_DONT_FORWARD_TO_CHILD) {
>+    return false;
>+  }
>+
>+  PBrowserParent* remoteBrowser = remoteFrameLoader->GetRemoteBrowser();
>+  TabParent* remote = static_cast<TabParent*>(remoteBrowser);
>+  if (!remote) {

NS_WARN_IF()?

>+already_AddRefed<nsIContent>
>+HardwareKeyHandler::GetFocusContent()
>+{
>+  nsFocusManager* fm = nsFocusManager::GetFocusManager();
>+  if (!fm) {

NS_WARN_IF()

>+    return nullptr;
>+  }
>+
>+  nsCOMPtr<nsIDOMWindow> focusedWindow;
>+  fm->GetFocusedWindow(getter_AddRefs(focusedWindow));
>+  nsCOMPtr<nsPIDOMWindow> ourWindow = do_QueryInterface(focusedWindow);
>+  if (!ourWindow) {

NS_WARN_IF()

>+    return nullptr;
>+  }
>+
>+  nsPIDOMWindow* rootWindow = ourWindow->GetPrivateRoot();
>+  if (!rootWindow) {

NS_WARN_IF()

>+    return nullptr;
>+  }
>+
>+  nsCOMPtr<nsPIDOMWindow> focusedFrame;
>+  nsCOMPtr<nsIContent> focusedContent =
>+    fm->GetFocusedDescendant(rootWindow, true, getter_AddRefs(focusedFrame));
>+
>+  return focusedContent.forget();

Hmm, this returns event target of keyboard event. So, if there is no focused element, keyboard events should be fired on <body> or something (not sure the exactly target, though).

So, I think that you should rename this method and when there is no focused element in the window, you should return <body> or something.

>+++ b/dom/inputmethod/HardwareKeyHandler.h
>@@ -0,0 +1,82 @@
>+/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
>+/* vim: set ts=8 sts=2 et sw=2 tw=80: */
>+/* This Source Code Form is subject to the terms of the Mozilla Public
>+ * License, v. 2.0. If a copy of the MPL was not distributed with this
>+ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
>+

Please add include guard like:

#ifndef mozilla_HardwareKeyHandler_h_
#define mozilla_HardwareKeyHandler_h_


And add
#endif // #ifndef mozilla_HardwareKeyHandler_h_
at the end of this file.

>+#include "mozilla/StaticPtr.h"
>+#include "nsCOMPtr.h"
>+#include "nsDeque.h"
>+#include "mozilla/EventForwards.h"  // For nsEventStatus
>+#include "nsIHardwareKeyHandler.h"

Could you sort the include order from A to Z?

>+
>+class nsDeque;
>+class nsIPresShell;
>+class nsIContent;
>+class nsINode;

And this order too.

>+
>+namespace mozilla {
>+class WidgetKeyboardEvent;
>+
>+struct KeyboardInfo {

put the |{| (of struct/class/enum) to next line.

>+  WidgetKeyboardEvent* event;
>+  nsINode* target;
>+  nsEventStatus* status;

Please add |m| prefix for member variables.

>+
>+  KeyboardInfo(WidgetKeyboardEvent* aEvent, nsINode* aTarget, nsEventStatus* aStatus)

Too long line. Breaking after each |,| must be easier to read.

>+  {
>+    event = aEvent;
>+    target = aTarget;
>+    status = aStatus;
>+  }
>+};
>+
>+class HardwareKeyHandler : public nsIHardwareKeyHandler
>+{
>+public:
>+  HardwareKeyHandler();
>+
>+  NS_DECL_ISUPPORTS
>+  NS_DECL_NSIHARDWAREKEYHANDLER

I wonder, don't you need to add this class to cycle collector? Because this class may be held by JS.

>+
>+  static already_AddRefed<HardwareKeyHandler> GetInstance();
>+
>+private:
>+  virtual ~HardwareKeyHandler();
>+
>+  // To keep tracking whether or not the input-method-app is active
>+  bool mInputMethodAppConnected;

Please move bool member to the end of this class (for memory alignment). In this case, after mHardwareKeyEventListener.

>diff --git a/dom/inputmethod/nsIHardwareKeyHandler.idl b/dom/inputmethod/nsIHardwareKeyHandler.idl
>+namespace mozilla {
>+  class WidgetKeyboardEvent;

nit: unnecessary indent.

>+}
>+
>+class nsINode;
>+
>+#include "mozilla/EventForwards.h" /* For nsEventStatus */
>+%}
>+
>+[ptr] native WidgetKeyboardEventPtr(mozilla::WidgetKeyboardEvent);
>+[ptr] native nsINodePtr(nsINode);
>+[ptr] native nsEventStatusPtr(nsEventStatus);
>+
>+/**
>+ * This interface should be registered to nsIHardwareKeyHandler through
>+ * |nsIHardwareKeyHandler.registerListener|. When nsIHardwareKeyHandler needs
>+ * to forward a hardware keyboard event to the active input method, this
>+ * listener is invoked to communicate with the active input method. Therefore,
>+ * this listener implementation should contain the communication with the
>+ * active input method.
>+ */
>+[scriptable, function, uuid(cd5aeee3-b4b9-459d-85e7-c0671c7a8a2e)]
>+interface nsIHardwareKeyEventListener : nsISupports
>+{
>+  /**
>+   * This method is called by nsIHardwareKeyHandler, when nsIHardwareKeyHandler
>+   * needs to forward a hardware keyboard event to the active input method.
>+   * Therefore, thie method implementation should forward a hardware keyboard

s/thie/this

>+   * event to the active input method.
>+   */
>+  bool onHardwareKey(in string aType,
>+                     in string aKey,
>+                     in string aCode,
>+                     in boolean aRepeat,
>+                     in unsigned long aKeyCode,
>+                     in unsigned long aCharCode,
>+                     in unsigned long long aTimeStamp);

I wonder, isn't using nsIDOMKeyEvent better for compatibility with future? Like this arguments requires to change the method when DOM Events will add new attributes to KeyboardEvent.

>+};
>+
>+/**
>+ * This interface has two main roles. One is to send a hardware keyboard event
>+ * to the active input method app and the second is to receive its result.

s/the second/the other

>+ * If a keyboard event is triggered from a hardware keyboard when an editor has
>+ * a focus, the event target should be an editor. However, the text input

s/a focus/focus
s/should be an editor/should be the editor

>+ * processor algorithm is implemented in an input method app and it should
>+ * handle the event earlier than the real event target to do the mapping such
>+ * as character conversion according to the language setting or the type of a
>+ * hardware keyboard.
>+ */
>+[scriptable, builtinclass, uuid(25b34270-caad-4d18-a910-860351690639)]
>+interface nsIHardwareKeyHandler : nsISupports
>+{
>+  /**
>+   * Registers a listener which has the communication channel with the active
>+   * input method to forward a hardware keyboard event.
>+   */
>+  void registerListener(in nsIHardwareKeyEventListener aListener);
>+
>+  /**
>+   * Sends a hardware keyboard event to the active input method app to give the
>+   * first chance to handle a hardware keyboard event. Internally, this method

s/a hardware keyboard event/the event

>+   * implementation should call |nsIHardwareKeyEventListener.onHardwareKey| of
>+   * the listener registered through |registerListener|. As a result, the input
>+   * method app having the active input method can receive a hardware keyboard
>+   * event through this method call.
>+   *
>+   * Returns true, if the event is sent to the active input method app.
>+   * Returns false, if the event is not sent.
>+   *
>+   * If it returns true, the caller should pause the further event processing
>+   * to wait for the completion of the event handling in input method app.
>+   * Once |onHandledByInputMethodApp| is called, the pending event processing
>+   * should be resumed according to the result from the active input method app.
>+   * If it returns false, the caller should continue the normal event
>+   * processing.
>+   */
>+  bool forwardKeyToInputMethodApp(in nsINodePtr aTarget,
>+                                  in WidgetKeyboardEventPtr aEvent,
>+                                  in nsEventStatusPtr aEventStatus);

Shouldn't this be [noscript]?
Attachment #8712076 - Flags: review?(masayuki) → review-
Comment on attachment 8712074 [details] [diff] [review]
part1 - add flags:IsSynthesizedByTIP and InputMethodAppState in WidgetKeyboardEvent

Now, I think we shouldn't create ResetForReuse().
Attachment #8712074 - Flags: review?(masayuki) → review-
Comment on attachment 8712078 [details] [diff] [review]
part4 - Interface between PresShell and HardwareKeyHandler

>--- a/layout/base/nsIPresShell.h
>+++ b/layout/base/nsIPresShell.h
>@@ -874,16 +875,21 @@ public:
>   /**
>    * Dispatch AfterKeyboardEvent with specific target.
>    */
>   virtual void DispatchAfterKeyboardEvent(nsINode* aTarget,
>                                           const mozilla::WidgetKeyboardEvent& aEvent,
>                                           bool aEmbeddedCancelled) = 0;
> 
>   /**
>+   * Return whethor or not event can be dispatched

s/whethor/whether

>+   */
>+  virtual bool CanDispatchEvent(const mozilla::WidgetGUIEvent* aEvent = nullptr) = 0;

You need to modify UUID of nsIPresShell due to adding this virtual method.

>@@ -1707,16 +1713,19 @@ protected:
> #endif
> 
>   // At least on Win32 and Mac after interupting a reflow we need to post
>   // the resume reflow event off a timer to avoid event starvation because
>   // posted messages are processed before other messages when the modal
>   // moving/sizing loop is running, see bug 491700 for details.
>   nsCOMPtr<nsITimer>        mReflowContinueTimer;
> 
>+  // Forward hardware key event to IME

s/event/events
s/IME/IME apps on B2G

>+  nsCOMPtr <nsIHardwareKeyHandler> mHardwareKeyHandler;

unnecessary whitespace after nsCOMPtr and before <.

>--- a/layout/base/nsPresShell.cpp
>+++ b/layout/base/nsPresShell.cpp
>@@ -727,16 +728,22 @@ PresShell::BeforeAfterKeyboardEventEnabled()
>   if (!sInitialized) {
>     Preferences::AddBoolVarCache(&sBeforeAfterKeyboardEventEnabled,
>       "dom.beforeAfterKeyboardEvent.enabled");
>     sInitialized = true;
>   }
>   return sBeforeAfterKeyboardEventEnabled;
> }
> 
>+/* static */ bool
>+PresShell::IsTargetIframe(nsINode* aTarget) {

Put |{| to the next line.

>@@ -776,16 +783,17 @@ PresShell::PresShell()
>                                  "dom.w3c_pointer_events.enabled", true);
>     addedPointerEventEnabled = true;
>   }
> 
>   mPaintingIsFrozen = false;
>   mHasCSSBackgroundColor = true;
>   mIsLastChromeOnlyEscapeKeyConsumed = false;
>   mHasReceivedPaintMessage = false;
>+  mHardwareKeyHandler = nullptr;

mHardwareKeyHandler is nsCOMPtr. So, you don't need to set it to nullptr explicitly.

>@@ -947,16 +955,21 @@ PresShell::Init(nsIDocument* aDocument,
> 
>   if (mPresContext->IsRootContentDocument()) {
>     mZoomConstraintsClient = new ZoomConstraintsClient();
>     mZoomConstraintsClient->Init(this, mDocument);
>     if (gfxPrefs::MetaViewportEnabled() || gfxPrefs::APZAllowZooming()) {
>       mMobileViewportManager = new MobileViewportManager(this, mDocument);
>     }
>   }
>+
>+  if (XRE_IsParentProcess()) {
>+    nsresult rv;
>+    mHardwareKeyHandler = do_GetService("@mozilla.org/HardwareKeyHandler;1", &rv);
>+  }

Hmm, this means that you create HardwareKeyHandler instances on every platform. You should restrict this. If you want to create it even on desktop for tests, perhaps, using a pref is better? Otherwise, using #ifdef? Anyway, maybe it's better GetInstance() to return nullptr if it shouldn't be created.

> bool
>-PresShell::CanDispatchEvent(const WidgetGUIEvent* aEvent) const
>+PresShell::CanDispatchEvent(const WidgetGUIEvent* aEvent)

Cannot you to define this method with const in nsIPresShell?

> void
> PresShell::HandleKeyboardEvent(nsINode* aTarget,
>                                WidgetKeyboardEvent& aEvent,
>                                bool aEmbeddedCancelled,
>                                nsEventStatus* aStatus,
>                                EventDispatchingCallback* aEventCB)
> {
>-  if (aEvent.mMessage == eKeyPress ||
>-      !BeforeAfterKeyboardEventEnabled()) {
>+  MOZ_ASSERT(aTarget);
>+
>+  if (aEvent.mIsSynthesizedByTIP) {

How about if the event was synthesized for tests?

>     EventDispatcher::Dispatch(aTarget, mPresContext,
>                               &aEvent, nullptr, aStatus, aEventCB);
>     return;
>   }
> 
>-  MOZ_ASSERT(aTarget);
>+  // return true if the real event target is in its child process
>+  bool targetIsIframe = IsTargetIframe(aTarget);
>+
>+  if (aEvent.mMessage == eKeyPress ||
>+      !BeforeAfterKeyboardEventEnabled()) {
>+    FinishDispatchingKeyboardEvent(targetIsIframe, aTarget, aEvent, aStatus, aEventCB);

too long line, please wrap after |aEvent,|.

>@@ -6802,19 +6820,19 @@ PresShell::HandleKeyboardEvent(nsINode* aTarget,
>     return;
>   }
> 
>   // Event listeners may kill nsPresContext and nsPresShell.
>   if (!CanDispatchEvent()) {
>     return;
>   }
> 
>-  // Dispatch actual key event to event target.
>-  EventDispatcher::Dispatch(aTarget, mPresContext,
>-                            &aEvent, nullptr, aStatus, aEventCB);
>+  if (FinishDispatchingKeyboardEvent(targetIsIframe, aTarget, aEvent, aStatus, aEventCB)) {

Too long line...

>+    return;
>+  }
> 
>   if (aEvent.mFlags.mDefaultPrevented) {
>     // When embedder prevents the default action of actual key event, attribute
>     // 'embeddedCancelled' of after event is false, i.e. |!targetIsIframe|.
>     // On the contrary, if the defult action is prevented by embedded iframe,
>     // 'embeddedCancelled' is true which equals to |!targetIsIframe|.
>     DispatchAfterKeyboardEventInternal(chain, aEvent, !targetIsIframe, chainIndex);
>     return;
>@@ -6825,16 +6843,74 @@ PresShell::HandleKeyboardEvent(nsINode* aTarget,
>     return;
>   }
> 
>   // Dispatch after events to all items in the chain.
>   DispatchAfterKeyboardEventInternal(chain, aEvent,
>                                      aEvent.mFlags.mDefaultPrevented);
> }
> 
>+bool
>+PresShell::FinishDispatchingKeyboardEvent(bool isTargetRemote,

Should be aIsTargetRemote.

>+                                          nsINode* aTarget,
>+                                          WidgetKeyboardEvent& aEvent,
>+                                          nsEventStatus* aStatus,
>+                                          EventDispatchingCallback* aEventCB)
>+{
>+  // In-process case: the real event target is in the current process
>+  if (!isTargetRemote) {
>+    if(ForwardKeyToInputMethodApp(aTarget, aEvent, aStatus)) {
>+      return true;
>+    }
>+
>+    // If keyboard doesn't be forward to input-method-app, then it should be

s/doesn't be/isn't

>+    // dispatched to its event target.
>+    EventDispatcher::Dispatch(aTarget, mPresContext,
>+                              &aEvent, nullptr, aStatus, aEventCB);
>+
>+    return false;
>+  }
>+
>+  // oop case: the real event target is in the child process of the current one.
>+  // Dispatch the keyboard event to the iframe that embed the remote
>+  // event target first.
>+  EventDispatcher::Dispatch(aTarget, mPresContext,
>+                            &aEvent, nullptr, aStatus, aEventCB);
>+
>+  if (aEvent.mFlags.mDefaultPrevented) {
>+    return false;
>+  }
>+
>+  // Check whethor or not the event should be forwarded to input-method-app.

s/whethor/whether

>+  return ForwardKeyToInputMethodApp(aTarget, aEvent, aStatus);
>+}
>+
>+bool
>+PresShell::ForwardKeyToInputMethodApp(nsINode* aTarget,
>+                                      WidgetKeyboardEvent& aEvent,
>+                                      nsEventStatus* aStatus)
>+{
>+  if (!mHardwareKeyHandler) {
>+    return false;
>+  }
>+
>+  bool forwarded = false;
>+  mHardwareKeyHandler->ForwardKeyToInputMethodApp(aTarget,
>+                                                  aEvent.AsKeyboardEvent(),
>+                                                  aStatus,
>+                                                  &forwarded);
>+  if (forwarded) {
>+    // No need to dispatch the forwarded keyboard event to it's child procss

s/procss/process

>+++ b/layout/base/nsPresShell.h
>@@ -747,17 +751,24 @@ protected:
>          const mozilla::WidgetKeyboardEvent& aEvent,
>          size_t& aChainIndex,
>          bool& aDefaultPrevented);
>   void DispatchAfterKeyboardEventInternal(
>          const nsTArray<nsCOMPtr<mozilla::dom::Element> >& aChain,
>          const mozilla::WidgetKeyboardEvent& aEvent,
>          bool aEmbeddedCancelled,
>          size_t aChainIndex = 0);
>-  bool CanDispatchEvent(const mozilla::WidgetGUIEvent* aEvent = nullptr) const;
>+  bool FinishDispatchingKeyboardEvent(bool isTargetRemote,
>+                                      nsINode* aTarget,
>+                                      mozilla::WidgetKeyboardEvent& aEvent,
>+                                      nsEventStatus* aStatus,
>+                                      mozilla::EventDispatchingCallback* aEventCB);
>+  bool ForwardKeyToInputMethodApp(nsINode* aTarget,
>+                                  mozilla::WidgetKeyboardEvent& aEvent,
>+                                  nsEventStatus* aStatus);

I'd like you to explain what these methods do with comment. Especially, the result is unclear.


Sorry for the delay to review. Unfortunately, time over today, I'll check remaining patches tomorrow and I should check this patch more carefully.
Attachment #8712078 - Flags: review?(masayuki) → review-
Comment on attachment 8712075 [details] [diff] [review]
part2 - Expose IsSynthesizedByTIP to KeyboardEvent

So we don't use KeyboardEventInit.isSynthesizedByTIP for anything, yet expose it to the web? (if one passes an object which has a getter isSynthesizedByTIP to constructor, that getter gets called.)
Doesn't sounds quite right.

Drop isSynthesizedByTIP from the dictionary and just have the [ChromeOnly] attribute in the interface. With that, r+.

If you do other changes, ask review again.
Attachment #8712075 - Flags: review?(bugs) → review+
(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #209)
Hi Masayuki-san,
Thank you for sharing your time to review the code.

> Comment on attachment 8712076 [details] [diff] [review]
> part3 - HardwareKeyHandler component
> 
> >+++ b/dom/inputmethod/HardwareKeyHandler.cpp 
> >+  keyInfo->event->mFlags.mDefaultPrevented = aDefaultPrevented;
> 
> I wonder, how about mDefaultPrevetnedBy*?

Do we really need to get the result of mDefaultPrevetnedBy*? The |OnHandledByInputMethodApp| will be called by Mozkeyboard.js or Keyboard.jsm, and we can't know the the value of mDefaultPrevetnedBy* from javascript. If we really need to pass the value of mDefaultPrevetnedBy*, we might need to set it by ourselves(need find the answer of Event::IsChrome first?)

And the following dispatching won't be affected if the |event.defaultPrevented| is ture, no matter the event is prevented by chrome or content. If we need to set the value of mDefaultPrevetnedBy*, I 'll add it in next version.

> >+bool
> >+HardwareKeyHandler::DispatchToTargetApp(nsINode* aTarget,
> >+                                        WidgetKeyboardEvent* aEvent,
> >+                                        nsEventStatus* aStatus)
> >+{
> >+  MOZ_ASSERT(aTarget, "No target provided");
> >+  MOZ_ASSERT(aEvent, "No event provided");
> >+
> >+  // Get current focus content as the event target
> >+  nsCOMPtr<nsIContent> currentFocusTarget = GetFocusContent();
> >+  if(!currentFocusTarget) {
> >+    return false;
> >+  }
> >+
> >+  nsIContent* originalTarget = aTarget->AsContent();
> >+
> >+  // Check the event target is same as the current focus element or not.
> >+  // The event target will change if focus is changed.
> >+  // If the event target is changed, then what should we do?
> >+  if (originalTarget != currentFocusTarget) {
> >+    NS_WARNING("The current focus element is different from the origial event target.");
> >+  }
> 
> We should fire on new focused element. That's valid behavior for UI Events
> spec.

I have a question here:

Suppose beforekey* and afterkey* are enabled. Imagine the following situation: 
1) At time T0, we focus on a input element in a remote app A and 
   the keyboard event is forwarding to IME via IPC during T0~T2, 
   from Keyboard.jsm to MozKeyboard.js. 
2) At time T1, the focus is changed to another element in remote app B due to some reason. 
3) At time T2, IME gets the key event and decides that it doesn't want to handle this key,
   so MozKeyboard.js sends the result back to Keyboard.jsm during T2~T3.
4) At time T3, the Keyboard.jsm gets the result that the |event.defaultPrevented| is false
   and pass the result back to |HardwareKeyHandler.onHandledByInputMethodApp|
5) At time T4, HardwareKeyHandler should dispatch the key event as original path.

In this case, if we dispatch the event to the focus element, which locates in app B, then we will have following problems:

1. The beforeKey* should be dispatched to the mozbrowser-embedder, and the mozbrowser-embedder of the focus element in app A and the one in app B may be different.
  e.g., In nested-oop case: 
  if the focus element is inside a remote mozbrowser iframe in the remote app A or B, 
  then the mozbrowser-embedder of focus element in app A and B will be different.

2. Do we need to dispatch the afterKey* event for the original event target? Do we need to dispatch the afterKey* event for the new event target? Same as above, the afterKey*'s event targets may be different after focus is changed. 

If we take the latest focus element as the new event target, we might need to restart the event propagation and dispatch a new keyboard event because I think they are different events now after changing event targets. However, how to deal with those un-dispatched pending keyboard events? Drop them?
(In reply to Chun-Min Chang[:chunmin] from comment #213)
> (In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #209)
> Hi Masayuki-san,
> Thank you for sharing your time to review the code.
> 
> > Comment on attachment 8712076 [details] [diff] [review]
> > part3 - HardwareKeyHandler component
> > 
> > >+++ b/dom/inputmethod/HardwareKeyHandler.cpp 
> > >+  keyInfo->event->mFlags.mDefaultPrevented = aDefaultPrevented;
> > 
> > I wonder, how about mDefaultPrevetnedBy*?
> 
> Do we really need to get the result of mDefaultPrevetnedBy*?

If it's possible, I think so.

> The |OnHandledByInputMethodApp| will be called by Mozkeyboard.js or
> Keyboard.jsm, and we can't know the the value of mDefaultPrevetnedBy* from
> javascript. If we really need to pass the value of mDefaultPrevetnedBy*, we
> might need to set it by ourselves(need find the answer of Event::IsChrome
> first?)

Indeed, it doesn't make sense to expose them only for now. If there will be some bugs caused by this issue, we should do that, though.

> And the following dispatching won't be affected if the
> |event.defaultPrevented| is ture, no matter the event is prevented by chrome
> or content. If we need to set the value of mDefaultPrevetnedBy*, I 'll add
> it in next version.

Okay, keep current design. We should try to fix if that causes some trouble actually.

> > >+bool
> > >+HardwareKeyHandler::DispatchToTargetApp(nsINode* aTarget,
> > >+                                        WidgetKeyboardEvent* aEvent,
> > >+                                        nsEventStatus* aStatus)
> > >+{
> > >+  MOZ_ASSERT(aTarget, "No target provided");
> > >+  MOZ_ASSERT(aEvent, "No event provided");
> > >+
> > >+  // Get current focus content as the event target
> > >+  nsCOMPtr<nsIContent> currentFocusTarget = GetFocusContent();
> > >+  if(!currentFocusTarget) {
> > >+    return false;
> > >+  }
> > >+
> > >+  nsIContent* originalTarget = aTarget->AsContent();
> > >+
> > >+  // Check the event target is same as the current focus element or not.
> > >+  // The event target will change if focus is changed.
> > >+  // If the event target is changed, then what should we do?
> > >+  if (originalTarget != currentFocusTarget) {
> > >+    NS_WARNING("The current focus element is different from the origial event target.");
> > >+  }
> > 
> > We should fire on new focused element. That's valid behavior for UI Events
> > spec.
> 
> I have a question here:
> 
> Suppose beforekey* and afterkey* are enabled. Imagine the following
> situation: 
> 1) At time T0, we focus on a input element in a remote app A and 
>    the keyboard event is forwarding to IME via IPC during T0~T2, 
>    from Keyboard.jsm to MozKeyboard.js. 
> 2) At time T1, the focus is changed to another element in remote app B due
> to some reason. 
> 3) At time T2, IME gets the key event and decides that it doesn't want to
> handle this key,
>    so MozKeyboard.js sends the result back to Keyboard.jsm during T2~T3.
> 4) At time T3, the Keyboard.jsm gets the result that the
> |event.defaultPrevented| is false
>    and pass the result back to |HardwareKeyHandler.onHandledByInputMethodApp|
> 5) At time T4, HardwareKeyHandler should dispatch the key event as original
> path.
> 
> In this case, if we dispatch the event to the focus element, which locates
> in app B, then we will have following problems:
> 
> 1. The beforeKey* should be dispatched to the mozbrowser-embedder, and the
> mozbrowser-embedder of the focus element in app A and the one in app B may
> be different.
>   e.g., In nested-oop case: 
>   if the focus element is inside a remote mozbrowser iframe in the remote
> app A or B, 
>   then the mozbrowser-embedder of focus element in app A and B will be
> different.

I don't 100% understand this scenario, though.

beforekeydown and afterkeydown should be fired on same target as keydown. Similarly, beforekeyup and afterkeyup should be fired on same target as keyup.

On the other hand, keydown, keyup and keypress should be fired on current focused element in active app since this is defined by standard spec. If you worry about something security issues, I think it's okay not to dispatch keypress and keyup events if active app is changed. But we shouldn't dispatch key events to non-focused element. It really breaks the standard spec.

> 2. Do we need to dispatch the afterKey* event for the original event target?
> Do we need to dispatch the afterKey* event for the new event target? Same as
> above, the afterKey*'s event targets may be different after focus is
> changed. 

So, I think the original event target is the right target of afterKey*.

Note that the purpose of beforeKey* and afterKey* are internal use only for the system. So, we don't have any rules for them but we should keep its behavior to work for the purpose.
Attachment #8712074 - Attachment is obsolete: true
Attachment #8712075 - Attachment is obsolete: true
Attachment #8712076 - Attachment is obsolete: true
Attachment #8712078 - Attachment is obsolete: true
Attachment #8712079 - Attachment is obsolete: true
Attachment #8712081 - Attachment is obsolete: true
Attachment #8712079 - Flags: review?(masayuki)
Attachment #8712081 - Flags: review?(masayuki)
Attachment #8715155 - Flags: review?(masayuki)
Attachment #8715158 - Flags: review?(masayuki)
Comment on attachment 8715158 [details] [diff] [review]
part3 - Interface between PresShell and HardwareKeyHandler

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #211)
> Comment on attachment 8712078 [details] [diff] [review]
> part4 - Interface between PresShell and HardwareKeyHandler
>
> > void
> > PresShell::HandleKeyboardEvent(nsINode* aTarget,
> >                                WidgetKeyboardEvent& aEvent,
> >                                bool aEmbeddedCancelled,
> >                                nsEventStatus* aStatus,
> >                                EventDispatchingCallback* aEventCB)
> > {
> >-  if (aEvent.mMessage == eKeyPress ||
> >-      !BeforeAfterKeyboardEventEnabled()) {
> >+  MOZ_ASSERT(aTarget);
> >+
> >+  if (aEvent.mIsSynthesizedByTIP) {
>
> How about if the event was synthesized for tests?
>
I am trying to find other or create one API that doesn't use TIP to simulate the hardware key input to avoid this.
(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #209)
> Comment on attachment 8712076 [details] [diff] [review]
> part3 - HardwareKeyHandler component
> 
> >+++ b/dom/inputmethod/HardwareKeyHandler.h
> 
> >+class HardwareKeyHandler : public nsIHardwareKeyHandler
> >+{
> >+public:
> >+  HardwareKeyHandler();
> >+
> >+  NS_DECL_ISUPPORTS
> >+  NS_DECL_NSIHARDWAREKEYHANDLER
> 
> I wonder, don't you need to add this class to cycle collector? Because this
> class may be held by JS.
I think nsIHardwareKeyHandler won't be part of reference cycle in current implementation.
However, I add this class into cycle collection just in case.

> Comment on attachment 8712076 [details] [diff] [review]
> part3 - HardwareKeyHandler component
> 
> >+++ b/dom/inputmethod/HardwareKeyHandler.cpp
> 
> >+bool
> >+HardwareKeyHandler::DispatchToTargetApp(nsINode* aTarget,
> >+                                        WidgetKeyboardEvent* aEvent,
> >+                                        nsEventStatus* aStatus)
> >+{
> >+  MOZ_ASSERT(aTarget, "No target provided");
> >+  MOZ_ASSERT(aEvent, "No event provided");
> >+
> >+  // Get current focus content as the event target
> >+  nsCOMPtr<nsIContent> currentFocusTarget = GetFocusContent();
> >+  if(!currentFocusTarget) {
> >+    return false;
> >+  }
> >+
> >+  nsIContent* originalTarget = aTarget->AsContent();
> >+
> >+  // Check the event target is same as the current focus element or not.
> >+  // The event target will change if focus is changed.
> >+  // If the event target is changed, then what should we do?
> >+  if (originalTarget != currentFocusTarget) {
> >+    NS_WARNING("The current focus element is different from the origial event target.");
> >+  }
> 
> We should fire on new focused element. That's valid behavior for UI Events
> spec.
Hmm...I still think this behavior is a little bit weird.
This implies that the event target can be changed during the event is dispatching.
I agreed with you that the event target should be the focused element,
but shouldn't it be decided at the moment that user press the physical keyboard?
I personally think it's weird that the event target will be set again to the new focused element
before user really do some actions to trigger the events(e.g., keypress, click, ..)?
Attachment #8715178 - Flags: review?(masayuki)
I expose the dictionary for keyboard event initialization to Keyboard.jsm and pass it to MozKeyboard.js to create a new keyboard event.
Attachment #8715179 - Flags: review?(masayuki)
Attachment #8715179 - Flags: review?(bugs)
Expose defaultPreventedBy* to MozKeyboard.js and pass its results to nsIHardwareKeyHandler
Attachment #8715181 - Flags: review?(masayuki)
Attachment #8715181 - Flags: review?(bugs)
Attachment #8715181 - Attachment description: Expose DefaultPreventedBy* → part6 - Expose DefaultPreventedBy*
Attachment #8715182 - Flags: review?(timdream)
Attachment #8715182 - Flags: review?(masayuki)
Comment on attachment 8715179 [details] [diff] [review]
part5 - Expose KeyboardEventInit dictionary

mEvent shouldn't be null ever.

I wonder how to keep KeyboardEvent::GetInitDict updated if/when new stuff is added to KeyboardEventInit or any dictionaries it inherits.
I don't really have good ideas for that.
Attachment #8715179 - Flags: review?(bugs) → review+
Comment on attachment 8715181 [details] [diff] [review]
part6 - Expose DefaultPreventedBy*

Interesting. I can see this being useful for UI code in FF.
Attachment #8715181 - Flags: review?(bugs) → review+
Comment on attachment 8715157 [details] [diff] [review]
part2 - Prevent PostHandleKeyboardEvent if it's handling by IME

The scope change is necessary for part4. So, it should've been in it. But it's okay.
Attachment #8715157 - Flags: review?(masayuki) → review+
Comment on attachment 8715179 [details] [diff] [review]
part5 - Expose KeyboardEventInit dictionary

>+  // legacy attributes
>+  aParam.mKeyCode = internalEvent->keyCode;
>+  aParam.mCharCode = internalEvent->charCode;

I think that these are not exactly correct. However, they must work well in most cases.

You should use KeyCode() and CharCode() for initializing them if you do not change the event type. Otherwise, you need to adjust the values before creating new event with different event type.

>+  aParam.mWhich = mInitializedWhichValue;

This is incorrect. mInitializedWhichValue is valid only when the event is created by JS. You should use Which().


If you don't change the event type, please using the methods, then, r=masayuki.  Otherwise, please ask review me again.

>diff --git a/dom/webidl/KeyboardEvent.webidl b/dom/webidl/KeyboardEvent.webidl
>+
>+  [Cached, ChromeOnly, Constant]
>+  readonly attribute KeyboardEventInit initDict;

Could you add comment before here for explaining above complicated issue. This should be used only for creating event with same event type.
Attachment #8715179 - Flags: review?(masayuki) → review+
Comment on attachment 8715158 [details] [diff] [review]
part3 - Interface between PresShell and HardwareKeyHandler

>-// f17842ee-f1f0-4193-814f-70d706b67060
>+// 3a0de342-93f6-4a40-9c77-574662bb2a43
> #define NS_IPRESSHELL_IID \
>-{ 0xf17842ee, 0xf1f0, 0x4193, \
>-  { 0x81, 0x4f, 0x70, 0xd7, 0x06, 0xb6, 0x70, 0x60 } }
>+{ 0x3a0de342, 0x93f6, 0x4a40, \
>+  { 0x9c, 0x77, 0x57, 0x46, 0x62, 0xbb, 0x2a, 0x43 } }

FYI: Now, we don't need to modify uuid only in mozilla-central.
https://groups.google.com/forum/#!topic/mozilla.dev.platform/HE1_qZhPj1I

> void
> PresShell::HandleKeyboardEvent(nsINode* aTarget,
>                                WidgetKeyboardEvent& aEvent,
>                                bool aEmbeddedCancelled,
>                                nsEventStatus* aStatus,
>                                EventDispatchingCallback* aEventCB)
> {
>+  MOZ_ASSERT(aTarget);
>+
>+  // return true if the real event target is in its child process
>+  bool targetIsIframe = IsTargetIframe(aTarget);
>+
>   if (aEvent.mMessage == eKeyPress ||
>-      !BeforeAfterKeyboardEventEnabled()) {
>+      !BeforeAfterKeyboardEventEnabled() ||
>+      aEvent.mIsSynthesizedByTIP) {
>+#ifdef MOZ_B2G
>+    FinishDispatchingKeyboardEvent(targetIsIframe, aTarget, aEvent,
>+                                   aStatus, aEventCB);
>+#else
>     EventDispatcher::Dispatch(aTarget, mPresContext,
>                               &aEvent, nullptr, aStatus, aEventCB);
>+#endif // MOZ_B2G

I don't like to insert a lot of #ifdef MOZ_B2G into normal path. I think that FinishDispatchingKeyboardEvent should have both implementation. I.e., if |#ifndef MOZ_B2G|, it just calls EventDispatcher::Dispatch().

>-  // Dispatch actual key event to event target.
>+#ifdef MOZ_B2G
>+  if (FinishDispatchingKeyboardEvent(targetIsIframe, aTarget, aEvent,
>+                                     aStatus, aEventCB)) {
>+    return;
>+  }
>+#else
>   EventDispatcher::Dispatch(aTarget, mPresContext,
>                             &aEvent, nullptr, aStatus, aEventCB);
>+#endif // MOZ_B2G

Same here.
Attachment #8715158 - Flags: review?(masayuki) → review-
(In reply to Olli Pettay [:smaug] from comment #223)
> Comment on attachment 8715179 [details] [diff] [review]
> part5 - Expose KeyboardEventInit dictionary
> 
> mEvent shouldn't be null ever.
> 
> I wonder how to keep KeyboardEvent::GetInitDict updated if/when new stuff is
> added to KeyboardEventInit or any dictionaries it inherits.
> I don't really have good ideas for that.

Anyway, I think that the patch should add warning about this to the definition of KeyboardEventInit.
(In reply to Chun-Min Chang[:chunmin] from comment #219)
> > >+bool
> > >+HardwareKeyHandler::DispatchToTargetApp(nsINode* aTarget,
> > >+                                        WidgetKeyboardEvent* aEvent,
> > >+                                        nsEventStatus* aStatus)
> > >+{
> > >+  MOZ_ASSERT(aTarget, "No target provided");
> > >+  MOZ_ASSERT(aEvent, "No event provided");
> > >+
> > >+  // Get current focus content as the event target
> > >+  nsCOMPtr<nsIContent> currentFocusTarget = GetFocusContent();
> > >+  if(!currentFocusTarget) {
> > >+    return false;
> > >+  }
> > >+
> > >+  nsIContent* originalTarget = aTarget->AsContent();
> > >+
> > >+  // Check the event target is same as the current focus element or not.
> > >+  // The event target will change if focus is changed.
> > >+  // If the event target is changed, then what should we do?
> > >+  if (originalTarget != currentFocusTarget) {
> > >+    NS_WARNING("The current focus element is different from the origial event target.");
> > >+  }
> > 
> > We should fire on new focused element. That's valid behavior for UI Events
> > spec.
> Hmm...I still think this behavior is a little bit weird.
> This implies that the event target can be changed during the event is
> dispatching.

Yes.

> I agreed with you that the event target should be the focused element,
> but shouldn't it be decided at the moment that user press the physical
> keyboard?

I don't think so. For example, data:text/html,<input onkeydown="event.target.blur();">, then, even if keydown is fired on the input element, keypress and keyup event will be fired on the root element (or <body>?).

If you worry about the security if a keyboard event is fired on *unexpected* application, we should stop dispatching keyboard event instead of firing on unfocused application.

> I personally think it's weird that the event target will be set again to the
> new focused element
> before user really do some actions to trigger the events(e.g., keypress,
> click, ..)?

But that's the current standard behavior...
Comment on attachment 8715178 [details] [diff] [review]
part4 - HardwareKeyHandler component

>+NS_IMETHODIMP
>+HardwareKeyHandler::ForwardKeyToInputMethodApp(nsINode* aTarget,
>+                                               WidgetKeyboardEvent* aEvent,
>+                                               nsEventStatus* aEventStatus,
>+                                               bool* aResult)

Oh... Sorry, I probably misled you. Actually, I said you should use [noscript] for this method definition.

But you can define C++ method directly in xpidl. For example, see here:
http://mxr.mozilla.org/mozilla-central/source/dom/interfaces/events/nsIDOMEventTarget.idl#107

Then, you can specify the result type as bool. Then, you can make implementation of this method and the users simpler. Sorry for my mistake. I should've realized that.

>+{
>+  MOZ_ASSERT(aTarget, "No target provided");
>+  MOZ_ASSERT(aEvent, "No event provided");
>+
>+  // No need to forward hardware key event to IME if:
>+  // 0. key's defaultPrevented is true;
>+  // 1. IME is disabled
>+  // 2. There is no nsIHardwareKeyEventListener in use
>+  // 3. This key event is generated by IME(from nsITextInputProcessor)
>+  // 4. The key event is already handling
>+  if (aEvent->mFlags.mDefaultPrevented ||
>+      !mInputMethodAppConnected ||
>+      !mHardwareKeyEventListener ||
>+      aEvent->mIsSynthesizedByTIP ||
>+      aEvent->mInputMethodAppState != WidgetKeyboardEvent::eNotHandled) {
>+    *aResult = false;
>+    return NS_OK;
>+  }

Then, you can make this simpler, even though the number of lines are increased:

// No need to forward hardware key event to IME if key's defaultPrevented is true.
if (aEvent->mFlags.mDefaultPrevented) {
  return false;
}
// No need to forward hardware key event to IME if IME is disabled.
if (!mInputMethodAppConnected) {
  return false;
}
...

This style makes maintaining the if state and reading the condition easier.

>+    // Create a Keyboard event for nsIHardwareKeyEventListener.onHardwareKey
>+    nsCOMPtr<EventTarget> evtTarget = do_QueryInterface(aTarget);

nit: eventTarget is better.

>+    nsPresContext* presCtx = GetPresContext(aTarget);

nit: presContext is better.

>+    KeyboardEvent* keyEvt = new KeyboardEvent(evtTarget, presCtx, aEvent);

nit: keyboardEvent is better.

>+  nsEventStatus status = aEventStatus? *aEventStatus : nsEventStatus_eIgnore;

nit: insert a whitespace before |?|.

>+  KeyboardInfo* copiedInfo =
>+    new KeyboardInfo(new WidgetKeyboardEvent(*aEvent), aTarget, &status);

You create an instance of WidgetKeyboardEvent here. But looks like that nobody (including KeyboardInfo) doesn't destroy it.

>+  size_t sizeBefore = mEventQueue.GetSize();
>+  mEventQueue.Push(copiedInfo);
>+  NS_ENSURE_TRUE(mEventQueue.GetSize() == sizeBefore + 1,
>+                 NS_ERROR_OUT_OF_MEMORY);

Could you not use NS_ENSURE_* anymore? And returning false or calling MOZ_CRASH() may be better.

>+    // Dispatch the keypress to its original event target if:

So, we need agreement about this...

I'm thinking that if target app is changed, we should stop dispatching remaining keyboard events for preventing unexpected operation in the new app.

Otherwise, i.e., focus is changed in the app, we should fine keyboard event in new focused element for conforming to the spec.

>+void
>+HardwareKeyHandler::PostHandleKeyboardEvent(nsINode* aTarget,
>+                                            WidgetKeyboardEvent* aEvent,
>+                                            nsEventStatus* aStatus)
>+{
>+  MOZ_ASSERT(aTarget, "No target provided");
>+  MOZ_ASSERT(aEvent, "No event provided");
>+
>+  nsPresContext* presCtx = GetPresContext(aTarget);

nit: presContext is better.