1110030 - Routing hardware key events to keyboard app when an input field is focusing.

Status: RESOLVED FIXED

(Core :: DOM: Events, defect)

Description

[Evelyn Hung]

Proposal:

    1. Let Keyboard app specify its key set for a layout in app's manifest
    2. System app uses mozContentEvent to notify Gecko when a keyboard layout is ready, as well as its handling key set. 
    3. Once step#2 is done, Gecko bypass those key strokes to Keyboard app directly.  (foreground app won't get them)

Comment 1

[Tim Guan-tin Chien [:timdream] (please needinfo)]

This should be an Gaia System:input mgmt + Gecko InputMethod API work.

We need to define how (3) works though, since the input app frame by default never takes focus, it doesn't make sense for it to receive normal DOM KeyboardEvent.

Comment 2

[Kan-Ru Chen [:kanru] (UTC+9)]

How about dispatch the KeyboardEvent to navigator.mozInputMethod or navigator.mozInputMethod.inputcontext?

Comment 3

[Marco Chen [:mchen]]

(In reply to Kan-Ru Chen [:kanru] from comment #2)
> How about dispatch the KeyboardEvent to navigator.mozInputMethod or
> navigator.mozInputMethod.inputcontext?

So keyboard app will not receive KeyboardEvent from EventListener but from that dedicated API.
This avoid that "non-focused window should not receives DOM KeyboardEvent".

oh...good point.

Comment 4

[Jayachandra Yakasiri]

what about adding a dedicated interface 'inputevent' in InputMethod with keyCode/charCode/Timestamp as attributes.
These attributes will be exposed to keyboard app via navigator and a listener required to listen to 'inputevent'

Comment 5

[Jayachandra Yakasiri]

by listening to 'inputeventchange' we can take keyCode/charCode/Timestamp attributes and we can convert those based on IME loaded, then we can convert the charCode/keyCode as per IME and timestamp[here timestamp is required for multitap]

Comment 6

[Jayachandra Yakasiri]

after converting the keyCode/charCode we can send them back to inputMethod to dispatch it to Editoreventlistener  via domWindowUtils.

Comment 7

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Marco Chen [:mchen] (PTO 12/10 ~ 12/12) from comment #3)
> So keyboard app will not receive KeyboardEvent from EventListener but from
> that dedicated API.
> This avoid that "non-focused window should not receives DOM KeyboardEvent".
> 
> oh...good point.

Would it violate the spec/convention etc if we reuse the KeyboardEvent instance for our API? If not I am totally for it.

By the way, we need to properly vet the API to ensure hardware behavior can be properly reflected on the target app. For example, |sendKey()| currently sends all three key events at once. We would need to change that to make sure keyboard app is enabled to relay the hw keyboard as close as possible.

Comment 8

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TPE) (please ni?) from comment #7)
> (In reply to Marco Chen [:mchen] (PTO 12/10 ~ 12/12) from comment #3)
> > So keyboard app will not receive KeyboardEvent from EventListener but from
> > that dedicated API.
> > This avoid that "non-focused window should not receives DOM KeyboardEvent".
> > 
> > oh...good point.
> 
> Would it violate the spec/convention etc if we reuse the KeyboardEvent
> instance for our API? If not I am totally for it.
> 
> By the way, we need to properly vet the API to ensure hardware behavior can
> be properly reflected on the target app. For example, |sendKey()| currently
> sends all three key events at once. We would need to change that to make
> sure keyboard app is enabled to relay the hw keyboard as close as possible.

The repeat use case was covered, my mistake:
http://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/forms.js#523-526

The other detail to settle is probably only like key properties etc. But let's not worry about this given the we won't need this too much with the new proposal.

Comment 9

[Evelyn Hung]

(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TPE) (please ni?) from comment #8) 
> The other detail to settle is probably only like key properties etc. But
> let's not worry about this given the we won't need this too much with the
> new proposal.

and Luke will update the new proposal here, ni him as a note. Simply speaking it's just chaining keyboard app into the event routing path of the current KeyboardEvent. [1] 
Active keyboard app will get events before the target element.

[1] https://wiki.mozilla.org/WebAPI/BrowserAPI/KeyboardEvent#Dispatch_KeyboardEvent_across_BrowserElements

Comment 10

[Luke Chang [:lchang] (inactive)]

The new proposal is updated on Wiki [1]. Please refer to it for further discussion.

[1] https://wiki.mozilla.org/WebAPI/InputMethod_API_with_hardware_keyboard

Comment 11

[Luke Chang [:lchang] (inactive)]

(In reply to Luke Chang [:lchang] from comment #10)
> The new proposal is updated on Wiki [1]. Please refer to it for further
> discussion.
> 
> [1] https://wiki.mozilla.org/WebAPI/InputMethod_API_with_hardware_keyboard

Hi XuLei,

This proposal introduces some APIs on mozInputMethod. It would be nice if you give us some feedback. Thanks a lot.

Comment 12

[Yuan Xulei [:yxl] (Leave from May 18)]

Looks good to me.

Just a comment about registering/unregistering a listener to handle the key events in keyboard app:

    mozInputMethod.inputcontext.addEventListener('keydown', function handler() {});
    mozInputMethod.inputcontext.removeEventListener('keydown', handler); 

It appears like a general key event handler to catch both software and hardware key events. I suggest we change the API a bit to tell that it only catches the hardware key events. It may be something like that:

    mozInputMethod.inputcontext.hardwareinput.addEventListener('keydown', function handler() {});
    mozInputMethod.inputcontext.hardwareinput.removeEventListener('keydown', handler);

Comment 13

[Luke Chang [:lchang] (inactive)]

To my knowledge, "keydown" and "keyup" usually stand for hardware key events so I'm not sure should we need to clarify it in particular.

Hi Tim, what do you think?

Comment 14

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Yuan Xulei [:yxl] from comment #12)
> It appears like a general key event handler to catch both software and
> hardware key events. I suggest we change the API a bit to tell that it only
> catches the hardware key events. It may be something like that:

Yeah. InputContext, which inherits EventTarget, only dispatches selectionchange and surroundingtextchange events currently. I don't think adding keydown and keyup event will confuse people. What do you mean?

Comment 15

[Yuan Xulei [:yxl] (Leave from May 18)]

The usage of `mozInputMethod.inputcontext.addEventListener('keydown'...)` implies it will listen to all key events regardless of whether a hardware a keyboard is attached or whether the events are generated by mozInputMethod.inputcontext.sendKey(...).

When we don't have a physical keyboard and use software keyboard only, should `mozInputMethod.inputcontext.addEventListener('keydown'...)` catch the `keydown` events from sendKey()?

If so, just forget about all my comments :-)
If not, I suggest change the API to make it clear that virtual key events won't be listened.

Comment 16

[Jayachandra Yakasiri]

(In reply to Yuan Xulei [:yxl] from comment #15)
> The usage of `mozInputMethod.inputcontext.addEventListener('keydown'...)`
> implies it will listen to all key events regardless of whether a hardware a
> keyboard is attached or whether the events are generated by
> mozInputMethod.inputcontext.sendKey(...).
> 
> When we don't have a physical keyboard and use software keyboard only,
> should `mozInputMethod.inputcontext.addEventListener('keydown'...)` catch
> the `keydown` events from sendKey()?
> 
> If so, just forget about all my comments :-)
> If not, I suggest change the API to make it clear that virtual key events
> won't be listened.

Hi Mr.Yuan Xulei,

For the question,
> should `mozInputMethod.inputcontext.addEventListener('keydown'...)` catch
> the `keydown` events from sendKey()?

Yes i think it will listen soft key also when sendKey()is called, which will lead to never ending loop.
the solution for this is event Location comparison. for all soft key the event location is Zero[DOM_KEY_LOCATION_STANDARD] and for all hardkey the event location will be 4[DOM_KEY_LOCATION_MOBILE]
[https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent]
based on this we can separate which key is soft and which key is from hardware.So we can process the hardkey event only one time and send back the processed key using sendKey()

Comment 17

[John Lu [:mnjul][Please NI or I'll miss]]

(In reply to Luke Chang [:lchang] from comment #10)
> The new proposal is updated on Wiki [1]. Please refer to it for further
> discussion.
> 
> [1] https://wiki.mozilla.org/WebAPI/InputMethod_API_with_hardware_keyboard

I discussed with Luke about the proposal a bit and was wondering about this particular scenario:
1. mozInputMethod.inputcontext dispatches the hardware keyXXX event to KeyboardApp.
2. KeyboardApp decides to process the event, and calls evt.preventDefault in that event handler.
3. KeyboardApp sends the (processed) key to the "client app" by calling mIM.inputcontext.sendKey(), and gets a Promise.
4. "client app" rejects the key (maybe by its keydown event's preventDefault(), etc) and KeyboardApp's Promise gets rejected.

5. At this time, should we allow Gecko to be notified about such rejection? What's the possible use case, and is there anything which we might need to do, but which requires knowledge from Gecko and not just KBApp?
Note that currently it's not possible to notify Gecko because the synchronous event handler at steps 1 and 2 is already over.

Comment 18

[Jayachandra Yakasiri]

Instead of adding a listener to Keydown in mozInputMethod.inputcontext, why can't we use another new interface in mozInputMethod[ex:inputHardwareEvent] which sends keycode/charcode to gaia for further processing.
In this case we have to implement the interface and make a way via forms.js, Keyboard.jsm, MozKeyboard.js 
 
we will add a listener in forms.js for keypress, So we will be listening to only editor hardware keypress and sending them to gaia for further processing. 


Currently we have a channel to editor via mozInputMethod from gaia [sendKey() soft touch]
with this new approach we will be adding reverse channel for hardware keys which gives the event info to gaia and processes then sends back to mozInputMethod by existing path [sendKey()]

With this approach we will not disturb the existing behavior, we will just redirect the hardware events only when editor is focused. 
Mr.Tim, Mr.Yuan, what is your opinion on this?

Comment 19

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Yuan Xulei [:yxl] from comment #15)
> The usage of `mozInputMethod.inputcontext.addEventListener('keydown'...)`
> implies it will listen to all key events regardless of whether a hardware a
> keyboard is attached or whether the events are generated by
> mozInputMethod.inputcontext.sendKey(...).
> 
> When we don't have a physical keyboard and use software keyboard only,
> should `mozInputMethod.inputcontext.addEventListener('keydown'...)` catch
> the `keydown` events from sendKey()?
> 
> If so, just forget about all my comments :-)
> If not, I suggest change the API to make it clear that virtual key events
> won't be listened.

No, sendKey() has nothing directly related to keyEvent handling in the proposal. Calling sendKey() does not make inputcontext dispatch KeyboardEvent generated from hardware keyboard since the event is not generated from hardware keyboard.

Comment 20

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Jayachandra Yakasiri from comment #18)

I have no opinion on the implementation detail. Let's only worry about if the API fits all use cases for now.

Comment 21

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Jayachandra Yakasiri from comment #18)
> With this approach we will not disturb the existing behavior, we will just
> redirect the hardware events only when editor is focused. 

PS I think the current spec also "disturb" the current behavior when the IM API is active. IM API is not active (thus will not redirect keyboard events) when the keyboard is not shown.

Comment 22

[Yuan Xulei [:yxl] (Leave from May 18)]

(In reply to Jayachandra Yakasiri from comment #18)
> Instead of adding a listener to Keydown in mozInputMethod.inputcontext, why
> can't we use another new interface in mozInputMethod[ex:inputHardwareEvent]
> which sends keycode/charcode to gaia for further processing.
> In this case we have to implement the interface and make a way via forms.js,
> Keyboard.jsm, MozKeyboard.js 
>  
> we will add a listener in forms.js for keypress, So we will be listening to
> only editor hardware keypress and sending them to gaia for further
> processing. 
> 
> 
> Currently we have a channel to editor via mozInputMethod from gaia
> [sendKey() soft touch]
> with this new approach we will be adding reverse channel for hardware keys
> which gives the event info to gaia and processes then sends back to
> mozInputMethod by existing path [sendKey()]
It is another user case that might be considered in the future. Since the data exchange between the editor and gaia keyboard app is async. This reservse channel can only be used as a way to post process hardware key events, while in this bug, we need a way to preprocess the events.

Comment 23

[Evelyn Hung]

Luke, please update our offline discussion here to clarify the issue Xulei brought up. Thank you all.

Comment 24

[Luke Chang [:lchang] (inactive)]

(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TPE) (please ni?) from comment #19)
> (In reply to Yuan Xulei [:yxl] from comment #15)
> > When we don't have a physical keyboard and use software keyboard only,
> > should `mozInputMethod.inputcontext.addEventListener('keydown'...)` catch
> > the `keydown` events from sendKey()?
> >
> > If so, just forget about all my comments :-)
> > If not, I suggest change the API to make it clear that virtual key events
> > won't be listened.
>
> No, sendKey() has nothing directly related to keyEvent handling in the
> proposal. Calling sendKey() does not make inputcontext dispatch
> KeyboardEvent generated from hardware keyboard since the event is not
> generated from hardware keyboard.

If I understand correctly, the concern Xulei mentioned is about the rationality of the API's naming. Since the "mozInputMethod.inputcontext" is regarded as an agent of the event target and "sendKey()" intends to send a standard DOM KeyboardEvent, it may confuse users whether "mozInputMethod.inputcontext.addEventListener('keydown')" can receive a "keydown" event generated by "sendKey()".

For now, a key event generated by "sendKey()" doesn't follow the "mozbrowserbeforekeyXXX" proposal [1] and contains only partial properties in its event object (e.g. "event.key" is absent). That means these software key events have already been considered to be a non-standard KeyboardEvent. So I second Tim's opinion that we don't dispatch these software key events to "inputcontext" at this moment.

For long-term, however, I think we do need to define a "hardwareinput" object to deal with some issues such as supporting multiple hardware input devices, listening whether devices are connected or not, distinguishing the events' source, detecting the type of devices, and so on. We may change the API accordingly until the "hardwareinput" is well defined. Fortunately, the current API proposed in this bug will be certified only. The future changes should not take much effort. Any thoughts?

[1] https://wiki.mozilla.org/WebAPI/BrowserAPI/KeyboardEvent

Comment 25

[Jayachandra Yakasiri]

Attachment: This is proposal for Hardkey event[12 keys] routing to Keyboard app

Here i am attaching a document for Hardware key event rerouting to Keyboard app proposal.
Please review and let us know your comments.

Comment 26

[Marco Chen [:mchen]]

Comment on attachment 8589584 [details]
This is proposal for Hardkey event[12 keys] routing to Keyboard app

Hi,

Thanks for sharing this diagram I will check whether it is align with what we discussed in last face to face meeting.

But according to some business travel next week, I might need some days to do it.

Comment 27

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Comment on attachment 8589584 [details]
This is proposal for Hardkey event[12 keys] routing to Keyboard app

(for ppl with MS Word installed)

https://docs.google.com/viewer?url=https://bugzilla.mozilla.org/attachment.cgi?id=8589584

Comment 28

[Marco Chen [:mchen]]

(In reply to Jayachandra Yakasiri from comment #25)
> Created attachment 8589584 [details]
> This is proposal for Hardkey event[12 keys] routing to Keyboard app
> 
Hi,

Thanks for taking the effort to come out this proposal. Great!
And I have some high level questions for understanding your proposal first.

Q1: In the diagram, does the arrow from Keyboard app to content app mean 
  a. will Keyboard event be sent to content app by keyboard app?
  b. or it just mean the logic sequence of apps to receive Keyboard event not a real event flow? (because the real flow now for touch case would be keyboard app process -> b2g process -> content process)

  I even don't know the arrow mean the original keyboard event pressed by user or the new key generated by IME of keyboard app. 

Suggestion: would suggest you have two diagrams, one is for logical flow for sequence of apps to receive keys. And the other is the real event flow between processes. Or I will confuse the meaning of what you want to show.

Q2: It seems that you only include one possible direction in you steps - "keyboard app consumes the key". 

Suggestion: would suggest you show up what other possible flows you considered as well. 
  ex: 1. system app default-prevent this event then no key will be sent to keyboard app.
      2. keyboard app consumes this key then no key will be sent to content process but will generate a new key for content process.
      3. keyboard app doesn't consume this key so it should be sent to content process by b2g process.

Q3: I think in your proposal, there will be a new API in MozInputContext for web app to retrieve keys. right?

Q4: What is the event queue you mean in the "steps"? could you show us where is it in the code?

Comment 29

[Jayachandra Yakasiri]

(In reply to Marco Chen [:mchen] from comment #28)
> (In reply to Jayachandra Yakasiri from comment #25)
> > Created attachment 8589584 [details]
> > This is proposal for Hardkey event[12 keys] routing to Keyboard app
> > 
> Hi,
> 
> Thanks for taking the effort to come out this proposal. Great!
> And I have some high level questions for understanding your proposal first.
> 
> Q1: In the diagram, does the arrow from Keyboard app to content app mean 
>   a. will Keyboard event be sent to content app by keyboard app?
>   b. or it just mean the logic sequence of apps to receive Keyboard event
> not a real event flow? (because the real flow now for touch case would be
> keyboard app process -> b2g process -> content process)
> 
>   I even don't know the arrow mean the original keyboard event pressed by
> user or the new key generated by IME of keyboard app. 
> 
> Suggestion: would suggest you have two diagrams, one is for logical flow for
> sequence of apps to receive keys. And the other is the real event flow
> between processes. Or I will confuse the meaning of what you want to show.
> 
> Q2: It seems that you only include one possible direction in you steps -
> "keyboard app consumes the key". 
> 
> Suggestion: would suggest you show up what other possible flows you
> considered as well. 
>   ex: 1. system app default-prevent this event then no key will be sent to
> keyboard app.
>       2. keyboard app consumes this key then no key will be sent to content
> process but will generate a new key for content process.
>       3. keyboard app doesn't consume this key so it should be sent to
> content process by b2g process.
> 
> Q3: I think in your proposal, there will be a new API in MozInputContext for
> web app to retrieve keys. right?
> 
> Q4: What is the event queue you mean in the "steps"? could you show us where
> is it in the code?

Please find my inline comments for the questions below

Q1: In the diagram, does the arrow from Keyboard app to content app mean 
  a. will Keyboard event be sent to content app by keyboard app?
  b. or it just mean the logic sequence of apps to receive Keyboard event not a real event flow? (because the real flow now for touch case would be keyboard app process -> b2g process -> content process)
Ans:---------------------
 Here the arrow indicates just giving the text to content process via b2g process [normal flow by calling inputContext.sendKey()which sends the typed text via domWindowUtils].For touch case also it is same, i.e after finding the target element it just sends it to editor of corresponding content process via domWindowUtils

https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/forms.js#511
---------------------
  
I even don't know the arrow mean the original keyboard event pressed by user or the new key generated by IME of keyboard app. 

Ans:---------------------
Once the charcode/keycode conversion is done by IME, we use inputContext.sendKey() to send the converted charcode to Editor[normal way of sending] via domWindowUtils where it creates a new WidgetKeyboardEvent and dispatches the event

https://dxr.mozilla.org/mozilla-central/source/dom/base/nsDOMWindowUtils.cpp#1177
---------------------

Suggestion: would suggest you have two diagrams, one is for logical flow for sequence of apps to receive keys. And the other is the real event flow between processes. Or I will confuse the meaning of what you want to show.

Ans:---------------------
Ok, I will attach flow diagrams soon.
---------------------
Q2: It seems that you only include one possible direction in you steps - "keyboard app consumes the key". 

Suggestion: would suggest you show up what other possible flows you considered as well. 
  ex: 1. system app default-prevent this event then no key will be sent to keyboard app.
Ans:---------------------
 Since we are rerouting the events NS_KEY_BEFORE_DOWN and NS_KEY_BEFORE_UP to Keyboard app, this scenario will not exists.
---------------------
      2. keyboard app consumes this key then no key will be sent to content process but will generate a new key for content process.
Ans:---------------------
 Yes, Once Keyboard consumes the events[NS_KEY_BEFORE_DOWN and NS_KEY_BEFORE_UP]we will not send next events in the series to content process, we will send a new event only via domWindowUtils where it creates a new WidgetKeyboardEvent and dispatches the event to corresponding editor of content process.
---------------------
      3. keyboard app doesn't consume this key so it should be sent to content process by b2g process.

Ans:---------------------
I have a question here

Is there any scenario where the keyboard app is not consuming the event even though the editor is focused and the content process want to use it for different purpose? 


 If Yes, then if Keyboard app is not consuming the event then we have to send it back to content process via b2g process, but there are technical problems involved in this scenario

Problem 1: Since IPC involved in this, All the events are asynchronous, so we will not come to know whether the event [NS_KEY_BEFORE_DOWN]is consumed by keyboard app or not before it sends the other events in the series[NS_KEY_DOWN, NS_KEY_AFTER_DOWN, NS_KEY_BEFORE_UP, NS_KEY_UP, NS_KEY_AFTER_UP]. 

Problem 2: Since IPC involved and we are firing a custom event from DOM input context, calling event.preventdefault() in keyboard app will not prevents the event series.

Currently we are assuming that Keyboard app will consume the event. We implemented in such a way that the event will be given to 3rd party keyboard app also if it is active. So I could not find any scenario where a 3rd party app used the event for other purpose event though the editor is focused and keyboard is active. 
---------------------

Q3: I think in your proposal, there will be a new API in MozInputContext for web app to retrieve keys. right?

Ans:---------------------
 No, once we get the event in Tab child of Keyboard Process, we used domWindowUtils to dispatch the event
utils->SendKeyEventToKeyboard(event.message,event.keyCode,event.charCode,event.location,&ignored);
and we added an EventListener for mozbrowserbeforekeydown ,sending this event to Keyboard app as a part of MozInputContext              this._window.addEventListener("mozbrowserbeforekeydown", this);

---------------------
Q4: What is the event queue you mean in the "steps"? could you show us where is it in the code?

Ans:---------------------

This is a proposal to overcome the problems mentioned in point3 in Question2. (if there is a scenario where the keyboard app is not consuming the event even though the editor is focused and keyboard is active).

Then the EventQueue proposal is for queueing the events for making the events synchronous due to the below problem when the keyboard app does not consumes the event.

Problem 1: Since IPC involved in this, All the events are asynchronous, so we will not come to know whether the event [NS_KEY_BEFORE_DOWN]is consumed by keyboard app or not before it sends the other events in the series[NS_KEY_DOWN, NS_KEY_AFTER_DOWN, NS_KEY_BEFORE_UP, NS_KEY_UP, NS_KEY_AFTER_UP]. 
 
 

This event queue will be linked to EventStateManager and makes the events synchronous. But there will be a delay in getting the event by content process.
Is there any other better solution for the problem mentioned above?
---------------------

Comment 30

[C.M.Chang[:chunmin]]

(In reply to Jayachandra Yakasiri from comment #25)
> Created attachment 8589584 [details]
> This is proposal for Hardkey event[12 keys] routing to Keyboard app
> 
> Here i am attaching a document for Hardware key event rerouting to Keyboard
> app proposal.
> Please review and let us know your comments.

Hi, Jayachandra, 
I think your proposal has some differences with our proposal mentioned above[1]. In your diagram, it seems that the NS_KEY_BEFORE_DOWN and NS_KEY_BEFORE_UP will be dispatched to keyboard app. AFAIK, NS_KEY_BEFORE_DOWN/UP event should only be received by shell.html or system/index.html(The goal of this design is to enable b2g to handle the keyXXX event before or after dispatching keyXXX event to target app, i.e., some certain keys[Power, VolumeUp, ...] should be handled by b2g only). In our proposal[1], we will dispatch Keydown or Keyup event to keyboard app directly if it's active.


[1] https://wiki.mozilla.org/WebAPI/InputMethod_API_with_hardware_keyboard#Flowchart

Comment 31

[Marco Chen [:mchen]]

Comment on attachment 8589584 [details]
This is proposal for Hardkey event[12 keys] routing to Keyboard app

(In reply to Chun-Min Chang[:chunmin] from comment #30)
> AFAIK,
> NS_KEY_BEFORE_DOWN/UP event should only be received by shell.html or
> system/index.html(The goal of this design is to enable b2g to handle the
> keyXXX event before or after dispatching keyXXX event to target app

More in general, only mozBrowser-embedder (and the target is a moz iframe embedded by it) can receive NS_KEY_BEFORE_DOWN and others. Please refer to [1].

In this case the keyboard app is not a mozBrowser-embedder and the target is not it's mozBrowser-embedded i-frame.

[1] https://wiki.mozilla.org/WebAPI/BrowserAPI/KeyboardEvent#Dispatch_KeyboardEvent_across_BrowserElements

Comment 32

[Jayachandra Yakasiri]

> I think your proposal has some differences with our proposal mentioned
> above[1]. In your diagram, it seems that the NS_KEY_BEFORE_DOWN and
> NS_KEY_BEFORE_UP will be dispatched to keyboard app. AFAIK,
> NS_KEY_BEFORE_DOWN/UP event should only be received by shell.html or
> system/index.html(The goal of this design is to enable b2g to handle the
> keyXXX event before or after dispatching keyXXX event to target app, i.e.,
> some certain keys[Power, VolumeUp, ...] should be handled by b2g only). In
> our proposal[1], we will dispatch Keydown or Keyup event to keyboard app
> directly if it's active.

As Mr.Marco mentioned, in this case both target and keyboard iframes are embedded iframes with respect to system app.
In this case there is no use of giving NS_KEY_BEFORE_DOWN/UP events of input keys to embedder iframe[system]? In Power, VolumeUp case system may require those events for fixing the behaviour before it consumed by target but do we really require this behaviour for input keys?

Comment 33

[Marco Chen [:mchen]]

(In reply to Jayachandra Yakasiri from comment #32)
> As Mr.Marco mentioned, in this case both target and keyboard iframes are
> embedded iframes with respect to system app.
> In this case there is no use of giving NS_KEY_BEFORE_DOWN/UP events of input
> keys to embedder iframe[system]? 

1. What I mentioned is "NS_KEY_BEFORE_DOWN/UP should be sent to EMBEDDER ONLY not embedded iframe". In your case, both of keyboard moz-iframe and target app moz-iframe should not receive these keys.

> In Power, VolumeUp case system may require those events for fixing the behaviour before it consumed > by target but do we really require this behaviour for input keys?

2. This mechanism is provided for all of keyboard events and doesn't have different behaviors for different kind of keyboard events.

Comment 34

[C.M.Chang[:chunmin]]

(In reply to Jayachandra Yakasiri from comment #32)
> As Mr.Marco mentioned, in this case both target and keyboard iframes are
> embedded iframes with respect to system app.
+--------------------------------+
| shell.html                     |
|                                |
|  +---------------------------+ |
|  | system/index.html         | |
|  |                           | |
|  | +----------+   +--------+ | |
|  | | keyboard |   | target | | |
|  | | app      |   | app    | | |
|  | +----------+   +--------+ | |
|  |                           | |
|  +---------------------------+ |
|                                |
+--------------------------------+

Yes, so both keyboard app and target app should not receive NS_KEY_BEFORE_DOWN or NS_KEY_BEFORE_UP events.

> In this case there is no use of giving NS_KEY_BEFORE_DOWN/UP events of input
> keys to embedder iframe[system]? In Power, VolumeUp case system may require
> those events for fixing the behaviour before it consumed by target but do we
> really require this behaviour for input keys?
I may not explain it well. Here[1] gives a good example to show why we need NS_KEY_BEFORE_DOWN/UP events.

As the mention from comment #33:
> This mechanism is provided for all of keyboard events and doesn't have
> different behaviors for different kind of keyboard events.
This is not designed to fix some special cases, but a general behaviors for all keyboard events. Please refer the bug 989198[2]

[1] https://groups.google.com/forum/#!topic/mozilla.dev.webapi/xPGXAHEiVmg
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=989198

Comment 35

[Jayachandra Yakasiri]

Attachment: This is updated proposal for Hardkey event[12 keys] routing to Keyboard app

I updated the proposal as per the comments. Please review it.

Comment 36

[C.M.Chang[:chunmin]]

(In reply to Jayachandra Yakasiri from comment #35)
> Created attachment 8595735 [details]
> This is updated proposal for Hardkey event[12 keys] routing to Keyboard app
> 
> I updated the proposal as per the comments. Please review it.
It looks better! However, these are some details you should notice:
1. Actually, beforeKey*, key*, afterKey* are all dispatched by EventDispatcher::Dispatch. In attachment 8595735 [details], only key* events are dispatched by EventDispatcher::Dispatch.
2. Does key* events will be dispatched from keyboard app to target app directly? I think the keyboard app should ask b2g to dispatch the key* event to target app
3. It will be better if you add the case that keyboard app call key*.preventDefault(). In this case, the key* event should not be dispatched to target app.

Comment 37

[Jayachandra Yakasiri]

(In reply to Chun-Min Chang[:chunmin] from comment #36)

> 1. Actually, beforeKey*, key*, afterKey* are all dispatched by
> EventDispatcher::Dispatch. In attachment 8595735 [details], only key* events
> are dispatched by EventDispatcher::Dispatch.
Yes... All key* events are dispatched via Event Dispatcher, but to make the diagram simple i did not included it for before and after key events.
> 2. Does key* events will be dispatched from keyboard app to target app
> directly? I think the keyboard app should ask b2g to dispatch the key* event
> to target app
Yes... I mentioned it in the diagram with dotted lines and with a text, which says it is not a direct dispatch.
> 3. It will be better if you add the case that keyboard app call
> key*.preventDefault(). In this case, the key* event should not be dispatched
> to target app.
I did not understood your point[3].
When Keyboard app consumes the event, it will call event.Preventdefault(). and creates an another event to give it to target app editor[not directly but via b2g], this is same behavior for touch case also.[it calls sendkey()]
https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/forms.js#511

Comment 38

[C.M.Chang[:chunmin]]

(In reply to Jayachandra Yakasiri from comment #37)
> I did not understood your point[3].
> When Keyboard app consumes the event, it will call event.Preventdefault().
> and creates an another event to give it to target app editor[not directly
> but via b2g], this is same behavior for touch case also.[it calls sendkey()]
> https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/forms.js#511
If keyboard app call key* event.preventDefault(), it means that it will handle this event. Yes, the target app will still receive a event after keyboard handle it, but it's not the original event. We should avoid dispatch the original event to target app, otherwise the target app will receive two events. One is the event synthesized by keyboard app. Another is the original event. For example, if we choose Japanese as the input method in keyboard app and character 'A' will be converted to 'ち'. When keyboard app is active and we press down the character 'A' on hardware keyboard: 
1) Before_Keydown_event(A) will be dispatched to system app 
2) Keydown_event(A) will be dispatched to system app
3) Keydown_event(A) will be dispatched to keyboard app
  - 1) 'A' is converted to 'ち'
  - 2) 'ち' will be sent to target app via b2g
4) Keydown_event(A) will be dispatched to target app
  ---> This event should be cancelled because the key 'A' is already converted to 'ち'
  ---> This event will be received by target app if we don't prevent the event dispatching in the original key event chain.
5) After_Keydown_event(A) will be dispatched to system app

Comment 39

[Jayachandra Yakasiri]

> If keyboard app call key* event.preventDefault(), it means that it will
> handle this event. Yes, the target app will still receive a event after
> keyboard handle it, but it's not the original event. We should avoid
> dispatch the original event to target app, otherwise the target app will
> receive two events. One is the event synthesized by keyboard app. Another is
> the original event. For example, if we choose Japanese as the input method
> in keyboard app and character 'A' will be converted to 'ち'. When keyboard
> app is active and we press down the character 'A' on hardware keyboard: 
> 1) Before_Keydown_event(A) will be dispatched to system app 
> 2) Keydown_event(A) will be dispatched to system app
> 3) Keydown_event(A) will be dispatched to keyboard app
>   - 1) 'A' is converted to 'ち'
>   - 2) 'ち' will be sent to target app via b2g
> 4) Keydown_event(A) will be dispatched to target app
>   ---> This event should be cancelled because the key 'A' is already
> converted to 'ち'
>   ---> This event will be received by target app if we don't prevent the
> event dispatching in the original key event chain.
> 5) After_Keydown_event(A) will be dispatched to system app

As per the proposal attached, The NS_KEY_DOWN event will be given to Keyboard based on Editor is focused or not. [it means we are rerouting the NS_KEY_DOWN event to keyboard via IPC, so the system app or target app will not get this event]
So Keydown_event(A) will not be dispatched to system app or target app if editor is focused, but after keyboard converts it to 'ち' it will create a custom event and dispatches it to target app via system app. 
So as per the proposal only one event will be dispatched to target app.

Comment 40

[C.M.Chang[:chunmin]]

(In reply to Jayachandra Yakasiri from comment #39)
> So Keydown_event(A) will not be dispatched to system app or target app if
> editor is focused

1. Do you mean that system app or target app will not be added to event chain if keyboard app is active?
2. Is the following events order your expectation?

1) Before_Keydown_event(A) will be dispatched to system app 
2) Keydown_event(A) will be dispatched to keyboard app
   - 1) 'A' is converted to 'ち'
   - 2) 'ち' will be sent to target app via b2g
3) After_Keydown_event(A) will be dispatched to system app

Comment 41

[Jayachandra Yakasiri]

(In reply to Chun-Min Chang[:chunmin] from comment #40)
> 1. Do you mean that system app or target app will not be added to event
> chain if keyboard app is active?
> 2. Is the following events order your expectation?
> 
> 1) Before_Keydown_event(A) will be dispatched to system app 
> 2) Keydown_event(A) will be dispatched to keyboard app
>    - 1) 'A' is converted to 'ち'
>    - 2) 'ち' will be sent to target app via b2g
> 3) After_Keydown_event(A) will be dispatched to system app

Yes... This is the proposal mentioned in the attachment.

Comment 42

[C.M.Chang[:chunmin]]

(In reply to Jayachandra Yakasiri from comment #41)
> (In reply to Chun-Min Chang[:chunmin] from comment #40)
> > 1. Do you mean that system app or target app will not be added to event
> > chain if keyboard app is active?
> > 2. Is the following events order your expectation?
> > 
> > 1) Before_Keydown_event(A) will be dispatched to system app 
> > 2) Keydown_event(A) will be dispatched to keyboard app
> >    - 1) 'A' is converted to 'ち'
> >    - 2) 'ち' will be sent to target app via b2g
> > 3) After_Keydown_event(A) will be dispatched to system app
> 
> Yes... This is the proposal mentioned in the attachment.

stating two opinions:

1. In order to keep event dispatch consistent with the behaviour when the pref["dom.beforeAfterKeyboardEvent.enabled", true] is disabled, key events should be still dispatched in the parent process. That is, the system app should receive NS_KEY_DOWN event. Therefore, system app should be added into event chain.

1) Before_Keydown_event(A) will be dispatched to system app
2) Keydown_event(A) will be dispatched to system app
3) Keydown_event(A) will be dispatched to keyboard app if it's active
...
...
n) After_Keydown_event(A) will be dispatched to system app


2. If target app is not in event chain, how will the following case be handled?
  - 1) keyboard app is active, so the NS_KEY_DOWN event will be dispatched to it.
  - 2) keyboard app does not want to handle this NS_KEY_DOWN event, so it doesn't call preventDefault()
    --> The NS_KEY_DOWN should be dispatched to target app if keyboard app does not process it.
    --> However, the target app is not in event chain, 
    --> so NS_KEY_DOWN event can not be dispatched to target app

Although the NS_KEY_DOWN event can be dispatched to target app through the path that keyboard app use, but personally I think its weird to do that. If keyboard app doesn't want to process this key event, it should not forward this event. I prefer remaining the target app in event chain and determine whether we should dispatch the keyboard event to it by checking the calling of event.preventDefault()

Comment 43

[Jayachandra Yakasiri]

(In reply to Chun-Min Chang[:chunmin] from comment #42)

> stating two opinions:
> 
> 1. In order to keep event dispatch consistent with the behaviour when the
> pref["dom.beforeAfterKeyboardEvent.enabled", true] is disabled, key events
> should be still dispatched in the parent process. That is, the system app
> should receive NS_KEY_DOWN event. Therefore, system app should be added into
> event chain.
> 
> 1) Before_Keydown_event(A) will be dispatched to system app
> 2) Keydown_event(A) will be dispatched to system app
> 3) Keydown_event(A) will be dispatched to keyboard app if it's active
> ...
> ...
> n) After_Keydown_event(A) will be dispatched to system app
> 
'keydown' is intended for mozbrowser-embedded iframe not for mozbrowser-embedder iframe. So system app will not be present in Keydown_event event chain.  
https://wiki.mozilla.org/WebAPI/BrowserAPI/KeyboardEvent#Dispatch_KeyboardEvent_across_BrowserElements
> 
> 2. If target app is not in event chain, how will the following case be
> handled?
>   - 1) keyboard app is active, so the NS_KEY_DOWN event will be dispatched
> to it.
>   - 2) keyboard app does not want to handle this NS_KEY_DOWN event, so it
> doesn't call preventDefault()
>     --> The NS_KEY_DOWN should be dispatched to target app if keyboard app
> does not process it.
>     --> However, the target app is not in event chain, 
>     --> so NS_KEY_DOWN event can not be dispatched to target app
> 
> Although the NS_KEY_DOWN event can be dispatched to target app through the
> path that keyboard app use, but personally I think its weird to do that. If
> keyboard app doesn't want to process this key event, it should not forward
> this event. I prefer remaining the target app in event chain and determine
> whether we should dispatch the keyboard event to it by checking the calling
> of event.preventDefault()

May be we have to send the NS_KEY_DOWN event to target app if the keyboard app is not consuming[not calling prevent default] it, Is there any use case for this? 
Since IPC is involved in this what about synchronization of events in event chain? and what about delay between events?

Comment 44

[C.M.Chang[:chunmin]]

(In reply to Jayachandra Yakasiri from comment #43)
> 'keydown' is intended for mozbrowser-embedded iframe not for
> mozbrowser-embedder iframe. So system app will not be present in
> Keydown_event event chain.  
> https://wiki.mozilla.org/WebAPI/BrowserAPI/
> KeyboardEvent#Dispatch_KeyboardEvent_across_BrowserElements

Here is the test case of keyboard events: 
https://dxr.mozilla.org/mozilla-central/source/dom/events/test/test_dom_before_after_keyboard_event_remote.html#27
It specifies the expected keyboard events order and shows the parent process of an embedded iframe will still receive the NS_KEY_DOWN/NS_KEY_UP.

For understanding the complicated keyboard events, I organize the code flow into a slide a few days ago.
http://goo.gl/ccjdCt
Please feel free to use it and point out my errors. I hope it can be a helpful reference.

> May be we have to send the NS_KEY_DOWN event to target app if the keyboard
> app is not consuming[not calling prevent default] it, Is there any use case
> for this? 

Let's use the 'BrowserBack' key(http://goo.gl/R4iFgM) as the example use case. Suppose the target app want to use the 'BrowserBack' key for navigation no matter keyboard app is active or not(tt's the real behavior of facebook app). Assume the keyboard app is active in this case, NS_KEY_DOWN of 'BrowserBack' key will be dispatched to keyboard app first. Then, keyboard app doesn't want to handle 'BrowserBack' key, so it won't call event.preventDefault(). Next, the 'BrowserBack' key should be dispatched to target app for navigating the app's view.

In a broad sense, all the non-printable keys need to be dispatched to target app if we allow target app to handle them.

Does this use case make sense to you?

> Since IPC is involved in this what about synchronization of events in event chain? 

... Do you mean the events order may be wrong? 

Suppose keyboard app is active, and we send 'A', 'BrowserBack','B' in order:

case 1: if ['A', 'B'] and ['BrowserBack'] are dispatched by different path(domWindowUtils.sendKeyEvent in keyboard app and EventDispatcher::Dispatch in nsPresShell), target app might receive:
  a) 'A', 'BrowserBack', 'B'
  b) 'A', 'B', 'BrowserBack'

case 2: if ['A', 'B'] and ['BrowserBack'] are dispatched by same path(domWindowUtils.sendKeyEvent in keyboard app), the order will be kept constant(No matter IPC is sync or async, the delivery is "in-order": https://developer.mozilla.org/en-US/docs/IPDL/Tutorial#Message_Delivery_Order). That is, target app will receive: 'A', 'BrowserBack', 'B'.

However, I wonder if it's important to keep the keyboard events in order when keyboard app is active and the non-printable, printable keys are dispatched to target app through different path like case 1. 

> and what about delay between events?
Yes, performance is important issue. 

1. If some specified keys are forwarded to target app by keyboard app via b2g(domWindowUtils.sendKeyEvent in keyboard app), it needs two async ipc to dispatch them to target app: http://goo.gl/4e3s6j
One is https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/MozKeyboard.js#682
the other is https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/Keyboard.jsm#360 

2. If some specified keys are ignored by keyboard app, and they will be dispatched by the same path of NS_BEFORE_KEY_DOWN, it still needs two async ipc to dispatch them to target app:
One is the notification that keyboard app finish processing these keys(ignore them actually)
The other is https://dxr.mozilla.org/mozilla-central/source/dom/ipc/TabParent.cpp#1570

No matter which case will be accepted, I think the keyboard app can be tuned to achieve the expected performance because now it is certified only.

Comment 45

[C.M.Chang[:chunmin]]

I hope these slides can be helpful to illustrate my thoughts:
- Keyboard events order now: http://goo.gl/ccjdCt
- How does keyboard app forward keyboard events to target app via b2g: http://goo.gl/4e3s6j
- Expected events order after routing hardware key events to keyboard app: http://goo.gl/wt3HiB
- Proposal for this bug: http://goo.gl/D5yW6c

Comment 46

[Jayachandra Yakasiri]

Thanks for your comment. I understood it completely.

Yes both the cases has advantages and disadvantages, Which one do you suggest in the below cases

case 1: forwarding only printable key events to Keyboard app, and non printable key events in normal flow[example :  ['A', 'B'] are dispatched via domWindowUtils.sendKeyEvent in keyboard app and ['BrowserBack'] is dispached via EventDispatcher::Dispatch in nsPresShell)] 

case 2: forwarding all keyboard events to Keyboard app, let it decide which one it want to consume and which one it has to give it back to target app in normal flow[example: ['A', 'B'] and ['BrowserBack'] are dispatched via same path to keybaord app, ['A', 'B'] will be consumed by keyboard app and for['BrowserBack'] event keyboard will not be calling prevent default. so let it dispatch in normal way from preshell again to Target app via EventDispatcher::Dispatch ]

or any other better case than this?

Comment 47

[Marco Chen [:mchen]]

Cancel the needinfo/review flag because Chun-Min will be the one to do discussion first.

Comment 48

[C.M.Chang[:chunmin]]

(In reply to Jayachandra Yakasiri from comment #46)
Sorry for coming back to you quite late. 

After a brief discussion with Luke, we suggest taking case 2. The reason is that this approach can provide more flexibility to keyboard app. 

Possible use case: input-method switch
Now, in keyboard app, we have some small buttons "En", "12&", ...etc to switch languages. If we want to allow using hardware keyboard shortcut to switch languages layout on keyboard app like PC(Ctrl-Space[Windows]/Cmd-Space[Mac]/Super-Space[Ubuntu]), the non-printable key should be sent to keyboard app first.

Comment 49

[Tim Guan-tin Chien [:timdream] (please needinfo)]

This might not be related but I am working on bug 1137557 to route the internal call in forms.js to nsITextInputProcessor and away from nsIDOMWindowUtil.

Comment 50

[C.M.Chang[:chunmin]]

There is still one concern in this proposal(http://goo.gl/D5yW6c): It may cause infinite key events if target app is **in-process**. 
The keyboard event sent by forms.js in target app will call PresShell::HandleKeyboardEvent in main process again and lead a infinite key events.

in-process target app
===============================
process:   [chrome]         [chrome]              [content]        [chrome]
           nsPressShell --> EventStateManager --> keyboard app --> form.js in target app
             ^                                                        |
             |                                                        |
             +--------------------------------------------------------+

Comment 51

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #50)
One possible solution is to check the *state* of MozKeyboard.js in keyboard app.
If we can add a variable named 'state' in MozKeyboard.js and set it like "state = sending" when sendKey[1] is called, then PresShell can distinguish the keyboard event is fired from keyboard app or hardware by accessing the "state" in keyboard app's MozKeyboard.js. After the keyboard event triggered by keyboard app finish dispatching, then we can set "state = idle" when keyboard app receive the ack "keyboard:SendKey:Result:OK"[2]. However, this way will add two IPC overhead.

[1] https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/MozKeyboard.js#679
[2] https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/MozKeyboard.js#513

Comment 52

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #50, comment #51)
comment #16 already discuss this issue

Comment 53

[Olli Pettay [:smaug][[e-mail]]]

Why not add a state to the key event itself? (totally fine to add some [ChromeOnly] attribute to KeyboardEvent.webidl)

Comment 54

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #52)
> (In reply to Chun-Min Chang[:chunmin] from comment #50, comment #51)
> comment #16 already discuss this issue
The keyboardEvent.location[1] does not work as expected, the return value of keyboard.location from hardware VolumeUp/VolumeDown and Power is still 0 on my Flame. It need to find out the reason. On the other hand, as smaug mention, add a state to the keyboard event can provide a solid solution.
[1] https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/location

Comment 55

[Jayachandra Yakasiri]

(In reply to Olli Pettay [:smaug] from comment #53)
> Why not add a state to the key event itself? (totally fine to add some
> [ChromeOnly] attribute to KeyboardEvent.webidl)

Yes... I think this is the right way to handle it.

Comment 56

[Marco Chen [:mchen]]

Comment on attachment 8595735 [details]
This is updated proposal for Hardkey event[12 keys] routing to Keyboard app

To clear the review flag and Chun-Min is the one now.

Comment 57

[Jayachandra Yakasiri]

(In reply to Olli Pettay [:smaug] from comment #53)
> Why not add a state to the key event itself? (totally fine to add some
> [ChromeOnly] attribute to KeyboardEvent.webidl)

Instead of adding [ChromeOnly] attribute to KeyboardEvent.webidl, why can not we add it to WidgetKeyboardEvent it self, 
Anyway the sendkey in forms.js will ask nsDOMWindowUtils to reconstruct the event as WidgetKeyboardEvent and dispatches to widget. At this time we will set the [ChromeOnly] attribute true, which will be compared to avoid infinite looping.

Comment 58

[Olli Pettay [:smaug][[e-mail]]]

Well, you of course want to be able to initialize the [ChromeOnly] property somehow, so yes, similar thing is needed on the dictionary too.

Comment 59

[Jayachandra Yakasiri]

Attachment: Updated Proposal of Hardware Key Events Routing

Hi all,

Here i am attaching an updated proposal for Hardware key event routing, please review it and let us know your opinions.

Comment 60

[C.M.Chang[:chunmin]]

Comment on attachment 8625393 [details]
Updated Proposal of Hardware Key Events Routing

Thanks for updating the proposal. It's clearer now. 
I have two questions here, hope you can explain more about them.

1. Does it really need to add an event queue?
Honestly, I am not sure whether or not the events order should be kept when two keys are dispatched via two different path. The event queue is designed to keep the events order if one key 'a' will be used in inputmethod, but the following key 'ESC' won't be used in inputmethod and the keyboard app is inactive at this moment. In such case, I reckon there is no need to keep the events order. If user want to exit the app by clicking 'Esc', it's not important to make sure that the exit will be fired after the app receive 'a'.

2. Why don't we call KeyboardAppProxy in both two cases?
In case 2, ESM will decide whether or not it should dispatch the keyboard event to target app based on the response of keyboard.jsm. If keyboard app doesn't consumes the keyboard event, then ESM will dispatch the keyboard event to target app. On the other hand, if keyboard app consumes the keyboard event, then ESM has no need to dispatch the event to target app. Is this case different from the case 1? If they're same, why don't we call KeyboardAppProxy to handle the case "when keyboard app consumes the event"(your case 1) in ESM?

Comment 61

[Marco Chen [:mchen]]

Comment on attachment 8625393 [details]
Updated Proposal of Hardware Key Events Routing

Hi,

1. Does the event queue you mentioned mean the mEventQueue in nsAppShell [1]? Which is the queue to collect the physical key event from Event Hub.

2. A minor issue on case 2 - The final AfterKeyboardEvent is fired by Preshell not Tab Parent.

Personally I think it is good to go first then seeing the result.

[1] https://dxr.mozilla.org/mozilla-central/source/widget/gonk/nsAppShell.cpp#554

Comment 62

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #60)
> 2. Why don't we call KeyboardAppProxy in both two cases?
To Correct the question: Why don't KeyboardAppProxy response to ESM in case 1 and let ESM to call nsPresShell? Is it possible to use same interface(ESM<-->KeyboardAppProxy) in both cases?

Comment 63

[Jayachandra Yakasiri]

> 1. Does it really need to add an event queue?
> Honestly, I am not sure whether or not the events order should be kept when
> two keys are dispatched via two different path. The event queue is designed
> to keep the events order if one key 'a' will be used in inputmethod, but the
> following key 'ESC' won't be used in inputmethod and the keyboard app is
> inactive at this moment. In such case, I reckon there is no need to keep the
> events order. If user want to exit the app by clicking 'Esc', it's not
> important to make sure that the exit will be fired after the app receive 'a'.

Yes,the proposed events queue is for making the events in order, I think we have to see the problems caused by events order after implementation and then decide whether it is really required or not...

> 2. Why don't we call KeyboardAppProxy in both two cases?
> In case 2, ESM will decide whether or not it should dispatch the keyboard
> event to target app based on the response of keyboard.jsm. If keyboard app
> doesn't consumes the keyboard event, then ESM will dispatch the keyboard
> event to target app. On the other hand, if keyboard app consumes the
> keyboard event, then ESM has no need to dispatch the event to target app. Is
> this case different from the case 1? If they're same, why don't we call
> KeyboardAppProxy to handle the case "when keyboard app consumes the
> event"(your case 1) in ESM?

Ok, let ESM dispatch both the crossprocess event and afterkey event[afterkey event via preshell]based on keyboardAppProxy decision. I agree with you.

Comment 64

[Jayachandra Yakasiri]

> 1. Does the event queue you mentioned mean the mEventQueue in nsAppShell
> [1]? Which is the queue to collect the physical key event from Event Hub.
No, its for making the events in order, bcz keyboard consumes few events others will be dispatched to target app if keyboard app is not prevented them. so there will be a timing problem.
The same issue is mentioned in comment 44
> 2. A minor issue on case 2 - The final AfterKeyboardEvent is fired by
> Preshell not Tab Parent.
Yes, it is via presShell, but initiated by Tabparent
https://dxr.mozilla.org/mozilla-central/source/dom/ipc/TabParent.cpp#2204
> Personally I think it is good to go first then seeing the result.
Yes, I agree.

Comment 65

[Marco Chen [:mchen]]

(In reply to Jayachandra Yakasiri from comment #64)
> No, its for making the events in order, bcz keyboard consumes few events
> others will be dispatched to target app if keyboard app is not prevented
> them. so there will be a timing problem.
> The same issue is mentioned in comment 44

It seems that the conclusion is to go option 2 then printable and non-printalbe keys will go to the same path. As a result what is the problem about order?

By the way, the order is important even they are dispatched as two different way.
If I understood it correctly, the [1] mentioned the sync event like keyboard event should be dispatched in order. 

From [1]
"Each event in this virtual queue MUST be delayed until the previous event has completed its propagation behavior, or been canceled."

[1] http://www.w3.org/TR/DOM-Level-3-Events/#sync-async

Comment 66

[C.M.Chang[:chunmin]]

Comment on attachment 8625393 [details]
After brief discussion with smaug, there might be another easier solution: Is it possible to dispatch the key* event to target app upon sending the key* event to system app? It can avoid modifying the EventStateManager, although I think it still need to modify nsPresShell.

Comment 67

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #66)
Further explain what I have mentioned: Maybe we can simply add the listener in system app to get the key* event. Once system app receives the key* event, it can forward this keyboard event to keyboard app. If keyboard app consumes the keyboard event, then system app should notify nsPresShell to fire the afterKey* event directly. On the other hand, if keyboard app doesn't consumes the keyboard event, then system app should notify nsPresShell to dispatch the keyboard event to target app and then fire the afterKey* event.

Comment 68

[Marco Chen [:mchen]]

(In reply to Chun-Min Chang[:chunmin] from comment #67)
This proposal will prevent keyboard app from being privileged app because currently there is no Web API for communication between app to app in privileged permission level.

But if this is for achieving the feature first then it is the way to go.
And there was other partner doing this similar approach already.

Comment 69

[C.M.Chang[:chunmin]]

(In reply to Marco Chen [:mchen] from comment #68)
The system app can communicate with keyboard app via MessageManager[1]. Thus, another solution mentioned in comment #66 can be achieved by hacking shell.js, which loads the Keyboard.jsm in [1]. The keyboard event can be intercepted in [2] and forward to keyboard app by calling Keyboard.sendToKeyboard('XXXX', { 'keyCode': evt.keyCode, 'charCode': evt.charCode, ..... });

[1] How does keyboard app dispatch keyboard event to web app now: http://goo.gl/4e3s6j
[2] https://dxr.mozilla.org/mozilla-central/source/b2g/chrome/content/shell.js#449

Comment 70

[Marco Chen [:mchen]]

Hi Chun-min,

Got your point and note that shell.js is not part of system app.
The former one is on the chrome window and the letter one is on the content window.

Comment 71

[C.M.Chang[:chunmin]]

(In reply to Marco Chen [:mchen] from comment #70)
Hi, Marco,
Thx for your reminder. Is it possible to use shell.js(and keyboardProxy xpcom) as a middle agent between system app and nsPresShell? If shell.js has a way to know when the key* events dispatch in system app is finished, then it still has a chance to achieve the new idea.

Comment 72

[Marco Chen [:mchen]]

(In reply to Chun-Min Chang[:chunmin] from comment #71)
> (In reply to Marco Chen [:mchen] from comment #70)
> Is it possible to use shell.js(and keyboardProxy
> xpcom) as a middle agent between system app and nsPresShell?

Yes, it is possible. Either you can invent a new Web API or leverage the mozChrome/ContentEvent to do communication between System app and shell.js (shell.html)

> If shell.js has a way to know when the key* events dispatch in system app is finished

According to system app is an i-frame embeded by shell.html, you can add a event listener for key on the capturing phase. Once the listener is called, the key traversal in the system app was done.

Comment 73

[C.M.Chang[:chunmin]]

(In reply to Marco Chen [:mchen] from comment #72)
From my test, the system/js/hardware_buttons.js[1] is able to catch the last keyboard event in system app. The last keyboard event is dispatched to system app's window in bubbling phase. After catching the last keyboard event, like you've said, we can send a mozContentEvent to shell.js. Then shell.js will know the keyboard event dispatch is finished in system app.

sample code
===============================
hardware_buttons.js
--------------------------
...
HardwareButtons.prototype.handleEvent = function hb_handleEvent(evt) {
  ...
  var cusEvt = new CustomEvent('mozContentEvent', { detail : { type: 'XXXX' } });
  window.dispatchEvent(cusEvt);
  ...
}
...


shell.js[2]
--------------------------
...
var CustomEventManager = {
  ...
  handleEvent: function custevt_handleEvent(evt) {
    ...
    switch(detail.type) {
      ...
      case 'XXXX':
        // Receive the notification that keyboard event dispatch is finished in system app!
        break;
    }
  }
}


[1] https://github.com/mozilla-b2g/gaia/blob/master/apps/system/js/hardware_buttons.js#L220
[2] https://dxr.mozilla.org/mozilla-central/source/b2g/chrome/content/shell.js#741

Comment 74

[Marco Chen [:mchen]]

According to event dispatch is asynchronous, there will be a unknown time gap between normal key traverse in chrome process and sending key to content process because you can't guarantee how many jobs in the queue before you dispatch the event. Basically you additionally add one more asynchronous procedures into this work. 

Maybe you can alternatively add event listener in the window of shell.html in the capturing phase and call any IPC to send key.

Comment 75

[C.M.Chang[:chunmin]]

(In reply to Marco Chen [:mchen] from comment #74)
If you add an event listener in the window of shell.html in the capturing phase, you will receive the key* event before dispatching to system app. At that time, if you forward the key* event to keyboard app in the event listener, then the order of key* events[1] may be wrong. We need to make sure that the key* event in keyboard app will be fired after key* events dispatch in system app is finished because keyboard app is an mozbrowser iframe embedded in system app.

Did I misunderstand what you were trying to say?
[1] https://goo.gl/JlWskF

Comment 76

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Comment on attachment 8625393 [details]
Updated Proposal of Hardware Key Events Routing

I have no strong opinion on internal Gecko impl, and look like others have already handling it.

Comment 77

[Marco Chen [:mchen]]

(In reply to Chun-Min Chang[:chunmin] from comment #75)
Right, right, right.

I argued about event order and I gave the wrong comment.
What I want to say would be on the bobbling phase of shell.html's document.

Comment 78

[C.M.Chang[:chunmin]]

(In reply to Marco Chen [:mchen] from comment #77)
> (In reply to Chun-Min Chang[:chunmin] from comment #75)
> Right, right, right.
> 
> I argued about event order and I gave the wrong comment.
> What I want to say would be on the bobbling phase of shell.html's document.
Yes, I think it would be easier to add an event listener at bubbling phase in shell.html to forward the keyboard event without hacking the EventStateManager.

Comment 79

[Youngwoo Jo]

(In reply to Chun-Min Chang[:chunmin] from comment #78)
> > I argued about event order and I gave the wrong comment.
> > What I want to say would be on the bobbling phase of shell.html's document.
> Yes, I think it would be easier to add an event listener at bubbling phase
> in shell.html to forward the keyboard event without hacking the
> EventStateManager.
Hi, all.
If shell.html forwards a keyboard event, how can we block the dispatch to target app in EventStateManager?
I think nsPresShell should check if a keyboard event was forwarded to keyboard app.
If then, nsPresShell does not call EventStateManager.PostHandleEvent() nor mozbrowserafterkey* event.
And then, when the response is received from Keyboard app, it decide whether the dispatch to target app continues or it fires a mozbrowserafterkey* event according to the status of preventDefault.
Right?

By the way, currently we have almost prepared the patch implementing the original architecture containing the modification in EventStateManager.
I expect the patch can be uploaded around the next week.
I hope to discuss these things based on the patch.

Comment 80

[C.M.Chang[:chunmin]]

(In reply to Youngwoo Jo from comment #79)
> Hi, all.
> If shell.html forwards a keyboard event, how can we block the dispatch to
> target app in EventStateManager?
> I think nsPresShell should check if a keyboard event was forwarded to
> keyboard app.
> If then, nsPresShell does not call EventStateManager.PostHandleEvent() nor
> mozbrowserafterkey* event.
> And then, when the response is received from Keyboard app, it decide whether
> the dispatch to target app continues or it fires a mozbrowserafterkey* event
> according to the status of preventDefault.
> Right?
> 
> By the way, currently we have almost prepared the patch implementing the
> original architecture containing the modification in EventStateManager.
> I expect the patch can be uploaded around the next week.
> I hope to discuss these things based on the patch.
It doesn't mean that we don't need to modify any parts of gecko code by the new approach discussed.
Yes, We still need to modify the nsPresShell to check whether or not a keyboard event should be forwarded to keyboard app(so it still need a keyboardProxy xpcom). The advantage of the new approach is to avoid modifying the complex EventStateManager.

Ok, it's good to hear that! If you already work on the patch that based on the original architecture, then we can discuss which approach is better then.

Comment 81

[Youngwoo Jo]

Attachment: IME_new_hardware_keyboard_event_routing_v0.1.patch

I uploaded the 1st patch.
And I hope this patch becomes the base of our discussion about architecture.

And this patch includes the modification in EventStateManager. (only one part)
However, I expect the modification in EventStateManager can be removed if the architecture is decided.

Comment 82

[Youngwoo Jo]

Attachment: IME_new_hardware_keyboard_event_routing_v0.1.patch

I'm sorry to make a wrong patch without KeyboardAppProxy.
I uploaded the new patch.

Comment 83

[Jayachandra Yakasiri]

In the attached patch i have not taken care of the issue mentioned in the below comment
https://bugzilla.mozilla.org/show_bug.cgi?id=1110030#c50
so the infinite loop problem still occurs when the editor is in b2g[i.e Inprocess Target app]

I propose a solution for this by adding a new flag in the structure "BaseEventFlags"
https://dxr.mozilla.org/mozilla-central/source/widget/BasicEvents.h#496
Initialize the flag in WidgetEvent constructor as false,
and then making the flag true when the event is recreated in nsDOMWindowUtils.cpp
This flag can be checked before sending the event to proxy in ESM[in the newly added method HandleKeyboardEvent] to stop the recursion.
Let me know your opinion on this

Comment 84

[C.M.Chang[:chunmin]]

Comment on attachment 8633917 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.1.patch

Review of attachment 8633917 [details] [diff] [review]:
-----------------------------------------------------------------

Overall looks good to me. 
There is two main issues needed to be solved:
1. Infinite loop to send keyboard events if the target app is in-process
2. Events order
   I agree with your earlier concern(comment #65): the events order is important. 
   The events fired from the same source(e.g. 12 keys) should be in order[1,2].

[1] http://www.w3.org/TR/DOM-Level-3-Events/#glossary-event-order
[2] http://www.w3.org/TR/DOM-Level-3-Events/#sync-async

::: dom/events/EventStateManager.cpp
@@ +1216,5 @@
> +  WidgetKeyboardEvent* keyboardEvent = aEvent->AsKeyboardEvent();
> +  if ((aEvent->mClass == eKeyboardEventClass) && (hasActiveIME) && (aEvent->message != NS_KEY_PRESS) && (keyboardEvent->keyCode != NS_EVENT_NULL)) {
> +    keyboardAppProxy->SetESMContext(this, *(aEvent->AsKeyboardEvent()));
> +    rv = keyboardAppProxy->SendEventToActiveKeyboard(aEvent->message, keyboardEvent->keyCode, keyboardEvent->charCode, keyboardEvent->time);
> +    if (rv == NS_OK) {

[Personal opinion] Is it possible to call keyboardAppProxy::SetESMContext inside keyboardAppProxy::SendEventToActiveKeyboard? it might looks cleanr

::: dom/events/KeyboardAppProxy.cpp
@@ +64,5 @@
> +  }
> +  else if (aType == NS_KEY_UP) {
> +    mKeyboardEventForwardCb->OnKeyboardEventReceived("keyup", aKeyCode, aCharCode, aTimeStamp);
> +  }
> +  else {

mozilla coding style:
if {
  ...
} else if {
  ...
}

@@ +90,5 @@
> +  uint32_t EventType;
> +  if (!strcmp(aEventType,"keydown")) {
> +    EventType = NS_KEY_DOWN;
> +  }
> +  else if (!strcmp(aEventType,"keyup")) {

Remember to change the coding style here too.

::: layout/base/nsPresShell.cpp
@@ +8276,5 @@
> +                keyboardAppProxy = do_GetService("@mozilla.org/keyboardAppProxy;1", &rv);
> +                if (!keyboardAppProxy) {
> +                   return NS_ERROR_FAILURE;
> +                }
> +                rv = keyboardAppProxy->SetEventTarget(eventTarget);

Could you explain why we need to set eventTarget here? Is the eventTarget different from WidgetEvent* aEvent->target in EventStateManager? If the answer is no, why not set the eventTarget in EventStateManager? Is it possible that the nsPresShell has no relationship with keyboardAppProxy?

Comment 85

[Youngwoo Jo]

(In reply to Chun-Min Chang[:chunmin] from comment #84)
> Comment on attachment 8633917 [details] [diff] [review]
> IME_new_hardware_keyboard_event_routing_v0.1.patch
> 
> Review of attachment 8633917 [details] [diff] [review]:
> -----------------------------------------------------------------

I've already almost changed the patch as per your comments.
And also the new patch will resolve the infinite loop problem by using the legacy WidgetEvent.mFlags.mFlagNotCrossProcessBoundaryFowarding.
Soon today I will update the patch.

Comment 86

[Youngwoo Jo]

Attachment: IME_new_hardware_keyboard_event_routing_v0.1.1.patch

This patch has been modified as per Chunmin's review.
And also this includes the following.
 - KeyboardAppProxy is changed to store the only current events but not an ESM nor a target.
 - KeyboardAppProxy is changed to use a queue for pending events.
 - Modification in nsPresShell is removed.
 - Added a new flag to nsDOMWindowUtils for resolving the infinite loop issue.
 - And minor changes

Comment 87

[Marco Chen [:mchen]]

Comment on attachment 8636941 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.1.1.patch

Review of attachment 8636941 [details] [diff] [review]:
-----------------------------------------------------------------

It seems that if the target app is in the b2g process (not remote=true), then the keyboard event will be dispatched by nsPreshell to target app directly.
And the hook point of KeyboardAppProxy is too late?

::: dom/inputmethod/KeyboardAppProxy.cpp
@@ +67,5 @@
> +NS_IMETHODIMP
> +KeyboardAppProxy::SetHasActiveIME(bool aHasActiveIME)
> +{
> +  mHasActiveIME = aHasActiveIME;
> +  return NS_OK;

One question:
  Is there a potential issue of wrong key events order? 
  while 
    1. there are pending events in the queue 
    2. and IME is became inactive
    3. then a new key event is coming

Comment 88

[C.M.Chang[:chunmin]]

Hi, smaug,
I was wondering if you could take a look on the following issues. I'll be grateful if you could give me some suggestions.


Based on the old proposal, there are some errors when target app is in-process
---------------------------------------------------------------------------------
1. Infinite loop to send keyboard events:
   - comment #50
   - Add a flag: keyboardevent.mFlags.mGeneratedFromIME to stop the infinite loop
2. Duplicated keydown, keypress, key up events: 
   - One is fired from hardware, the other is fired from forms.js
3. Duplicated beforeKeydown, afterKeydown, beforeKeyup, afterKeyup
   - The cause is same as the above one.
   - The beforeKey* and afterKey* should follow the original key* events
     - so the key* events fired from forms.js shouldn't trigger them


Possible solution:
---------------------------------------------------------------------------------
We have three use cases: 
a) If keyboard app isn’t active
b) If keyboard app is active and keyboard IME want to handle this keyboard event
   (event.preventDefault() will be called in keyboard IME)
c) If keyboard app is active but keyboard IME doesn’t want to handle this keyboard event
   (event.preventDefault() won't be called in keyboard IME)

To produce appropriate results in all cases, we could use additional conditions to decide the code flow

1) To decide whether or not beforekey* and afterKey* will be dispatched
if (processIsB2G && IMEIsActive && fromIME) {
  // Cancel beforekey* event
  // Cancel afterkey* event
} else {
  // dispatch beforekeykey* event
  // dispatch afterkey* event
}


2) To decide whether or not key* will be fired
if (processIsB2G && IMEIsActive && !fromIME && !targetIsRemote) {
  // block key* event
} else {
  // dispatch key*
}


However, the main concern for this method is that we need to use "targetIsRemote"(target app is in remote process) to decide the code flow. It will be hard to trace code if we use this condition. We need to figure out the use case is in-process or not first. Could we have better solution to get rid of the “targetIsRemote”?

I draw some slides to demonstrate these issues, please go https://goo.gl/1IigKD for detail information.

Comment 89

[C.M.Chang[:chunmin]]

Random thought:
I think comment #66 might be an another solution for comment #88. If we forward key* events via shell.js(or system app) instead of nsPresShell(or EventStateManager), then nsPresShell(or EventStateManager) may not need to know the target app is in-process or out-of-process because shell.js can receive keydown events in both case. However, we still need the keyboardevent.mFlags.mGeneratedFromIME to stop the the infinite loop.

Comment 90

[Youngwoo Jo]

Attachment: IME_new_hardware_keyboard_event_routing_v0.2.patch

This is a patch to resolve the in-process target related issues.
However, this patch is not regarding comment #89 and a temporary patch to verify the event flow.

This patch includes the following changes.
- Added a new flag, WidgetEvent.mFlags.mGeneratedFromIME, to check the case.
- forms.js and nsDOMWindowUtils are changed to set the above flag.
- Removed modification in EventStateManager.
- nsPresShell::HandleKeyboardEvent is modified to use KeyboardAppProxy and to control the routing.
- KeyboardAppProxy is modified to follow the event flow as discussed with Chunmin.
- Minor changes

Comment 91

[Youngwoo Jo]

Attachment: Flow_of_IME_new_hardware_keyboard_event_routing_v0.2.svg

IME_new_hardware_keyboard_event_routing_v0.2.patch follows this event flow as explained in this diagram.
I hope this is useful to you.

Comment 92

[Olli Pettay [:smaug][[e-mail]]]

Sorry about delay, I'm way behind my needinfo queue (since I tend to process review queue first).

Adding a new flag to event.flags is fine, and if you want to expose it to JS, just
add [ChromeOnly] readonly attribute to the relevant .webidl interface.

Why do we get the duplicates? Couldn't forms.js figure out that the event is dispatched by the hardware, and in such case not dispatch event. I thought that was the idea.

Comment 93

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

Hey Vance, 
This is essential features we'll need for 2.2R.
Could you please confirm the ETA (9/25 or earlier) with partner?

Comment 94

[C.M.Chang[:chunmin]]

(In reply to Olli Pettay [:smaug] from comment #92)
Hi, Smaug,
Thx for your reply. Yes, we use a flag event.mFlags.fromIME to distinguish whether the event is dispatched by the hardware, or not. On the other hand, we need to use another condition: "eventTargetIsRemoteProcess" to control the proper event flow. However, it will become harder to trace/maintain code. Is that ok?

Besides, is it necessary to dispatch event to system app first when target app is remote(yellow area of this fig:https://goo.gl/34OiSC)? If the purpose of this behavior is same as mozbrowserbeforeKey*, can we drop it?

Comment 95

[Olli Pettay [:smaug][[e-mail]]]

Why kind of condition is eventTargetIsRemoteProcess?
Is it enough to just check
event.originalTarget == <xul:browser remote="true">/mozbrowser ?
Something like
http://mxr.mozilla.org/mozilla-central/source/dom/events/EventStateManager.cpp?rev=18911ac13934&mark=1151-1151,1156-1160,1163-1166#1151
We could expose that method to JS, if it is not exposed yet.

Currently we do dispatch events first to system process and then EventStateManager forwards to
child process if system process hasn't consumed (preventDefault()) the event - by default at least.
Would it help if the dispatch in system process wouldn't happen?
If the event is cancelable you could of course add event listener somewhere in the system process and call preventDefault(), then manually forward the event to child process.

Comment 97

[C.M.Chang[:chunmin]]

(In reply to Olli Pettay [:smaug] from comment #95)
> Why kind of condition is eventTargetIsRemoteProcess?
> Is it enough to just check
> event.originalTarget == <xul:browser remote="true">/mozbrowser ?
> Something like
> http://mxr.mozilla.org/mozilla-central/source/dom/events/EventStateManager.
> cpp?rev=18911ac13934&mark=1151-1151,1156-1160,1163-1166#1151
> We could expose that method to JS, if it is not exposed yet.

counter-example for this condition: target element is an iframe and it's in in-process app.

If I understand right, the event target for keyboard event will be set on the focus element. AFAIK, every window has a focus. When A.html embeds an iframe that is sourced from B.html, then we will have two window(A's window and B's window) and two focus(one is in A.html, the other is in B.html). 

If A.html is an in-process app, then the keyboard event will be dispatched to focus element in B.html only. Calling A.iframe.focus() indeed can snatch the focus on A.html, but it will also set the focus element of B.html to B.document.body. Thus, the event target will be set to B.document.body in this case.
(If A.html is an oop app, then we will dispatch keyboard events to A's process with event target: A.iframe and B's process with event target:B's focus element)


Summary
-------------------
1) If we use iframe, then we will have two focus.
2) If the app is in-process, then the keyboard event will be dispatched to the inner focus element in iframe's source page.
3) => if the app is in-process, then the event target is not iframe.
   => if event target is iframe, then app is not in-process (oop)

Comment 98

[Jayachandra Yakasiri]

(In reply to Chun-Min Chang[:chunmin] from comment #89)
> Random thought:
> I think comment #66 might be an another solution for comment #88. If we
> forward key* events via shell.js(or system app) instead of nsPresShell(or
> EventStateManager), then nsPresShell(or EventStateManager) may not need to
> know the target app is in-process or out-of-process because shell.js can
> receive keydown events in both case. However, we still need the
> keyboardevent.mFlags.mGeneratedFromIME to stop the the infinite loop.

Hi Chun-Min, 

If we forward the events from shell.js instead of nsPresShell, if we are not calling preventDefault() after forwarding the event it will still propagate to target in in-process case.
If we call preventDefault()after forwarding, then nsAppShell will not generate Keypress event which is required for keyboard app not consumed case to send it to target. 
https://dxr.mozilla.org/mozilla-central/source/widget/gonk/nsAppShell.cpp#346

we can not stop the event using stopPropagation() since it can stop propagation only in its own Document.
Is there any way to solve this problem?

Comment 99

[C.M.Chang[:chunmin]]

(In reply to Jayachandra Yakasiri from comment #98)
> Hi Chun-Min, 
> 
> If we forward the events from shell.js instead of nsPresShell, if we are not
> calling preventDefault() after forwarding the event it will still propagate
> to target in in-process case.
> If we call preventDefault()after forwarding, then nsAppShell will not
> generate Keypress event which is required for keyboard app not consumed case
> to send it to target. 
> https://dxr.mozilla.org/mozilla-central/source/widget/gonk/nsAppShell.cpp#346
> 
> we can not stop the event using stopPropagation() since it can stop
> propagation only in its own Document.
> Is there any way to solve this problem?

Hi, Jayachandra

Maybe you can try calling Event::StopCrossProcessForwarding()[1] to make flag mNoCrossProcessBoundaryForwarding = true after forwarding event to IME, then the event dispatch will be stopped in EventStateManager[2].

Another question for this alternative method(forward keyboard event via shell.js) is that: 
Is there any event listener that will be called before shell's window? (Assume we add an event listener in shell's window to intercept/forward the key events) If the answer is yes, then is it possible to cancel it?

AFAIK, if several listeners are attached to the same element for the same event type, they are called in order in which they have been added. If during one such call, event.stopImmediatePropagation()[3] is called, no remaining listeners will be called. Thus, if we can make sure that our event listener will be added first, then we have a chance to stop whole the event dispatch.

[1]https://dxr.mozilla.org/mozilla-central/source/dom/events/Event.cpp#485
[2]https://dxr.mozilla.org/mozilla-central/source/dom/events/EventStateManager.cpp#1207
[3]https://dxr.mozilla.org/mozilla-central/source/dom/events/Event.cpp#477

Comment 100

[Rachelle Yang [:ryang][[e-mail]]]

Hi Jinyoon, 

Would you please kindly provide your confirmation and feedback on this request ?
Will this work be able to be done(landed on 2.2R) by the end of Sep ?

Thank you very much !!

Comment 101

[Jinyoon]

Rachelle, this work will be done on the given schedule.

Comment 102

[Youngwoo Jo]

Attachment: IME_new_hardware_keyboard_event_routing_v0.3.patch

This is a patch for minimizing the modification in nsPresShell, as per Chunmin's proposal.
So I moved almost codes to shell.js from nsPresShell.
However, the flow is almost same as the previous patch.

Chunmin, could you review this patch?

Comment 103

[C.M.Chang[:chunmin]]

Comment on attachment 8647457 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.3.patch

Review of attachment 8647457 [details] [diff] [review]:
-----------------------------------------------------------------

Overall looks good to me.

::: b2g/chrome/content/shell.js
@@ +352,1 @@
>      window.removeEventListener('keyup', this, true);

In the case that IME is active and IME wan to handle this keyboard event, shell.js will forward the keyboard event once it receive it in capturing phase ans stop the event propagation. If the target app is oop, then we will lose the keyboard event to system app here because the keyboard event will be dispatched to system app and then dispatched to oop target app. I am not sure whether we can intercept the keyboard event here. It might need more tests to verify whether it's ok or not.

::: widget/BasicEvents.h
@@ +492,5 @@
>    // the default action has been performed.
>    bool    mDefaultPrevented : 1;
> +  // mIsGeneratedFromIME is used to check whether the event is generated from
> +  //  nsDOMWindowUtils or actual event from nsAppShell.
> +  bool    mFromIME : 1;

Instead of using "nsDOMWindowUtils", maybe we could just say "Input Method Editor" here because keyboard events will be dispatched by nsITextInputProcessor in the future. See more on Bug 1137557.

::: widget/gonk/nsAppShell.cpp
@@ +317,5 @@
> +       event.mIMEState == WidgetKeyboardEvent::eIMEState::ePending &&
> +       aEventMessage == NS_KEY_DOWN) {
> +      return nsEventStatus_eIgnore;
> +    }
> +    return status;

hmm...It's too hacky to me. This might break the rule of preventDefault(). Is it possible to stop the event propagation by .stop(Immediate)Propagation and .mNoCrossProcessBoundaryForwarding? Then we could avoid hack nsAppShell here.

Comment 104

[Youngwoo Jo]

(In reply to Chun-Min Chang[:chunmin] from comment #103)
> ::: widget/gonk/nsAppShell.cpp
> @@ +317,5 @@
> > +       event.mIMEState == WidgetKeyboardEvent::eIMEState::ePending &&
> > +       aEventMessage == NS_KEY_DOWN) {
> > +      return nsEventStatus_eIgnore;
> > +    }
> > +    return status;
> 
> hmm...It's too hacky to me. This might break the rule of preventDefault().
> Is it possible to stop the event propagation by .stop(Immediate)Propagation
> and .mNoCrossProcessBoundaryForwarding? Then we could avoid hack nsAppShell
> here.

I agree with your opinion, that looks too hacky. however, I do not have any better way.
We have to do the followings.
 - We call only |EventDispatcher::Dispatch| in nsPresShell.
 - This method dispatches a keyboard event only to shell.js(in-process)
   not dispatching to system app nor the target.
I think there is no normal way to make them.

If you think this is too hacky and we can not avoid this, we can adopt the previous patch, IME_new_hardware_keyboard_event_routing_v0.2.patch.

What do you think about that?

Comment 105

[C.M.Chang[:chunmin]]

(In reply to Youngwoo Jo from comment #104)
From my tests on b2g master and v2.2, in-process app won't receive any keyboard event if we call evt.stopImmediatePropagation() in handleEvent of shell.js[1], so we might not need to hack nsAppShell. On the other hand, if we still receive keyboard events when we call evt.stopImmediatePropagation() in shell.js, then it implies that there are some problems in gecko code that will break the rule of event dispatch. We must check our current patch again.

Comment 106

[Youngwoo Jo]

(In reply to Chun-Min Chang[:chunmin] from comment #105)
I confirmed your point is right. my previous verification was wrong.
However, nsEditorEventListener is still checking if defaultPrevented is called to decide handling.
https://dxr.mozilla.org/mozilla-central/source/editor/libeditor/nsEditorEventListener.cpp#624

As a result, so as not to modify nsEditorEventListener, evt.preventDefault() is still required.
Or we have to add the new condition check into nsEditorEventListener.

I think adding some checks into nsEditorEventListener is better than a hack in nsAppShell.
So now I'm trying to find out the new condition check in nsEditorEventListener.

Comment 107

[C.M.Chang[:chunmin]]

(In reply to Youngwoo Jo from comment #106)
Could we check the evt.mFlags.mPropagationStopped? (I'm not sure about this)

Comment 108

[Youngwoo Jo]

(In reply to Chun-Min Chang[:chunmin] from comment #107)
> Could we check the evt.mFlags.mPropagationStopped? (I'm not sure about this)
nsEditorEventListener adds the event listener as system group.
https://dxr.mozilla.org/mozilla-central/source/editor/libeditor/nsEditorEventListener.cpp#164

And system group is not affected by evt.stopPropagation and evt.stopImmediatePropagation called by non-system group, because the flags are reset before dispatch to system group.
https://dxr.mozilla.org/mozilla-central/source/dom/events/EventDispatcher.cpp#347

So we can not use evt.mFlags.mPropagationStopped to check if shell.js calls stopPropagation or not.

Comment 109

[Youngwoo Jo]

We can add another condition check to nsEditorEventListener.
Each event handler of keydown, keypress and keyup can check the condition like the below.
if (KeyboardAppProxy &&
    KeyboardAppProxy.IsEventPending(aEvent)) // Check if aEvent is in queue.
  return NS_OK;
So it can ignore the event handling.

How about that?

Comment 110

[Youngwoo Jo]

Attachment: IME_new_hardware_keyboard_event_routing_v0.4.patch

I removed a hack from nsAppShell. Instead of calling evt.preventDefault in shell.js, I used only evt.stopImmediatePropagation. And nsEditorEventListener can use a new state variable of WidgetKeyboardEvent to check its ignorance.

if (keyEvent.mIMEState == WidgetKeyboardEvent::eIMEState::ePending) {
  return NS_OK;  // Ignored in KeyPress() in nsEditorEventListener
}

Comment 111

[Youngwoo Jo]

My previous patches are not sending a keypress to keyboard app, because I thought keyboard app does not need a keypress and it's just a redundant communication.

Keypress is generated from nsAppShell using a kcm(keycode character mapping) file, if the kcm file defines the mapping for the keycode.
I think a kcm file for qwerty will define the mapping like the below cases.
 - shift + 'a' => 'A'
 - shift + '1' => '!'
So, if we do not send a keypress to keyboard app, keyboard app always should convert the keycode to the proper character code like the above cases.
However, it's duplicated with kcm file's role in case of qwerty.

So, I will change the patch to send a keypress to keyboard app.
If it's not correct, please let me know.
If then, I will stop implementing that.

Comment 112

[Youngwoo Jo]

Attachment: IME_new_hardware_keyboard_event_routing_v0.5.patch

This patch contains the below fixes.
- Sending a keypress to keyboard app.
- Fixed the event order problem caused by the quick multiple inputs.
=> To ensure the event order, I made all the following events pending until KeyboardAppProxy receives keydown's reply.

I think the almost known issues have been resolved.
Chun-min and smaug, could you review this patch?

Comment 113

[Olli Pettay [:smaug][[e-mail]]]

This is possibly something masayuki should look at too, but please add it to my review queue.
(random comment, better to not leave printfs to patches ;) )

Comment 114

[Youngwoo Jo]

Attachment: IME_new_hardware_keyboard_event_routing_v0.6.patch

This patch fixed the infinite waiting queue problem.

I found the infinite waiting queue problem when I tested it with spatial navigation(Bug 1008862, it changes the focus very many times.). KeyboardAppProxy manages the queue for sending events and pending events to make them in-order in the communication between KeyboardAppProxy and keyboard app. However, if the communication has some problem, the queue becomes broken. As a result, the pending events become waiting infinitely and no inputs are handled. The problem is caused by not handling the failure of |KeyboardAppProxy.sendKey| or by no responding from keyboard app with 'Keyboard:SendHardwareKeyEvent'. So, I fixed them in this patch.

Masayuki, could you review this patch please?

Comment 115

[Youngwoo Jo]

Comment on attachment 8654730 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.6.patch

I'm sorry that I've changed the reviewer to smaug.
Masayuki, could you check my patch before the review?

Comment 116

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8654730 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.6.patch

Although, I've not read the above discussion, why do we need to handle keypress too? I feel that keydown/keyup, which are the physical key events, are enough for keyboard apps. And note that we need to change keypress event for conforming UI Events spec. So, keypress events will become not fired for non-printable keydown events. In that time, sending keypress events to keyboard apps may cause compatibility issue.

And looks like that this still uses nsIDOMWindowUtils::SendKey() in forms.js, however, Tim already landed the patch which makes forms.js use nsITextInputProcessor. So, I don't think that you need to change nsIDOMWindowUtils... And also NS_KEY_(DOWN|PRESS|UP) are changed to eKey(Down|Press|Up) and WidgetEvent::message is now WidgetEvent::mMessage. So, you must write this patch with old code. 

And this is a nit, looks like that you forget to change the UUID of nsIPresShell.

Finally, I think that you shouldn't add new flag to BaseEventFlags in this case since it's necessary only in WidgetKeyboardEvent. So, it should be added to WidgetKeyboardEvent.

Anyway, I think that you should separate this patch to smaller pieces. This includes a lot of ideas and steps to change.

Comment 117

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8654730 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.6.patch

Based on masayuki's comment, this patch will need some changes.
(and I wouldn't mind if masayuki reviewed this. I'm in general rather overloaded with reviews, and masayuki knows this code at least as well as I do ;) )

Comment 118

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Comment on attachment 8654730 [details] [diff] [review]
IME_new_hardware_keyboard_event_routing_v0.6.patch

Review of attachment 8654730 [details] [diff] [review]:
-----------------------------------------------------------------

(Just a drive-by comment, not a review of all files.)

::: dom/inputmethod/Keyboard.jsm
@@ +22,5 @@
>  XPCOMUtils.defineLazyGetter(this, "appsService", function() {
>    return Cc["@mozilla.org/AppsService;1"].getService(Ci.nsIAppsService);
>  });
>  
> +XPCOMUtils.defineLazyGetter(this, "KeyboardAppProxy", function() {

This should be lower case "keyboardAppProxy".

Comment 119

[Tim Guan-tin Chien [:timdream] (please needinfo)]

BTW, code in shell.js is not testable -- you would need to find another way to invoke the same call in your XPCOM instance, and left the shell.js code untested.

(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #118)
> This should be lower case "keyboardAppProxy".

Actually, if this is not a trouble for you, the interface name should be some thing like MozInputMethodHardwareKeyboardEventProxy and it's instance named hardwareKeyboardEventProxy, just to avoid the word "keyboard app" from creeping into Gecko.

Thanks!

Comment 120

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano (:masayuki) (Mozilla Japan) from comment #116)
> And looks like that this still uses nsIDOMWindowUtils::SendKey() in
> forms.js, however, Tim already landed the patch which makes forms.js use
> nsITextInputProcessor. 
They're based on v2.2r, so they still use nsIDOMWindowUtils::SendKey() to generate a keyboard event. 

Hi, Youngwoo,
Could you move the modified part of nsIDOMWindowUtils::SendKey() to nsITextInputProcessor when you're available to rebase this patch to master?

and...Do we still need to set KEY_FLAG_NOT_CROSS_PROCESS_BOUNDARY_FORWARDING if we use evt.preventDefault() to stop dispatching keyboard events to other process?

Comment 121

[Youngwoo Jo]

I have a question about type of keyboard event which keyboard app receives via InputMethod.
The previous patch uses it as just a CustomEvent as the below.
  {
    keyCode : value
    charCode : value
    timeStamp : value
  }
timestamp is not defined in KeyboardEvent(standard). Without timestamp, keyboard app can be also implemented. However, it might be required in order that keyboard app can handle the events more correctly.
I think it should be defined newly as a dictionary or it should be a KeyboardEvent without timestamp.

Masayuki, which one is better? Or could you suggest any other solution?

Comment 122

[Youngwoo Jo]

(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #119)
> BTW, code in shell.js is not testable -- you would need to find another way
> to invoke the same call in your XPCOM instance, and left the shell.js code
> untested.
Thank you for your review, Guan-tin.
Right, the code in shell.js is not testable. nsPresShell is triggering the dispatch to shell.js.
I think, only for test, it's better to move the code in shell.js to nsPresShell. However, there is a reason to change shell.js. Its history is presented in the above comments which had been started from Comment 66 and Comment 69. For minimizing the modification in common gecko source code, shell.js is more proper. However, as you commented, there is no way to test the changed code in shell.js even if we make a new method to the proxy component. In my opinion, it's not bad to move the code in shell.js to nsPresShell.
Chunmin, what do you think about that?

> Actually, if this is not a trouble for you, the interface name should be
> some thing like MozInputMethodHardwareKeyboardEventProxy and it's instance
> named hardwareKeyboardEventProxy, just to avoid the word "keyboard app" from
> creeping into Gecko.
I understood that. I'm changing the name to your suggestion, "MozInputMethodHardwareKeyboardEventProxy", However I think "Moz" prefix is not required because this is just an internal xpcom. So I will use the name, "InputMethodHardwareKeyboardEventProxy" without "Moz".

Comment 123

[Youngwoo Jo]

(In reply to Chun-Min Chang[:chunmin] from comment #120)
> They're based on v2.2r, so they still use nsIDOMWindowUtils::SendKey() to
> generate a keyboard event. 
Right, this patch is based on v2.2r. I want the code fix in v2.2r first because of my schedule. 

> Could you move the modified part of nsIDOMWindowUtils::SendKey() to
> nsITextInputProcessor when you're available to rebase this patch to master?
Sure, I will do that. When I tested it on master, I confirmed it works fine with the several minor changes such as the variable name changes. So that's no problem.

> and...Do we still need to set KEY_FLAG_NOT_CROSS_PROCESS_BOUNDARY_FORWARDING
> if we use evt.preventDefault() to stop dispatching keyboard events to other
> process?
I also thought about that. So I will remove KEY_FLAG_NOT_CROSS_PROCESS_BOUNDARY_FORWARDING after I test it.

Comment 124

[Youngwoo Jo]

(In reply to Youngwoo Jo from comment #121)
I'm sorry that Event.webidl has a property, timestamp.
If then, I think we do not need a new type definition for this case.
I will change the code to use KeyboardEvent defined in KeyboardEvent.webidl.

Could you let me know that is incorrect, please.

Comment 125

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Youngwoo Jo from comment #124)
> (In reply to Youngwoo Jo from comment #121)
> I'm sorry that Event.webidl has a property, timestamp.
> If then, I think we do not need a new type definition for this case.

You're right.

Comment 126

[Youngwoo Jo]

Attachment: Part1_IME_prevent_dispatch_mozbrowserbeforeafterkeyevent_for_ime_generating_keyevents.patch

I make the patch to the small pieces.
Part 1 prevents dispatching mozbrowserbeforeafterkey* events for IME generating keyboard events.

Comment 127

[Youngwoo Jo]

Attachment: Part2_IME_added_ime_state_to_WidgetKeyboardEvent.patch

Part2 adds mIMEState to WidgetKeyboardEvent in order to indicate the state of handling a hardware keyboard event by keyboard app.

Comment 128

[Youngwoo Jo]

Attachment: Part3_IME_added_DispatchKeyboardEvent_to_PresShell.patch

Part 3 adds a new method to nsPresShell as a xpcom interface's method in order to dispatch a keyboard event directly to the DOM tree.
It's used by InputMethodHardwareKeyboardEventProxy to dispatch a pending keyboard event after keyboard app replies.

Comment 129

[Youngwoo Jo]

Attachment: Part4_IME_added_InputMethodHardwareKeyboardEventProxy_component.patch

Part 4 is a new xpcom component named, nsIInputMethodHardwareKeyboardEventProxy.
It sends a hardware keyboard event to keyboard app and keyboard app handles the keyboard event. Once keyboard app replies, it runs the remaining flow according to the reply from keyboard app. And, in order to make them, it has a queue to manage the events in order.

Comment 130

[Youngwoo Jo]

Attachment: Part5_IME_communication_between_InputMethodHardwareKeyboardEventProxy_and_keyboard_app.patch

Part 5 adds the communication between Keyboard.jsm and an InputMethod in keyboard app via MessageManager. And, once InputMethod receives a keyboard event, InputMethod fires a custom event for keyboard event to MozInputContext.
This implementation is dependent on Bug 1161424.

Comment 131

[Youngwoo Jo]

Attachment: Part6_IME_hooking_by_InputMethodHardwareKeyboardEventProxy_in_shell.js.patch

Part 6 hooks a keyboard event from shell.js in capturing phase by calling InputMethodHardwareKeyboardEventProxy.shouldEventPending() and sendKey(). And if it calls sendKey(), it stops the propagation by calling evt.stopImmediatePropagation().
This implementation has two constraints that it must be the first line of the handlers of shell.js and it can not be tested according to Comment 122.

Comment 132

[Youngwoo Jo]

Attachment: Part7_IME_prevent_duplicated_processing_in_nsEditorEventListener.patch

Part7 fixed a problem which prints a character two times in the editor for one keyboard event in case of an in-process target app.
You can find the history from comment #98 to comment #110.

Comment 133

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8655830 [details] [diff] [review]
Part1_IME_prevent_dispatch_mozbrowserbeforeafterkeyevent_for_ime_generating_keyevents.patch

>diff --git a/dom/inputmethod/forms.js b/dom/inputmethod/forms.js
>index 4f8ba47..eb01caf 100644
>--- a/dom/inputmethod/forms.js
>+++ b/dom/inputmethod/forms.js
>@@ -505,19 +505,20 @@ let FormAssistant = {
>         let event = target.ownerDocument.createEvent('HTMLEvents');
>         event.initEvent('input', true, false);
>         target.dispatchEvent(event);
>         break;
>       }
> 
>       case "Forms:Input:SendKey":
>         CompositionManager.endComposition('');
> 
>-        let flags = domWindowUtils.KEY_FLAG_NOT_SYNTHESIZED_FOR_TESTS;
>+        let flags = domWindowUtils.KEY_FLAG_NOT_SYNTHESIZED_FOR_TESTS |
>+                    domWindowUtils.KEY_FLAG_GENERATED_FROM_IME;
>         this._editing = true;
>         let doKeypress = domWindowUtils.sendKeyEvent('keydown', json.keyCode,
>                                                      json.charCode, json.modifiers, flags);
>         if (doKeypress) {
>           domWindowUtils.sendKeyEvent('keypress', json.keyCode,
>                                       json.charCode, json.modifiers, flags);
>         }
> 
>         if(!json.repeat) {

Hey, why don't you write the patches based on the latest code? We've already stopped using nsIDOMWindowUtils.sendKeyEvent() already:
http://mxr.mozilla.org/mozilla-central/source/dom/inputmethod/forms.js?rev=478dd6079573&mark=727-727,797-797,801-801,806-806,812-812#723

Comment 134

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8655831 [details] [diff] [review]
Part2_IME_added_ime_state_to_WidgetKeyboardEvent.patch

Hmm, this file is based on too old tree... Please rebase to the latest m-c.

>diff --git a/widget/TextEvents.h b/widget/TextEvents.h
>index fc967e9..240e77f 100644
>--- a/widget/TextEvents.h
>+++ b/widget/TextEvents.h
>@@ -96,18 +96,19 @@ public:
>     , mKeyNameIndex(mozilla::KEY_NAME_INDEX_Unidentified)
>     , mCodeNameIndex(CODE_NAME_INDEX_UNKNOWN)
>     , mNativeKeyEvent(nullptr)
>     , mUniqueId(0)
> #ifdef XP_MACOSX
>     , mNativeKeyCode(0)
>     , mNativeModifierFlags(0)
> #endif
>     , mIsFromIME(false)
>+    , mIMEState(eIMEState::eNotHandled)
>   {
>   }

I think that you should initialize mIMEState in the default constructor too if you don't copy the value at IPC.

>+  // Indicates that the event is being handled by keyboard app
>+  enum eIMEState {

nit: "{" should be the next line (below "e" of "enum").

And please remove "e" from the type name and IMEState sounds odd because this indicates the state of keyboard apps. So, perhaps, "KeyboardAppState"? And also it should be uint8_t for reducing the class size. So, please define this enum as:

typedef uint8_t KeyboardAppStateType;
enum KeyboardAppState : KeyboardAppStateType
{
  eNotHandled,...

>+    eNotHandled,  // not yet handled by keyboard app
>+    ePending,     // being handled by keyboard app
>+    eHandled      // handled by keyboard app
>+  };
>+  eIMEState mIMEState;

So, mKeyboardAppState?

Comment 135

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8655841 [details] [diff] [review]
Part4_IME_added_InputMethodHardwareKeyboardEventProxy_component.patch

>diff --git a/dom/inputmethod/InputMethodHardwareKeyboardEventProxy.cpp b/dom/inputmethod/InputMethodHardwareKeyboardEventProxy.cpp
>new file mode 100644
>index 0000000..a2f7957
>--- /dev/null
>+++ b/dom/inputmethod/InputMethodHardwareKeyboardEventProxy.cpp
>@@ -0,0 +1,356 @@

Please add these two lines before the license text:

/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* vim: set ts=8 sts=2 et sw=2 tw=80: */

See https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Coding_Style#Mode_Line

>+/* This Source Code Form is subject to the terms of the Mozilla Public
>+ * License, v. 2.0. If a copy of the MPL was not distributed with this file,
>+ * You can obtain one at http://mozilla.org/MPL/2.0/. */
>+
>+#include "mozilla/ClearOnShutdown.h"
>+#include "nsFocusManager.h"
>+#include "nsIContent.h"
>+#include "nsPIDOMWindow.h"
>+#include "mozilla/BasicEvents.h"
>+#include "nsFrameLoader.h"
>+#include "nsPresShell.h"
>+#include "InputMethodHardwareKeyboardEventProxy.h"
>+#include "mozilla/dom/TabParent.h"
>+#include "mozilla/TextEvents.h"
>+#include "nsDeque.h"
>+#include "nsIDOMKeyEvent.h"

Could you sort out the order from "a" to "z"?

>+
>+namespace mozilla {
>+
>+using namespace dom;
>+
>+NS_IMPL_ISUPPORTS(InputMethodHardwareKeyboardEventProxy, nsIInputMethodHardwareKeyboardEventProxy)

Over 80 character, perhaps:

NS_IMPL_ISUPPORTS(InputMethodHardwareKeyboardEventProxy,
                  nsIInputMethodHardwareKeyboardEventProxy)

>+StaticRefPtr<InputMethodHardwareKeyboardEventProxy> InputMethodHardwareKeyboardEventProxy::sSingleton;

Same. Perhaps:

StaticRefPtr<InputMethodHardwareKeyboardEventProxy>
  InputMethodHardwareKeyboardEventProxy::sSingleton;

And I like sInstance because a lot of singleton classes use the name.

>+InputMethodHardwareKeyboardEventProxy::InputMethodHardwareKeyboardEventProxy()
>+  : mHasActiveIME(false)
>+  , mEventQueue(new nsDeque)

Looks odd. Why don't you define bool members at last? Member variables less than 4 byte should be defined at last for reducing the memory.

And why do you need to define mEventQueue as nsDeque* rather than just nsDeque?

>+InputMethodHardwareKeyboardEventProxy::~InputMethodHardwareKeyboardEventProxy()
>+{
>+  if (mEventQueue) {

if (!mEventQueue) {
  return;
}

Then, following while statement will be less than 80 characters.

>+    WidgetKeyboardEvent* event = nullptr;
>+    while ((event = static_cast<WidgetKeyboardEvent*>(mEventQueue->PopFront()))) {
>+      delete event;
>+    }
>+    delete mEventQueue;
>+    mEventQueue = nullptr;
>+  }
>+}
>+
>+NS_IMETHODIMP
>+InputMethodHardwareKeyboardEventProxy::ShouldHookEvent(nsIDOMKeyEvent* aEvent,
>+                                  bool *aShouldHook)

Odd indent. "b" of bool should be below the "n" of nsIDOMKeyEvent. And please add * to the type, not before the argument name.

>+{
>+  WidgetKeyboardEvent* keyEvent =
>+    aEvent->GetInternalNSEvent()->AsKeyboardEvent();
>+  MOZ_ASSERT(keyEvent,
>+             "DOM key event's internal event must be WidgetKeyboardEvent");
>+
>+  if (!mHasActiveIME ||
>+      keyEvent->mIsFromIME ||
>+      keyEvent->mIMEState != WidgetKeyboardEvent::eIMEState::eNotHandled) {
>+    *aShouldHook = false;
>+    return NS_OK;
>+  }
>+
>+  *aShouldHook = true;

Looks like that this can be:

*aShouldHook = mHasActiveIME && !keyEvent->mIsFromIME &&
               keyEvent->mIMEState == WidgetKeyboardEvent::eNotHandled;


>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+InputMethodHardwareKeyboardEventProxy::RegisterKeyboardEventForwarder(nsIKeyboardEventForwardCb* aCb)

Too long, perhaps:

InputMethodHardwareKeyboardEventProxy::RegisterKeyboardEventForwarder(
                                         nsIKeyboardEventForwardCb* aCb)


>+{
>+  mKeyboardEventForwardCb = aCb;

And I don't like "Cb". They should be aCallback and mKeyboardEventForwardCallback or something.

>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+InputMethodHardwareKeyboardEventProxy::SendKey(nsIDOMKeyEvent* aEvent, bool* aResult)

Too long, perhaps:

InputMethodHardwareKeyboardEventProxy::SendKey(nsIDOMKeyEvent* aEvent,
                                               bool* aResult)

>+{
>+  MOZ_ASSERT(mHasActiveIME, "Keyboard is not activated!");
>+  MOZ_ASSERT(mKeyboardEventForwardCb,
>+               "KeyboardEventForwardCb should be set before calling SendKey");

nit: you can reduce two spaces of this line.

>+
>+  WidgetKeyboardEvent* originalEvent =
>+    aEvent->GetInternalNSEvent()->AsKeyboardEvent();
>+  MOZ_ASSERT(keyEvent,
>+             "DOM key event's internal event must be WidgetKeyboardEvent");
>+
>+  // Copy and store a keyboard event to reuse it when its reply arrives.
>+  nsAutoPtr<WidgetKeyboardEvent> copiedEvent(
>+    new WidgetKeyboardEvent(*originalEvent));
>+
>+  if (copiedEvent->message == NS_KEY_PRESS) {
>+    mEventQueue->Push(copiedEvent);
>+  } else {
>+    nsString eventType;
>+    switch (copiedEvent->message) {
>+      case NS_KEY_DOWN:
>+        eventType = NS_LITERAL_STRING("keydown");
>+        break;
>+      case NS_KEY_UP:
>+        eventType = NS_LITERAL_STRING("keyup");
>+        break;
>+    }
>+    bool isSent = false;
>+    if (mKeyboardEventForwardCb) {
>+      mKeyboardEventForwardCb->OnKeyboardEventReceived(eventType,
>+                                                       copiedEvent->keyCode,
>+                                                       copiedEvent->charCode,
>+                                                       copiedEvent->time,
>+                                                       &isSent);

Hmm, too poor information... Why don't you send nsIDOMKeyEvent? Then, whole information available at the receiver.

>+    }
>+
>+    if (isSent) {
>+      mEventQueue->Push(copiedEvent);
>+    } else {
>+      *aResult = false;
>+      return NS_OK;
>+    }

if (!isSent) {
  *aResult = false;
  return NS_OK;
}
mEventQueue->Push(copiedEvent);

>+NS_IMETHODIMP
>+InputMethodHardwareKeyboardEventProxy::ReplyKey(const nsAString& aEventType,
>+                           bool aDefaultPreventedStatus)

Odd indent...

InputMethodHardwareKeyboardEventProxy::ReplyKey(const nsAString& aEventType,
                                                bool aDefaultPreventedStatus)

And aDefaultPrevented is better name.

>+{
>+  MOZ_ASSERT((aEventType.EqualsLiteral("keydown") ||
>+              aEventType.EqualsLiteral("keyup")),
>+             "Invalid event type");
>+
>+  // We can not handle this reply because the pending event had been already
>+  // removed from the sending queue before this reply arrives.
>+  if (mEventQueue->GetSize() == 0) {

nit: if (!mEventQueue->GetSize()) {

# I wonder why nsDeque doesn't have IsEmpty()...

>+    return NS_OK;
>+  }
>+
>+  nsAutoPtr<WidgetKeyboardEvent> keyEvent(
>+    static_cast<WidgetKeyboardEvent*>(mEventQueue->PopFront()));
>+
>+  MOZ_ASSERT(((aEventType.EqualsLiteral("keydown") && keyEvent->message == NS_KEY_DOWN) ||
>+              (aEventType.EqualsLiteral("keyup") && keyEvent->message == NS_KEY_UP)),

should be:

  MOZ_ASSERT(((aEventType.EqualsLiteral("keydown") &&
               keyEvent->message == NS_KEY_DOWN) ||
              (aEventType.EqualsLiteral("keyup") &&
               keyEvent->message == NS_KEY_UP)),



>+             "Replied key event does not match with the pending event");
>+
>+  // Set the flag to specify the reply phase.
>+  keyEvent->mIMEState = WidgetKeyboardEvent::eIMEState::eHandled;

You didn't define the enum as enum class. So, you don't need to write the enum name. Please remove that for reducing each line length.

>+  // Reset the flags to dispatch this event again.
>+  keyEvent->mFlags.mDefaultPrevented = false;
>+  keyEvent->mFlags.mIsBeingDispatched = false;

I wonder, could you add WidgetEvent::ResetForReuse()? Resetting here could be broken by other developers (including me!).

>+
>+  if (aDefaultPreventedStatus) {
>+    keyEvent->mFlags.mDefaultPrevented = true;

Although, I've not understood who calls this, you need to tread mDefaultPreventedByContent and mDefaultPreventedByChrome too.


I'll check following code in this patch, later, but I hope you will rebase the patches and conform to 80 characters per line rule...

Comment 136

[C.M.Chang[:chunmin]]

Hi, Youngwoo,
I think the goal of the test is to verify the event dispatch flow. If there's a way to activate/deactivate the IME(e.g. focus on a input text), can we test whether KeyboardAppProxy::ShouldHookEvent works fine or not from behavior instead of testing shell.js directly? [1,2] may be good test examples. However, we need to find a way to simulate the IME if we use mochitest.

[1] https://dxr.mozilla.org/mozilla-central/source/dom/events/test/test_dom_before_after_keyboard_event_remote.html
[2] https://dxr.mozilla.org/mozilla-central/source/dom/events/test/test_dom_before_after_keyboard_event.html

Comment 137

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Sorry, I couldn't review the remaining part today. Could you rebase the patches first and fix the lines which breaks 80 char par line rule? Then, I can review them easier.

Comment 138

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

(In reply to Masayuki Nakano (:masayuki) (Mozilla Japan) from comment #134)
> Comment on attachment 8655831 [details] [diff] [review]
> Part2_IME_added_ime_state_to_WidgetKeyboardEvent.patch
> 
> Hmm, this file is based on too old tree... Please rebase to the latest m-c.
> 
Just to clarify the expectation:
* we appreciate partner's commitment on contribution this work
* we agreed in the workshop in June that if necessary, partner can focus on landing to 2.2R firstly and the m-c.

Comment 139

[Youngwoo Jo]

(In reply to Chun-Min Chang[:chunmin] from comment #136)
> Hi, Youngwoo,
> I think the goal of the test is to verify the event dispatch flow. If
> there's a way to activate/deactivate the IME(e.g. focus on a input text),
> can we test whether KeyboardAppProxy::ShouldHookEvent works fine or not from
> behavior instead of testing shell.js directly? [1,2] may be good test
> examples. However, we need to find a way to simulate the IME if we use
> mochitest.
Chunmin, Thank you for your opinion.
As you mentioned, IME can be simulated in the test. However, shell.js is not loaded in mochitest. If then, which one should call |KeyboardAppProxy::ShouldHookEvent| and |KeyboardAppProxy::SendKey|? According to your opinion, should the test script call them to simulate shell.js? I'm not sure if the test is correct or not.
Could you let me know that is a correct way or not?

Comment 140

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8655835 [details] [diff] [review]
Part3_IME_added_DispatchKeyboardEvent_to_PresShell.patch

Sorry, I don't have much time to review them. For reducing the review time, please rebase your patches with the latest m-c and conform to the 80 chars per line rule. Then, I can focus to check the main purpose of the changes.

Comment 141

[C.M.Chang[:chunmin]]

(In reply to Youngwoo Jo from comment #139)
Hi, Youngwoo,
I think Marionette maybe an alternative solution. However, I can't find any API to simulate the hardware keyboard event(it has a way to send key event from nsIDOMWindowUtils but it's not what we want). I am trying to contact the testing guys and let you know if I have further information.

Comment 142

[Youngwoo Jo]

Attachment: Part1_IME_prevent_dispatch_mozbrowserbeforeafterkeyevent_for_ime_generating_keyevents.patch

I'm sorry for the late new patch.
I've rebased it to m-c and modified it as your review.
Masayuki, could you review this again, please?

Comment 143

[Youngwoo Jo]

Attachment: Part2_IME_added_ime_state_to_WidgetKeyboardEvent.patch

I've changed |WidgetKeyboardEvent.imeState| to |WidgetKeyboardEvent.keyboardAppState| and changed the more things as your review.

Comment 144

[Youngwoo Jo]

Attachment: Part3_IME_added_DispatchKeyboardEvent_to_PresShell.patch

It's rebased to m-c.

Comment 145

[Youngwoo Jo]

Attachment: Part4_IME_added_InputMethodKeyEventProxy_component.patch

It's rebased to m-c and changed as Masayuki's review.

Comment 146

[Youngwoo Jo]

Attachment: Part5_IME_communication_between_InputMethodKeyEventProxy_and_keyboard_app.patch

It's rebased to m-c.

Comment 147

[Youngwoo Jo]

Attachment: Part6_IME_hooking_by_InputMethodKeyEventProxy_in_shell.js.patch

It's rebased to m-c.
Masayuki, thanks for your kindly review.
Could you review my new patches?

Comment 148

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #141)
> (In reply to Youngwoo Jo from comment #139)
> Hi, Youngwoo,
> I think Marionette maybe an alternative solution. However, I can't find any
> API to simulate the hardware keyboard event(it has a way to send key event
> from nsIDOMWindowUtils but it's not what we want). I am trying to contact
> the testing guys and let you know if I have further information.
I find a way to simulate the hardware keyboard event in python version of Marionette.


import mozdevice
dm = mozdevice.DeviceManagerADB()
'''simulate to press b by adb shell'''
print '==> press physical key: b'
dm.shellCheckOutput(['sendevent', 'dev/input/event0','1', '48', '1'])
dm.shellCheckOutput(['sendevent', 'dev/input/event0','0', '0', '0'])
dm.shellCheckOutput(['sendevent', 'dev/input/event0','1', '48', '0'])
dm.shellCheckOutput(['sendevent', 'dev/input/event0','0', '0', '0'])

adb shell command can simulate not only physical keyboard events but also touch events, sensors events, ...etc, so I hope that we can expend this idea into a library and make it reusable by other test cases.

However, the bad news is that python version of Marionette will be phased out on try-server years later. 
I will open another bug to follow up this staff.

Comment 149

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8662924 [details] [diff] [review]
Part1_IME_prevent_dispatch_mozbrowserbeforeafterkeyevent_for_ime_generating_keyevents.patch

Perhaps, mIsFromIME should be called mIsSynthesizedByTIP because mIsFromIME sounds like that the keyboard event was caused by IME including native IME of other platforms.

(TIP is also another name of IME in TSF terms on Windows, however, it's not major to other developers who don't touch TSF specific part.)

Comment 150

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8662926 [details] [diff] [review]
Part2_IME_added_ime_state_to_WidgetKeyboardEvent.patch

>@@ -380,26 +379,29 @@ struct ParamTraits<mozilla::WidgetKeyboardEvent>
>     WriteParam(aMsg, aParam.location);
>     WriteParam(aMsg, aParam.mUniqueId);
> #ifdef XP_MACOSX
>     WriteParam(aMsg, aParam.mNativeKeyCode);
>     WriteParam(aMsg, aParam.mNativeModifierFlags);
>     WriteParam(aMsg, aParam.mNativeCharacters);
>     WriteParam(aMsg, aParam.mNativeCharactersIgnoringModifiers);
>     WriteParam(aMsg, aParam.mPluginTextEventString);
> #endif
>+    WriteParam(aMsg, static_cast<uint8_t>(aParam.mKeyboardAppState));

Use KeyboardAppStateType instead of uint8_t.

>     WriteParam(aMsg, aParam.mIsFromIME);
>+
>     // An OS-specific native event might be attached in |mNativeKeyEvent|,  but
>     // that cannot be copied across process boundaries.
>   }
> 
>   static bool Read(const Message* aMsg, void** aIter, paramType* aResult)
>   {
>     uint32_t keyNameIndex = 0, codeNameIndex = 0;
>+    uint8_t keyboardAppState = 0;

Same here.

>@@ -408,25 +410,28 @@ struct ParamTraits<mozilla::WidgetKeyboardEvent>
>         ReadParam(aMsg, aIter, &aResult->location) &&
>         ReadParam(aMsg, aIter, &aResult->mUniqueId)
> #ifdef XP_MACOSX
>         && ReadParam(aMsg, aIter, &aResult->mNativeKeyCode)
>         && ReadParam(aMsg, aIter, &aResult->mNativeModifierFlags)
>         && ReadParam(aMsg, aIter, &aResult->mNativeCharacters)
>         && ReadParam(aMsg, aIter, &aResult->mNativeCharactersIgnoringModifiers)
>         && ReadParam(aMsg, aIter, &aResult->mPluginTextEventString)
> #endif
>+        && ReadParam(aMsg, aIter, &keyboardAppState)
>         && ReadParam(aMsg, aIter, &aResult->mIsFromIME)
>         )
>     {
>       aResult->mKeyNameIndex = static_cast<mozilla::KeyNameIndex>(keyNameIndex);
>       aResult->mCodeNameIndex =
>         static_cast<mozilla::CodeNameIndex>(codeNameIndex);
>       aResult->mNativeKeyEvent = nullptr;
>+      aResult->mKeyboardAppState =
>+        static_cast<mozilla::WidgetKeyboardEvent::KeyboardAppState>(keyboardAppState);

Then, this is always safe even in the future.

# Although, mKeyNameIndex and mCodeNameIndex are unsafe, it should be fixed in another bug.

Comment 151

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8662930 [details] [diff] [review]
Part4_IME_added_InputMethodKeyEventProxy_component.patch

>diff --git a/dom/inputmethod/nsIInputMethodKeyEventProxy.idl b/dom/inputmethod/nsIInputMethodKeyEventProxy.idl
>new file mode 100644
>index 0000000..3ed53ba
>--- /dev/null
>+++ b/dom/inputmethod/nsIInputMethodKeyEventProxy.idl
>@@ -0,0 +1,63 @@
>+/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
>+/* vim: set ts=8 sts=2 et sw=2 tw=80: */
>+/* This Source Code Form is subject to the terms of the Mozilla Public
>+ * License, v. 2.0. If a copy of the MPL was not distributed with this
>+ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
>+
>+#include "nsISupports.idl"
>+
>+%{C++
>+#define NS_INPUTMETHOD_KEY_EVENT_PROXY_CID \
>+  { 0x8b86dacb, 0xc0ad, 0x42c7, { 0x95, 0x11, 0x49, 0x8f, 0x1e, 0xed, 0x4e, 0x80 } }

nit: over 80 characters. Perhaps:

>+#define NS_INPUTMETHOD_KEY_EVENT_PROXY_CID \
>+  { 0x8b86dacb, 0xc0ad, 0x42c7, \
>+    { 0x95, 0x11, 0x49, 0x8f, 0x1e, 0xed, 0x4e, 0x80 } }

>+#define NS_INPUTMETHOD_KEY_EVENT_PROXY_CONTRACTID \
>+  "@mozilla.org/InputMethodKeyEventProxy;1"
>+%}
>+
>+interface nsIDOMKeyEvent;
>+
>+[scriptable, function, uuid(de699629-fc2a-4fff-b268-69e50a8f1f08)]
>+interface nsIKeyEventForwardCallback : nsISupports

How about nsIHardwareKeyEventListener?

Could you add the summary of this interface above uuid definition? (example: http://mxr.mozilla.org/mozilla-central/source/dom/interfaces/base/nsITextInputProcessorCallback.idl)

>+{
>+  /**
>+   * Forwards a keyboard event to the module having the communication
>+   * channel with the active InputMethod.
>+   */
>+  bool onKeyEventReceived(in nsIDOMKeyEvent aEvent);

How about OnHardwareKey()?

>+};
>+
>+[scriptable, uuid(f468a3ea-6d58-405c-ba63-1dcec3944d03)]

I guess this should be "builtinclass".

>+interface nsIInputMethodKeyEventProxy : nsISupports

"Proxy" is also used by a name of design pattern. How about nsIHardwareKeyHandler or something?

And also I'd like you to add the summary of this interface as I mentioned above.

>+{
>+  void registerKeyEventForwarder(in nsIKeyEventForwardCallback aForwarder);

Please explain about that with a comment what happens if another nsIKeyEventForwardCallback has already been registered.

>+  /**
>+   * Returns true if a keyboard event should be hooked by a content
>+   * having a InputMethod. Otherwise, returns false.
>+   */
>+  bool shouldHookEvent(in nsIDOMKeyEvent aEvent);
>+
>+  /**
>+   * Sends a hardware keyboard event to the active InputMethod to
>+   * give the first chance to handle a keyboard event in InputMethod.
>+   * Content having an InputMethod can receive a keyboard event via
>+   * this method.
>+   */
>+  bool sendKey(in nsIDOMKeyEvent aEvent);

According to the part6, I think that you can merge these methods to |bool maybeSendKeyToInputMethodApp()| or |bool maybeSendKeyToKeyboardApp()|. If it should return true when the sending keyboard event is handled by the keyboard app.

BTW, we should sort out the terms here. I think that we should call the app to input text as one of IME, InputMethodApp or KeyboardApp. IME is shorter but not clear in XP level code. So, I like InputMethodApp or KeyboardApp. In the future, if Firefox OS will support speech input or something, InputMethodApp might be better.

Please choose one from them or suggest better name. And use it for part1 too.

>+  /**
>+   * A Reply paired with |sendKey| from the active InputMethod.
>+   * Content having an InputMethod hanndls a keyboard event and it can
>+   * call |evt.preventDefault|. So this method informs that the default
>+   * prevented or not.
>+   */
>+  void replyKey(in DOMString aEventType,
>+                in bool aDefaultPrevented,
>+                in bool aDefaultPreventedByContent,
>+                in bool aDefaultPreventedByChrome);

How about |OnHandledByInputMethodApp()| or something?

>+  /**
>+   * Indicates that InputMethod is activated or not. In other words,
>+   * this implies an InputContext exists or not.
>+   */
>+  attribute bool hasActiveIME;

I have a question, this is an interface for singleton class, so, I guess that this is created per application's process, right? I worry about that an instance of this interface can be used two or more IME at once. If it's possible, you should check if unexpected situation occurs.

Anyway, I think that OnInputMethodAppConnected() and OnInputMethodAppDisconnected() are better to me.

>+};


I'd like to discuss the names and their structure first. My suggestions are not orders, please tell me your idea.

Comment 152

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8662928 [details] [diff] [review]
Part3_IME_added_DispatchKeyboardEvent_to_PresShell.patch

Now I'm lost with this. Why would we ever want to bypass most of the
default key event handling happening in EventStateManager. Please explain.

And looks wrong to add the method to presshell, given that we don't really use presshell itself for anything in that method.

Comment 153

[C.M.Chang[:chunmin]]

(In reply to Youngwoo Jo from comment #139)
Hi, Youngwoo,
After applying this patch, you could simulate to fire hardware keyboard events via ADB
https://github.com/ChunMinChang/gaia/commit/bd991a794c62fe2e8e96e227f3d4fca0f83e0ed9

This patch use Marionette as testing framework, so you need to setup the following environment first
https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Automated_testing/gaia-ui-tests

The test file is run on emulator, so remember to open the hardware keyboard setting in <B2G>/external/qemu/android/avd/hardware-properties.ini
…
…
# Keyboard support (qwerty/azerty)
name        = hw.keyboard                                                                      
type        = boolean
#default     = no
default     = yes
abstract    = Keyboard support
description = Whether the device has a QWERTY keyboard.
…
…

Comment 154

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #153)
To use this test on red-tai reference device, the following line:
https://github.com/ChunMinChang/gaia/commit/bd991a794c62fe2e8e96e227f3d4fca0f83e0ed9#diff-ffe3a2c997dc0d592b72f922cb77f791R39
should be modified as:
kbevt = deviceEventInputer.KeyboardEvents(keys, devieName = 'keypad')
and |keys| should also be changed into the keys that used on device keypad(e.g. 0~9, #, *,..)

Comment 155

[Youngwoo Jo]

(In reply to Masayuki Nakano (:masayuki) (Mozilla Japan) from comment #151)
> Comment on attachment 8662930 [details] [diff] [review]
> Part4_IME_added_InputMethodKeyEventProxy_component.patch
First of all, I appreciate your review.

> "Proxy" is also used by a name of design pattern. How about
> nsIHardwareKeyHandler or something?
How about nsIHardwareKeyEventHandler instead of nsIHardKeyHandler? However, the name is getting logger. It makes me too diffcult to make the lines shorter than 80 characters, because I'm not familiar with mozilla coding style. :(

> BTW, we should sort out the terms here. I think that we should call the app
> to input text as one of IME, InputMethodApp or KeyboardApp. IME is shorter
> but not clear in XP level code. So, I like InputMethodApp or KeyboardApp. In
> the future, if Firefox OS will support speech input or something,
> InputMethodApp might be better.
> 
> Please choose one from them or suggest better name. And use it for part1 too.
In my opinion, InputMethodApp* looks better than KeyboardApp* And, if the name is decided, I will change all other names related with this concept.

> I have a question, this is an interface for singleton class, so, I guess
> that this is created per application's process, right? I worry about that an
> instance of this interface can be used two or more IME at once. If it's
> possible, you should check if unexpected situation occurs.
I added the below check to ensure it's created only in b2g process. So the other processes can not create a new instance.

+/* static */ already_AddRefed<InputMethodKeyEventProxy>
+InputMethodKeyEventProxy::GetInstance()
+{
+  if (!XRE_IsParentProcess()) {
+    return nullptr;
+  }

> Anyway, I think that OnInputMethodAppConnected() and
> OnInputMethodAppDisconnected() are better to me.
I agree. It looks better to me.

> I'd like to discuss the names and their structure first. My suggestions are
> not orders, please tell me your idea.
About the other things, totally I agree with your idea. I will change all things by following your opinion. Until now, my update has been delayed because I had many works interrupting me. ASAP I will upload the new patch. :)

Comment 156

[Youngwoo Jo]

(In reply to Olli Pettay [:smaug] from comment #152)
> Comment on attachment 8662928 [details] [diff] [review]
> Part3_IME_added_DispatchKeyboardEvent_to_PresShell.patch
Thanks for your review, Olli.

> Now I'm lost with this. Why would we ever want to bypass most of the
> default key event handling happening in EventStateManager. Please explain.
The discussion to remove the changes in EventStateManager had started since Comment 66. At that time, I understood that is to make debug easily to reduce the code complexity caused by multi process and to minimize the modifications in legay codes and in legacy event flows.

> And looks wrong to add the method to presshell, given that we don't really
> use presshell itself for anything in that method.
You're right. It's not using nsPresShell itself. Therefor I can remove the this patch.

Comment 157

[C.M.Chang[:chunmin]]

(In reply to Youngwoo Jo from comment #130)
> Created attachment 8655849 [details] [diff] [review]
> Part5_IME_communication_between_InputMethodHardwareKeyboardEventProxy_and_key
> board_app.patch
handleFocusChange has already been removed from Keyboard.jsm after bug 1197700 was landed. Remember to rebase the code to the latest one when you're available.

Comment 158

[C.M.Chang[:chunmin]]

Attachment: [WIP] test cases

(In reply to Chun-Min Chang[:chunmin] from comment #153)
Now three cases are all testable:
1) Send keys when IME is inactive
2) Send Keys when IME is active, and IME will consume these keys
3) Send Keys when IME is active, but IME won't consume these keys

Comment 159

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Youngwoo Jo from comment #155)
> (In reply to Masayuki Nakano (:masayuki) (Mozilla Japan) from comment #151)
> > Comment on attachment 8662930 [details] [diff] [review]
> > Part4_IME_added_InputMethodKeyEventProxy_component.patch
> First of all, I appreciate your review.

np, and sorry for the delay to reply.

> > "Proxy" is also used by a name of design pattern. How about
> > nsIHardwareKeyHandler or something?
> How about nsIHardwareKeyEventHandler instead of nsIHardKeyHandler? However,
> the name is getting logger. It makes me too diffcult to make the lines
> shorter than 80 characters, because I'm not familiar with mozilla coding
> style. :(

How about nsIHardwareKeyHandler? Indeed, "Hard" can mean "Hardware", but not explicitly.

I like clearer name better even if it's long (but not *too* long).

> > BTW, we should sort out the terms here. I think that we should call the app
> > to input text as one of IME, InputMethodApp or KeyboardApp. IME is shorter
> > but not clear in XP level code. So, I like InputMethodApp or KeyboardApp. In
> > the future, if Firefox OS will support speech input or something,
> > InputMethodApp might be better.
> > 
> > Please choose one from them or suggest better name. And use it for part1 too.
> In my opinion, InputMethodApp* looks better than KeyboardApp* And, if the
> name is decided, I will change all other names related with this concept.

Okay, let's use it.

> > I have a question, this is an interface for singleton class, so, I guess
> > that this is created per application's process, right? I worry about that an
> > instance of this interface can be used two or more IME at once. If it's
> > possible, you should check if unexpected situation occurs.
> I added the below check to ensure it's created only in b2g process. So the
> other processes can not create a new instance.

Okay.

> > I'd like to discuss the names and their structure first. My suggestions are
> > not orders, please tell me your idea.
> About the other things, totally I agree with your idea. I will change all
> things by following your opinion. Until now, my update has been delayed
> because I had many works interrupting me. ASAP I will upload the new patch.
> :)

I see, thanks. And sorry for the delay again. I'll be back in normal work style at late this week.

Comment 160

[Youngwoo Jo]

(In reply to Chun-Min Chang[:chunmin] from comment #158)
> Created attachment 8670129 [details] [diff] [review]
> [WIP] test cases
> 
> (In reply to Chun-Min Chang[:chunmin] from comment #153)
> Now three cases are all testable:
> 1) Send keys when IME is inactive
> 2) Send Keys when IME is active, and IME will consume these keys
> 3) Send Keys when IME is active, but IME won't consume these keys

Thanks for your test cases. With your help, I've finished setting up the test environment for emulator gaia-ui-test. And now I'm trying to test my patches. And then I will upload the unit test cases using mochitest asap.

Comment 161

[Youngwoo Jo]

Attachment: Part1_IME_prevent_dispatch_mozbrowserbeforeafterkeyevent_for_ime_generating_keyevents.patch

I changed mIsFromIME to mIsSynthesizedByTIP as per your suggestion.

Comment 162

[Youngwoo Jo]

Attachment: Part2_IME_added_inputmethod_app_state_to_WidgetKeyboardEvent.patch

I changed the prefix names, KeyboardApp* to InputMethodApp as per masayuki's suggestion.

Comment 163

[Youngwoo Jo]

Attachment: Part3_IME_added_HardwareKeyHandler_component.patch

As per smaug's review, I removed |nsPresShell::DispatchKeyboardEvent| added by me. Instead of that, I implemented it directly in HardwareKeyHandler.
As a result, the previous part3 patch and part4 patch are merged into this patch.
And this patch contains the modifications as per masayuki's review in Comment 151.

Comment 164

[Youngwoo Jo]

Attachment: Part4_IME_communication_between_HardwareKeyHandler_and_inputmethod_app.patch

This patch replaces the previous Part5 patch which forwards a hardware keyboard event to the active input method app and receive a reply from the active input method app.

Comment 165

[Youngwoo Jo]

Attachment: Part5_IME_hooking_by_HardwareKeyHandler_in_shell.js.patch

This patch replaces the previous part6 patch which hooks a hardware keyboard event in shell.js. With this patch, HardwareKeyHandler is actually hooking a hardware keyboard event.

Comment 166

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Sorry for the delay to review this. I have some urgent bugs to fix ASAP and your patches are big. I'll be back ASAP.

Comment 167

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8672495 [details] [diff] [review]
Part3_IME_added_HardwareKeyHandler_component.patch

>+StaticRefPtr<HardwareKeyHandler> HardwareKeyHandler::sSingleton;

nit: I like sInstance since a lot of singleton classes use this name.

>+HardwareKeyHandler::HardwareKeyHandler()
>+  : mInputMethodAppConnected(false)
>+{
>+}
>+
>+HardwareKeyHandler::~HardwareKeyHandler()
>+{
>+  while (mEventQueue.GetSize()) {
>+    delete static_cast<WidgetKeyboardEvent*>(mEventQueue.PopFront());
>+  }
>+}
>+
>+NS_IMETHODIMP
>+HardwareKeyHandler::RegisterListener(nsIHardwareKeyEventListener* aListener)
>+{
>+  mHardwareKeyEventListener = aListener;
>+  return NS_OK;

When mHardwareKeyEventListener is not nullptr, the new key event listener will override it. I.e., the old listener won't receive hardware keyboard events.

Is this your intentional behavior?

If such case never occurs, it's okay. In this case, I'd like you to add:

if (NS_WARN_IF(mHardwareKeyEventListener && aListener)) {
  return NS_ERROR_NOT_AVAILABLE;
}

If this may be occurred and the old listener shouldn't need new events anymore, we should let the old listener know that a new listener steals new key events via nsIHardwareKeyEventListener.onHardwareKey() with additional argument. For example, make the method as:

nsIHardwareKeyEventListener.onNotify(DOMString aMessage, nsIDOMKeyEvent aEvent).

Then, we can notify the listener of other things with aEvent = nullptr.

>+NS_IMETHODIMP
>+HardwareKeyHandler::MaybeSendKeyToInputMethodApp(nsIDOMKeyEvent* aEvent,
>+                                                 bool* aResult)
>+{
>+  MOZ_ASSERT(mInputMethodAppConnected, "No InputMethod app is connected");
>+  MOZ_ASSERT(mHardwareKeyEventListener,
>+             "HardwareKeyEventListener should be set");
>+
>+  WidgetKeyboardEvent* originalEvent =
>+    aEvent->GetInternalNSEvent()->AsKeyboardEvent();
>+  MOZ_ASSERT(keyEvent,
>+             "DOM key event's internal event must be a WidgetKeyboardEvent");
>+
>+  if (!mInputMethodAppConnected || originalEvent->mIsSynthesizedByTIP ||
>+      originalEvent->mInputMethodAppState != WidgetKeyboardEvent::eNotHandled) {
>+    *aResult = false;
>+    return NS_OK;
>+  }

Hmm, these are unexpected cases, aren't them? If so, I'd like this to return NS_ERROR_NOT_AVAILABLE for !mInputMethodAppConnected case and NS_ERROR_INVALID_ARG for the other cases. (i.e., should cause exception) At least I feel the latter case are unexpected cases.

>+
>+  // Copy and store a keyboard event to reuse it when its reply arrives.
>+  nsAutoPtr<WidgetKeyboardEvent> copiedEvent(
>+    new WidgetKeyboardEvent(*originalEvent));
>+
>+  if (copiedEvent->mMessage == eKeyPress) {
>+    mEventQueue.Push(copiedEvent);
>+  } else {
>+    bool isSent = false;
>+    if (mHardwareKeyEventListener) {
>+      mHardwareKeyEventListener->OnHardwareKey(aEvent, &isSent);
>+    }
>+    if (!isSent) {
>+      *aResult = false;
>+      return NS_OK;
>+    }
>+    mEventQueue.Push(copiedEvent);
>+  }
>+
>+  // Set the flags to specify the sending phase.
>+  originalEvent->mInputMethodAppState = WidgetKeyboardEvent::eHandling;
>+
>+  copiedEvent.forget();
>+  *aResult = true;
>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+HardwareKeyHandler::OnInputMethodAppConnected()
>+{

Don't you need to check mInputMethodAppConnected here? Like:

if (NS_WARN_IF(mInputMethodAppConnected)) {
  return NS_ERROR_UNEXPECTED;
}

>+  mInputMethodAppConnected = true;
>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+HardwareKeyHandler::OnInputMethodAppDisconnected()
>+{

Similar, how about:

if (NS_WARN_IF(!mInputMethodAppConnected)) {
  return NS_ERROR_UNEXPECTED;
}

>+  mInputMethodAppConnected = false;
>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+HardwareKeyHandler::OnHandledByInputMethodApp(const nsAString& aEventType,
>+                                              bool aDefaultPrevented,
>+                                              bool aDefaultPreventedByContent,
>+                                              bool aDefaultPreventedByChrome)
>+{
>+  MOZ_ASSERT((aEventType.EqualsLiteral("keydown") ||
>+              aEventType.EqualsLiteral("keyup")),
>+             "Invalid event type");
>+
>+  // We can not handle this reply because the pending event had been already
>+  // removed from the sending queue before this reply arrives.
>+  if (mEventQueue.GetSize() == 0) {

nit: if (!mEventQueue.GetSize()) {

>+    return NS_OK;
>+  }
>+
>+  nsAutoPtr<WidgetKeyboardEvent> keyEvent(
>+    static_cast<WidgetKeyboardEvent*>(mEventQueue.PopFront()));
>+
>+  MOZ_ASSERT(((aEventType.EqualsLiteral("keydown") &&
>+               keyEvent->mMessage == eKeyDown) ||
>+              (aEventType.EqualsLiteral("keyup") &&
>+               keyEvent->mMessage == eKeyUp)),
>+             "Handled key event does not match with the pending event");

I'm afraid that isn't this is available a way to crash the process intentionally (even though only on debug build)?

I like you to use:

if (NS_WARN_IF(aEventType.EqualsLiteral("keydown") &&
                 keyEvent->mMessage != eKeyDown) ||
    NS_WARN_IF(aEventType.EqualsLiteral("keyup") &&
                 keyEvent->mMessage != eKeyUp)) {
  return NS_ERROR_INVALID_ARG;
}

>+void
>+HardwareKeyHandler::DispatchAfterKeyEvent(WidgetKeyboardEvent* aEvent)
>+{
>+  if (!PresShell::BeforeAfterKeyboardEventEnabled() ||
>+      aEvent->mMessage == eKeyPress) {
>+    return;
>+  }
>+
>+  nsCOMPtr<nsIContent> target = GetEventTarget();

I feel this is odd. If a keyboard event handler in the process has changed the focus, this is different from the target of normal key events. Is that okay? Looks like that PresShell doesn't refer new focused target at dispatching after key events.

>+  if(!target) {
>+    return;
>+  }
>+  nsCOMPtr<nsIPresShell> presShell = GetPresShell(target);

So, I think that presShell and target should be passed with arguments.

>+bool
>+HardwareKeyHandler::DispatchToTargetApp(WidgetKeyboardEvent* aEvent)
>+{
>+  MOZ_ASSERT(aEvent, "Event is null");
>+
>+  nsCOMPtr<nsIContent> target = GetEventTarget();
>+  if (!target) {
>+    return false;
>+  }
>+
>+  nsCOMPtr<nsIPresShell> presShell = GetPresShell(target);
>+  if (!presShell) {
>+    return false;
>+  }
>+
>+  // 1. Dispatch keydown or keyup to in-process
>+  DispatchToCurrentProcess(presShell, target, aEvent);

Dispatching an event may cause destroying the widget. So, please check if the widget is available and the presShell is still available. You should expose PresShell::CanDispatchEvent() and use it.

>+  if (IsTargetIframe(target)) { // If target app is OOP,
>+    // 1-1. If the default prevented, it stops dispatching keydown or keyup.
>+    //      And then it dispatches mozbrowserafterkey* to in-process.
>+    if (aEvent->mFlags.mDefaultPrevented) {
>+      DispatchAfterKeyEvent(aEvent);
>+      return true;
>+    }
>+
>+    // 2. Dispatch keydown or keyup to oop target.
>+    if (!DispatchToCrossProcess(target, aEvent)) {
>+      return false;
>+    }
>+  } else { // If target app is in-process
>+    // 2. Dispatch mozbrowserafterkey* to in-process target.
>+    DispatchAfterKeyEvent(aEvent);
>+  }
>+  return true;
>+}

>+bool
>+HardwareKeyHandler::IsTargetIframe(nsIContent* aTarget)
>+{
>+  return aTarget && aTarget->IsHTMLElement(nsGkAtoms::iframe);

Don't you need to check additional attributes if the iframe is actually a target? (I'm not sure, though)

>+/* static */ already_AddRefed<HardwareKeyHandler>
>+HardwareKeyHandler::GetInstance()
>+{
>+  if (!XRE_IsParentProcess()) {
>+    return nullptr;
>+  }
>+
>+  if (!sSingleton) {
>+    sSingleton = new HardwareKeyHandler();
>+    ClearOnShutdown(&sSingleton);
>+  }
>+
>+  nsRefPtr<HardwareKeyHandler> service = sSingleton.get();

FIY: nsRefPtr is renamed to RefPtr

>+class HardwareKeyHandler : public nsIHardwareKeyHandler
>+{
>+  public:
>+    HardwareKeyHandler();
>+
>+    NS_DECL_ISUPPORTS
>+    NS_DECL_NSIHARDWAREKEYHANDLER
>+
>+    static already_AddRefed<HardwareKeyHandler> GetInstance();
>+
>+  private:
>+    virtual ~HardwareKeyHandler();
>+
>+    void DispatchAfterKeyEvent(WidgetKeyboardEvent* aEvent);
>+    bool DispatchToTargetApp(WidgetKeyboardEvent* aEvent);
>+    void DispatchToCurrentProcess(nsIPresShell* presShell,
>+                                  nsIContent* aTarget,
>+                                  WidgetKeyboardEvent* aEvent);
>+    bool DispatchToCrossProcess(nsINode* aTarget, WidgetKeyboardEvent* aEvent);
>+
>+    bool IsTargetIframe(nsIContent* aTarget);
>+    already_AddRefed<nsIContent> GetEventTarget();
>+    already_AddRefed<nsIPresShell> GetPresShell(nsINode* aTarget);
>+
>+    nsDeque mEventQueue;
>+    nsCOMPtr<nsIHardwareKeyEventListener> mHardwareKeyEventListener;
>+    bool mInputMethodAppConnected;
>+
>+    static StaticRefPtr<HardwareKeyHandler> sSingleton;

You can reduce 2 white spaces of each line. The scope should be started at start of each line.

>+/**
>+ * This interface should be registered to nsIHardwareKeyHandler through
>+ * |nsIHardwareKeyHandler.registerListener|. When nsIHardwareKeyHandler needs
>+ * to forward a hardware keyboard event to the active input method, this
>+ * listener is invoked to communicate with the active input method. Therefor,

Therefore?


>+  /**
>+   * Reply paired with |maybeSendKeyToInputMethodApp| from the active input
>+   * method. When the processing in content finishes, the result is notified
>+   * through this interface. Therefor, this interface should be called by a

Therefore?

>+   * module having the communication channel with the active input method.
>+   * Probably it might be a listener registered through |registerListener|.
>+   *
>+   * The result should contain the original event type and the info whether
>+   * the default is prevented or not. So the below parameters should be passed.
>+   *
>+   * @param aEventType                 The type of an original event.
>+   * @param aDefaultPrevented          |evt.preventDefault| is called or not
>+   * @param aDefaultPreventedByContent True, if prevented by input method app.

This sounds odd. defaultPreventedByContent means that it's consumed by web apps. For example, if a keydown event shouldn't be blocked by content, mDefaultPrevented will be ignored. E.g., Ctrl+Tab on Windows and Linux.

>+   * @param aDefaultPreventedByChrome  True, if prevented by chrome.
>+   */
>+  void onHandledByInputMethodApp(in DOMString aEventType,
>+                                 in bool aDefaultPrevented,
>+                                 in bool aDefaultPreventedByContent,
>+                                 in bool aDefaultPreventedByChrome);

3 bool arguments are ugly... e.g., |true, true false| isn't clear for other users. Although, I know you use this method clearer than that since you specify variables whose name indicates them. However, I'd like you to use flags like:

NOT_DEFAULT_PREVENTED        = 0x00;
DEFAULT_PREVENTED            = 0x01;
DEFAULT_PREVENTED_BY_CONTENT = 0x02;
DEFAULT_PREVENTED_BY_CHROME  = 0x04;

>+  /**
>+   * Notifies to nsIHardwareKeyHandler that the communication channel between

Notifies nsIHardwareKeyHandler of the communcation channel...

>+   * b2g process and the active input method app has been estabilshed. This
>+   * notification is required to check if sending a hardware keyboard event to
>+   * the active keyboard app is needed or not.
>+   */
>+  void onInputMethodAppConnected();
>+
>+  /**
>+   * Notifies to nsIHardwareKeyHandler that the communication channel between

Notifies nsIHardwareKeyHandler of the communication channel...

>+  /**
>+   * Reset the flags related to event flow control for redispatch
>+   */
>+  void ResetForReuse() {

nit: |{| should be below the 'v' of void. (i.e., put it to the start of next line.

Comment 168

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8672501 [details] [diff] [review]
Part4_IME_communication_between_HardwareKeyHandler_and_inputmethod_app.patch

>+  bool DefaultPreventedByContent() const
>+  {
>+    return mEvent->mFlags.mDefaultPreventedByContent;

I think that should check mEvent->mFlags.mDefaultPrevented too.

>+  }
>+
>+  bool DefaultPreventedByChrome() const
>+  {
>+    return mEvent->mFlags.mDefaultPreventedByChrome;
>+  }

Ditto.

>+  onHardwareKey: function onHardwareKeyReceived(event) {
>+    let keyboardEventDict = {
>+      // dictionary KeyboardEvetInit
>+      key: event.key,
>+      code: event.code,
>+      location: event.location,
>+      repeat: event.repeat,
>+      isComposing: event.isComposing,
>+      charCode: event.charCode,
>+      keyCode: event.keyCode,
>+      which: event.which,
>+      // dictionary EventModifierInit
>+      ctrlKey: event.getModifierState('Control'),
>+      shiftKey: event.getModifierState('Shift'),
>+      altKey: event.getModifierState('Alt'),
>+      metaKey: event.getModifierState('Meta'),
>+      modifierAltGraph: event.getModifierState('AltGraph'),
>+      modifierCapsLock: event.getModifierState('CapsLock'),
>+      modifierFn: event.getModifierState('Fn'),
>+      modifierFnLock: event.getModifierState('FnLock'),
>+      modifierNumLock: event.getModifierState('NumLock'),
>+      modifierOS: event.getModifierState('OS'),
>+      modifierScrollLock: event.getModifierState('ScrollLock'),
>+      modifierSymbol: event.getModifierState('Symbol'),
>+      modifierSymbolLock: event.getModifierState('SymbolLock'),

Hmm, this is not good for maintenance. I guess that here is always run as chrome, right? If so, we can use chrome only method. Then, cannot we create dictionary object from dom/KeyboardEvent.cpp? If it's possible, we can support new modifier flags automatically if it's coded with macro and all members of dictionary is defined in EventForwards.h.

>+      case 'Keyboard:ReplyHardwareKeyEvent':
>+        hardwareKeyHandler
>+        .onHandledByInputMethodApp(msg.data.eventType,
>+                                   msg.data.defaultPrevented,
>+                                   msg.data.defaultPreventedByContent,
>+                                   msg.data.defaultPreventedByChrome);

So, I'm happy if you'd use flags as I mentioned at reviewing the previous patch.

>+    // Notify the communication channel with the active input method app has
>+    // been destroyed.
>+    hardwareKeyHandler.onInputMethodAppDisconnected();

I wonder is this enough? If this object is deleted by GC or something, blur event handler might not run before that...

>           this._inputcontext.updateSelectionContext(data, false);
>         }
>         break;
>       case 'Keyboard:GetContext:Result:OK':
>         this.setInputContext(data);
>         break;
>       case 'Keyboard:SupportsSwitchingTypesChange':
>         this._supportsSwitchingTypes = data.types;
>         break;
>+      case 'Keyboard:SendHardwareKeyEvent':
>+        let res;
>+        if (this.inputcontext) {
>+          res = this._inputcontext.dispatchKeyboardEvent(data);
>+        }
>+        cpmmSendAsyncMessageWithKbID(this, 'Keyboard:ReplyHardwareKeyEvent', {
>+          eventType: data.eventType,
>+          defaultPrevented: !!(res && res.defaultPrevented),
>+          defaultPreventedByContent: !!(res && res.defaultPreventedByContent),
>+          defaultPreventedByChrome: !!(res && res.defaultPreventedByChrome)

So, I'm happy if you use flags for representing this.


I'm really not familiar with around this module, so, if there is somebody better to review here, I'd like you to ask review additionally to hem/her after my next review.

Comment 169

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8672502 [details] [diff] [review]
Part5_IME_hooking_by_HardwareKeyHandler_in_shell.js.patch

>@@ -621,35 +621,41 @@ nsEditorEventListener::KeyPress(nsIDOMKeyEvent* aKeyEvent)
>   // If the client pass cancelled the event, defaultPrevented will be true
>   // below.
> 
>   bool defaultPrevented;
>   aKeyEvent->GetDefaultPrevented(&defaultPrevented);
>   if (defaultPrevented) {
>     return NS_OK;
>   }
> 
>+  WidgetKeyboardEvent* keyEvent =
>+    aKeyEvent->GetInternalNSEvent()->AsKeyboardEvent();
>+  MOZ_ASSERT(keyEvent,
>+             "DOM key event's internal event must be WidgetKeyboardEvent");
>+  // We have to ignore the event being handled by input method app.
>+  // If it's not ignored, the duplicated keypresses will be handled.
>+  if (keyEvent->mInputMethodAppState == WidgetKeyboardEvent::eHandling) {
>+    return NS_OK;
>+  }

Oh, this is too bad. If somebody handles keypress event before editor, they also need to check this state. Hmm, cannot we stop the propagation at EventDispatcher if the content shouldn't handle the keypress event?

Comment 170

[Youngwoo Jo]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #169)
> Comment on attachment 8672502 [details] [diff] [review]
> Part5_IME_hooking_by_HardwareKeyHandler_in_shell.js.patch
> 
> Oh, this is too bad. If somebody handles keypress event before editor, they
> also need to check this state. Hmm, cannot we stop the propagation at
> EventDispatcher if the content shouldn't handle the keypress event?

I'm sorry for the late reply.
I reviewed all my patches again. However, I could not find a good solution for your above comment. And I had thought about the more problems.

We dispatches the event to shell.js and it decides to forward the event to input method app or not.
The above dispatch to shell.js makes the problems as the below.
(1) EventDispatcher can not prevent the event handling in system group event handlers.
(2) EventDispatcher will call PreHandleEvent() of all target event chain elements.
(3) EventDispatcher will call PostHandleEvent() of all target event chain elements.
(4) We can not ensure that our new event handler in shell.js is the first event handler of shell.js.
    i.e. What if the handler prior to our new handler consumes the hardware keyboard event?

The above problems are caused by our calling only evt.stopPropagation() in shell.js and by our calling EventDispatcher::Dispatch() to target element.

If then, how about calling also evt.preventDefault() in shell.js? It will resolve only (1) problem. However, it still has the several differnt problems including the above problems.
(1) System dependent keypress can not be generated from nsAppShell since the default prevented.
(2) nsAppShell's modification will require the modifications in all platforms(window, gtk, cocoa,..).
(3) And mozbrowserafterkey* events will be generated even though the event processing does not finish.
(4) When nsHardwareKeyHandler dispatches the event to target app after the input method app handles the event, the event will be regarded as a totally new event. Is that OK?

To fix the all above problems, they need the many hacky codes for each case. I could not find out the good solution without the hacky codes.

So, I suggest that the code to forward the keyboard event to input method app should be moved prior to calling EventDispatcher::Dispatch(target).
New location to forward the event to input method app,
 - https://dxr.mozilla.org/mozilla-central/source/layout/base/nsPresShell.cpp#6933

However, this change will make that input method app become the first event handler prior to shell.html and system app. This is commented already in https://wiki.mozilla.org/WebAPI/InputMethod_API_with_hardware_keyboard#Limitation. According to the above post, we have to trust the built-in keyboard app works correctly and it assumes that system supports mozborwserbefore/afterkey* events. So I think this patch should be binded with the feature preference of mozborwserbefore/afterkey*.

Masayuki, how about this change?

Comment 171

[C.M.Chang[:chunmin]]

(In reply to Youngwoo Jo from comment #170)
Hi, Youngwoo, 
I was just wondering if these problems could be solved by original architecture used in attachment 8641387 [details] [diff] [review](comment #90).

Comment 172

[C.M.Chang[:chunmin]]

Summary of above
-------------------------
- mozbrowserafterkeyXX/mozbrowserbeforekeyXXX:
  only mozBrowser-embedder (and the target is a moz iframe embedded by it) can receive them(comment #31)
- Using "state" to avoid the infinite loop(comment #50, comment #53, comment #57)
- Event queue: It's used for synchronous.
  The events fired from the same source(e.g. physical keyboards) should be in order[1,2].
- Keypress: It's used for decoding "keycode"(comment #111).
  - If we call |evt.preventDefault()| after receiving keydown,
  then keypress will NOT be generated from nsAppShell[3](comment #98).
  - However, the "keycode" is a legacy attribute. Can't we just ignore it?
- The |mozbrowserafterkeyXX| or |mozbrowserbeforekeyXXX| can be fired once
  IME(keyboard app) call |preventDefault()|.
  It dosen't need to wait for the response from IME(keyboard app) because
  the keyboard event generated from IME(keyboard app) is another "independent" event.

1. http://www.w3.org/TR/DOM-Level-3-Events/#glossary-event-order
2. http://www.w3.org/TR/DOM-Level-3-Events/#sync-async
3. https://dxr.mozilla.org/mozilla-central/source/widget/gonk/nsAppShell.cpp#346

Comment 173

[C.M.Chang[:chunmin]]

Comment on attachment 8670129 [details] [diff] [review]
[WIP] test cases

Review of attachment 8670129 [details] [diff] [review]:
-----------------------------------------------------------------

::: tests/python/gaia-ui-tests/gaiatest/apps/hw_keyevent/DeviceEvent/deviceEventInputer.py
@@ +366,5 @@
> +            keyupEvent = keyup.toSendEvtCmdList()
> +
> +            cmdList = keydownEvent + [';'] + synEvent + [';'] + keyupEvent + [';'] + synEvent
> +            # print cmdList
> +            self.dmADB.shellCheckOutput(cmdList)

Since we move tests from GIP to GIJ, I'd like to replace this tests. However, I can't find what is the equivalent GIJ api to the GIP api "mozdevice.DeviceManagerADB().shellCheckOutput(...)".

Comment 174

[C.M.Chang[:chunmin]]

Hi, Ghislain,
I was wondering if you could tell me what is the equivalent GIJ api to the GIP api "mozdevice.DeviceManagerADB().shellCheckOutput(...)".

Comment 175

[C.M.Chang[:chunmin]]

It's no longer as a part of red-tai project. It's included in TV project now.

Comment 176

[Aus Lacroix [:aus]]

(In reply to Chun-Min Chang[:chunmin] from comment #174)
> Hi, Ghislain,
> I was wondering if you could tell me what is the equivalent GIJ api to the
> GIP api "mozdevice.DeviceManagerADB().shellCheckOutput(...)".

Unfortunately we do not have direct support for this at this time. It will, however, be part of the fxos-device-service which will replace the current bits we use to startup Mulet / Devices when executing tests.

In the meantime, I believe you can call adb shell yourself directly using nodejs apis for child processes. It would probably look something like this:

var spawnSync = require('child_process').spawnSync;

...

var childProcess = spawnSync('adb', ['shell', '_YOUR_SHELL_COMMAND_']);

childProcess will be one of these objects as described here -- https://nodejs.org/dist/latest-v4.x/docs/api/child_process.html#child_process_child_process_spawnsync_command_args_options

Hope that helps :)

Comment 177

[C.M.Chang[:chunmin]]

Three use cases for this bug
------------------------------------
1) If keyboard app isn’t active
2) If keyboard app is active and keyboard IME want to handle the keyboard event
3) If keyboard app is active but keyboard IME doesn’t want to handle the keyboard event

In our current architecture, we have several possible code flows:
case 2 >> 
|event.preventDefault()| and |event.stopImmediatePropagation()| should be called by shell.js or keyboard IME to stop dispatching event, then IME will generate a new keyboard event with IME's values.

case 3 >> we have 2 possible ways:
- |event.preventDefault()| and |event.stopImmediatePropagation()| won't be called, and the events will be dispatched as original once IME notify shell.js that it doesn't want to handle this event.
- |event.preventDefault()| and |event.stopImmediatePropagation()| still be called, but IME then will generate a new keyboard with original values.

If we take approach "a", then we need to use a synchronous call from shell.js to Keyboard.jsm to get to know whether or not the IME app want to handle the keyboard events.

If we take approach "b", then we can always use an asynchronous from shell.js to Keyboard.jsm. IME app always can generate a new keyboard events with appropriate values, depending on whether or not IME app want to handle the keyboard events. If IME want to handle the key, then IME can generate a new keyboard events with IME's values. If IME doesn't want to handle the key, then IME can just generate a new keyboard events with all original values.

However, these two approaches will fire the |mozbrowserafterkey| in different time. 
- approach "a"
If IME is active, |mozbrowserafterkey| will be fired after receiving result from IME
("result" = IME want to handle the key or not)
- approach "b"
If IME is active, |mozbrowserafterkey| will be fired upon |event.preventDefault()| is called.

Hi Masayuki,
I was wondering if you could share your opinions about this.

Comment 178

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #177)
It occurred to me that one case might be affected if we fire |mozbrowserafterkey| in different time.
In APP-First scenario[0]:
  One key can be handled by app first. If app doesn't want to handle the key, 
  system will handle this key. To achieve this scenario, system listen the |mozbrowserafterkey|
  to handle the key. If we take approach "b", |mozbrowserafterkey| will always be fired 
  when IME is active, upon |event.preventDefault()| is called. 
  Therefore, system might able to handle the key before target app handle it.


The code flow for these two different approaches
-----------------------------------------------------
approach "a":
1) IME is active. shell.js asked whether or not IME want to handle this key. 
   The event dispatch will suspend to wait the response from IME.
2) If IME doesn't want to handle this key, |event.preventDefault()| won't be called. 
   The event dispatch will follow the original path.
   ---> event will be dispatched to target app
   ------> If target app want to handle this key, it can handle the key at this time.
3) The timing of |mozbrowserafterkey| will be fired as original
   ---> System can check whether the key is handled by target app or not
   ------> If target app doesn't handle the key yet, system can handle the key at this time.

approach "b":
1) IME is active, so shell.js will call |event.preventDefault()| 
   no matter whether IME want to handle this key or not
2) |mozbrowserafterkey| will be fired once |event.preventDefault()| is called
   ---> The target app doesn't handle this key yet, but system might be able to handle this key
        before target app!
3) If IME doesn't want to handle this key, IME will generate a new keyboard event 
   with same values as original one.
   ---> The new keyboard event is generated by nsITextInputProcessor, and this synthesized event
        won't fire any |mozbrowserafterkey| or |mozbrowserbeforekey|.
        Therefore, there is no chance for system to handle the key *after* target app handle it.

However, if APP-First scenario doesn't need to exist when IME is active, then approach "b" can be an available solution.


[0] https://wiki.mozilla.org/WebAPI/BrowserAPI/KeyboardEvent#Dispatch_KeyboardEvent_across_BrowserElements

Comment 179

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #178)
Hi, Tim,
May I ask your suggestions about this issue?

Comment 180

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Chun-Min Chang[:chunmin] from comment #177)
> case 3 >> we have 2 possible ways:
> - |event.preventDefault()| and |event.stopImmediatePropagation()| won't be
> called, and the events will be dispatched as original once IME notify
> shell.js that it doesn't want to handle this event.
> - |event.preventDefault()| and |event.stopImmediatePropagation()| still be
> called, but IME then will generate a new keyboard with original values.
> 
> If we take approach "a", then we need to use a synchronous call from
> shell.js to Keyboard.jsm to get to know whether or not the IME app want to
> handle the keyboard events.
> 
> If we take approach "b", then we can always use an asynchronous from
> shell.js to Keyboard.jsm. IME app always can generate a new keyboard events
> with appropriate values, depending on whether or not IME app want to handle
> the keyboard events. If IME want to handle the key, then IME can generate a
> new keyboard events with IME's values. If IME doesn't want to handle the
> key, then IME can just generate a new keyboard events with all original
> values.
> 
> However, these two approaches will fire the |mozbrowserafterkey| in
> different time. 
> - approach "a"
> If IME is active, |mozbrowserafterkey| will be fired after receiving result
> from IME
> ("result" = IME want to handle the key or not)
> - approach "b"
> If IME is active, |mozbrowserafterkey| will be fired upon
> |event.preventDefault()| is called.
> 

This is a hard question. And it's possible we have cornered ourselves to this due to bad API design/planning (that does not consider the async nature of the cross-process communications). We should discuss this in person and maybe find an alternative (plz. grab Luke too).

Comment 181

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Well, I'm still being confused, though...

> However, if APP-First scenario doesn't need to exist when IME is active, then approach "b" can be
> an available solution.

Note that on desktop OSes, IME handles (native) key events *before* Gecko and some key inputs which are registered by system, they may not be notified to applications (e.g., Win+Tab key on Windows).

So, I think that this order isn't problem:

1. handlable system first
2. handlable IME next
3. handlable app/content
4. handlable system finally

With approach A and B, how this order will be changed?

Comment 182

[C.M.Chang[:chunmin]]

Attachment: [WIP] forward hardware keyboard event via shell.js

Comment 183

[C.M.Chang[:chunmin]]

Attachment: [WIP] Convert hardware keyboard events to IME's keyboard events

Comment 184

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) (offline until 1/6) from comment #181)
> So, I think that this order isn't problem:
> 
> 1. handlable system first
> 2. handlable IME next
> 3. handlable app/content
> 4. handlable system finally
> 
> With approach A and B, how this order will be changed?

With approach A and B, the order of |keydown| or |keyup| won't be changed. 
However, the |afterkeydown| or |afterkeyup| will be changed. 
Let's use |keydown| for example.

Usecase: IME is active, but IME doesn't want to handle this key

approach A
-----------------
1) system receive 'beforekeydown' 
2) system receive 'keydown' fired from hardware
   --> we will ask IME want to handle or not here. 
   -----> wait for IME's response:
          IME doesn't want to handle this key, so we will dispatch key event as original.
3) app(the event target) receive 'keydown' fired from hardware
4) system receive 'afterkeydown'


approach B
-----------------
1) system receive 'beforekeydown'
2) system receive 'keydown' fired from hardware
   --> we don't need to ask IME here.
       we will forward the key event's information to IME if it's active.
   -----> system will call event.preventDefault() directly 
          because we assume IME will handle it.
3) system receive 'afterkeydown' 
   ('afterkeydown' will be fired upon system call event.preventDefault())
4) app(the event target) receive 'keydown' generated by IME
   --> IME will generate a same key event as received one if IME doesn't want to handle this key.

Comment 185

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #184)
To make sure that the 'afterkeydown' is always fired after 'keydown', it seems that approach A is better than B. Nevertheless, even we take approach A, there are some issues needed to be concerned.

# forward key event via shell.js
--------------------------------------
To ask whether or not the IME want to handle the key, system need to send a request to IME and wait for the response. There are two ways:

1. If the request is asynchronous, then we won't forward key event to IME until receiving a message indicating that IME will handle this key(that is, we will dispatch key event directly to it's target app before IME tell system that it want to handle the key). Suppose that 'A', 'B' on physical keyboard will be converted respectively to 'ち' and 'こ', then we will expect the result is 'ちちここ' if we type 'AABB' on hardware keyboard. However, the result might be 'AちBこ' because system don't receive the response from IME yet when the first 'A' and first 'B' are pressed.

2. If the request is synchronous, then the above situation can be solved. System will wait for the response, and decide how to dispatch the key event depending on the response. However, this means that IME(content process) will block system(chrome process). That's not good.

To keep request asynchronous(content process shouldn't block chrome process) but the result won't be 'AちBこ', we can intercept keyboard event in nsPresShell instead of shell.js. 

# forward key event in nsPresShell
--------------------------------------
We actually can determine how to dispatch the key event before it's dispatched to system. We can intercept keyboard event and use asynchronous IPC to ask IME whether or not it want to handle the key(if IME call event.preventDefault(), it means that IME already handle the key). After receiving the feedback from IME, the key event routing path can be set. That is, we might need to back to old method, which will be similar to those reviewed patches.

Comment 186

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #185)
> # forward key event in nsPresShell
> --------------------------------------
The main concern of this approach is performance. In usecase: IME is active, but IME doesn't want to handle this key, we need to use one async IPC round trip to know that IME won't handle it. One simple method is to use a blacklist-cache to remember which key doesn't need to be send to IME. Once system get refuse-to-handle of key 'X' from IME, then we can add key 'X' into blacklist-cache. After then, if key is in this blacklist-cache, then we can dispatch the key to its target directly.

On the other handle, whitelist is an alternative solution. If we could get a whitelist that contains all the key needed to be handled by IME, then we have no need to wait for the result from IME. However, every input-method has its own unique layout. It means that every input-method has its own unique whitelist. It will be a big work to build all whitelist we need.

Comment 187

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Chun-Min Chang[:chunmin] from comment #186)
> (In reply to Chun-Min Chang[:chunmin] from comment #185)
> > # forward key event in nsPresShell
> > --------------------------------------
> The main concern of this approach is performance. In usecase: IME is active,
> but IME doesn't want to handle this key, we need to use one async IPC round
> trip to know that IME won't handle it. One simple method is to use a
> blacklist-cache to remember which key doesn't need to be send to IME. Once
> system get refuse-to-handle of key 'X' from IME, then we can add key 'X'
> into blacklist-cache. After then, if key is in this blacklist-cache, then we
> can dispatch the key to its target directly.

That unfortunately won't work; IMEs might consider whether or not to consume a key based on it's own current state, e.g. having a candidate menu shown, has pending compositions etc.

> On the other handle, whitelist is an alternative solution. If we could get a
> whitelist that contains all the key needed to be handled by IME, then we
> have no need to wait for the result from IME. However, every input-method
> has its own unique layout. It means that every input-method has its own
> unique whitelist. It will be a big work to build all whitelist we need.

That's what we talked about last week; my only concern is that this API is quite unique compare to other text input frameworks/services in other OSes thus sounds like a workaround to our IPC-caused latency. I would rather get the interfaces right and revert the architectures, if possible; but that's separate conversation though.

Comment 188

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Chun-Min Chang[:chunmin] from comment #184)
> (In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) (offline until 1/6)
> from comment #181)
> > So, I think that this order isn't problem:
> > 
> > 1. handlable system first
> > 2. handlable IME next
> > 3. handlable app/content
> > 4. handlable system finally
> > 
> > With approach A and B, how this order will be changed?
> 
> With approach A and B, the order of |keydown| or |keyup| won't be changed. 
> However, the |afterkeydown| or |afterkeyup| will be changed. 
> Let's use |keydown| for example.
> 
> Usecase: IME is active, but IME doesn't want to handle this key
> 
> approach A
> -----------------
> 1) system receive 'beforekeydown' 
> 2) system receive 'keydown' fired from hardware
>    --> we will ask IME want to handle or not here. 
>    -----> wait for IME's response:
>           IME doesn't want to handle this key, so we will dispatch key event
> as original.
> 3) app(the event target) receive 'keydown' fired from hardware
> 4) system receive 'afterkeydown'
> 
> 
> approach B
> -----------------
> 1) system receive 'beforekeydown'
> 2) system receive 'keydown' fired from hardware
>    --> we don't need to ask IME here.
>        we will forward the key event's information to IME if it's active.
>    -----> system will call event.preventDefault() directly 
>           because we assume IME will handle it.
> 3) system receive 'afterkeydown' 
>    ('afterkeydown' will be fired upon system call event.preventDefault())
> 4) app(the event target) receive 'keydown' generated by IME
>    --> IME will generate a same key event as received one if IME doesn't
> want to handle this key.

Sounds like A is really better.

(In reply to Chun-Min Chang[:chunmin] from comment #185)
> # forward key event via shell.js
> --------------------------------------
> To ask whether or not the IME want to handle the key, system need to send a
> request to IME and wait for the response. There are two ways:
> 
> 1. If the request is asynchronous, then we won't forward key event to IME
> until receiving a message indicating that IME will handle this key(that is,
> we will dispatch key event directly to it's target app before IME tell
> system that it want to handle the key). Suppose that 'A', 'B' on physical
> keyboard will be converted respectively to 'ち' and 'こ', then we will expect
> the result is 'ちちここ' if we type 'AABB' on hardware keyboard. However, the
> result might be 'AちBこ' because system don't receive the response from IME
> yet when the first 'A' and first 'B' are pressed.
> 
> 2. If the request is synchronous, then the above situation can be solved.
> System will wait for the response, and decide how to dispatch the key event
> depending on the response. However, this means that IME(content process)
> will block system(chrome process). That's not good.
> 
> To keep request asynchronous(content process shouldn't block chrome process)
> but the result won't be 'AちBこ', we can intercept keyboard event in
> nsPresShell instead of shell.js.

Yeah, we shouldn't take synchronous IPC approach. We need to send physical keyboard events to another process asynchronously and wait the result from IPC.
 
> # forward key event in nsPresShell
> --------------------------------------
> We actually can determine how to dispatch the key event before it's
> dispatched to system. We can intercept keyboard event and use asynchronous
> IPC to ask IME whether or not it want to handle the key(if IME call
> event.preventDefault(), it means that IME already handle the key). After
> receiving the feedback from IME, the key event routing path can be set. That
> is, we might need to back to old method, which will be similar to those
> reviewed patches.

Yeah, it must be reasonable you to update the patch.

(In reply to Chun-Min Chang[:chunmin] from comment #186)
> The main concern of this approach is performance. In usecase: IME is active,
> but IME doesn't want to handle this key, we need to use one async IPC round
> trip to know that IME won't handle it.

I don't worry about the performance because other OSes must do same thing, i.e., sending key events from system process or something to focused process. If system (desktop or something) needs to intercept some key combination like Alt+Tab on Windows, it consumes key events and focused process won't receive such key events.

> One simple method is to use a
> blacklist-cache to remember which key doesn't need to be send to IME. Once
> system get refuse-to-handle of key 'X' from IME, then we can add key 'X'
> into blacklist-cache. After then, if key is in this blacklist-cache, then we
> can dispatch the key to its target directly.
>
> On the other handle, whitelist is an alternative solution. If we could get a
> whitelist that contains all the key needed to be handled by IME, then we
> have no need to wait for the result from IME. However, every input-method
> has its own unique layout. It means that every input-method has its own
> unique whitelist. It will be a big work to build all whitelist we need.

Yeah, such API becomes complicated and breaks compatibility with existing IMEs. So, I don't think that you need to worry about optimizing the keyboard event's performance at least now.

(In reply to Tim Guan-tin Chien [:timdream] (please needinfo) from comment #187)
> (In reply to Chun-Min Chang[:chunmin] from comment #186)
> > (In reply to Chun-Min Chang[:chunmin] from comment #185)
> > > # forward key event in nsPresShell
> > > --------------------------------------
> > The main concern of this approach is performance. In usecase: IME is active,
> > but IME doesn't want to handle this key, we need to use one async IPC round
> > trip to know that IME won't handle it. One simple method is to use a
> > blacklist-cache to remember which key doesn't need to be send to IME. Once
> > system get refuse-to-handle of key 'X' from IME, then we can add key 'X'
> > into blacklist-cache. After then, if key is in this blacklist-cache, then we
> > can dispatch the key to its target directly.
> 
> That unfortunately won't work; IMEs might consider whether or not to consume
> a key based on it's own current state, e.g. having a candidate menu shown,
> has pending compositions etc.

Absolutely.

> > On the other handle, whitelist is an alternative solution. If we could get a
> > whitelist that contains all the key needed to be handled by IME, then we
> > have no need to wait for the result from IME. However, every input-method
> > has its own unique layout. It means that every input-method has its own
> > unique whitelist. It will be a big work to build all whitelist we need.
> 
> That's what we talked about last week; my only concern is that this API is
> quite unique compare to other text input frameworks/services in other OSes
> thus sounds like a workaround to our IPC-caused latency. I would rather get
> the interfaces right and revert the architectures, if possible; but that's
> separate conversation though.

As I mentioned above, I don't worry about the performance since other OSes must do same thing (i.e., sending key events across process boundary). Or, is our IPC really slow than other OSes?

Comment 189

[C.M.Chang[:chunmin]]

Attachment: [WIP] Simulate hardware keystroke in mochitest

Comment 190

[C.M.Chang[:chunmin]]

Comment on attachment 8707271 [details] [diff] [review]
[WIP] Simulate hardware keystroke in mochitest

Review of attachment 8707271 [details] [diff] [review]:
-----------------------------------------------------------------

Hi Masayuki-san,
We need to use a new method to simulate native keyboard event after adding a flag |IsSynthesizedByTIP| in |KeyboardEvent|. The |mozbowserbeforekeyXX| and |mozbowserafterkeyXX| won't be fired when |KeyboardEvent.IsSynthesizedByTIP| is true, so some old tests like |test_dom_before_after_keyboard_event_remote.html| should not use |synthesizeKey| to send key event.

To simulate hardware keyboard event like real user's input, I use a shellscript to generate it and launch this shellscript via nsIFile and nsIProcess. I was wondering if you could share your opinions about this method.

Comment 191

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #190)
> To simulate hardware keyboard event like real user's input, I use a
> shellscript to generate it and launch this shellscript via nsIFile and
> nsIProcess. I was wondering if you could share your opinions about this
> method.
To simulate hardware key input on Linux, we might need third-party's library(e.g., xdotool[0], python-uinput[1]) or write one by ourselves.

For Android, we can simply generate any native hardware events by |sendevent| via adb shell

[0] https://github.com/jordansissel/xdotool
[1] http://tjjr.fi/sw/python-uinput/

Comment 192

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #190)
> Comment on attachment 8707271 [details] [diff] [review]
> [WIP] Simulate hardware keystroke in mochitest
> 
> To simulate hardware keyboard event like real user's input, I use a
> shellscript to generate it and launch this shellscript via nsIFile and
> nsIProcess.

(In reply to Chun-Min Chang[:chunmin] from comment #191)
> To simulate hardware key input on Linux, we might need third-party's
> library(e.g., xdotool[0], python-uinput[1]) or write one by ourselves.
> 
> For Android, we can simply generate any native hardware events by
> |sendevent| via adb shell
> 
> [0] https://github.com/jordansissel/xdotool
> [1] http://tjjr.fi/sw/python-uinput/

Hi Andrew,
Could you give me some suggestions about this approach?

Comment 193

[Andrew Halberstadt [:ahal]]

I'm don't think I can provide much input here. I looked into this for mochitest about 5 years ago, but we abandoned the project after realizing it would be too much work. Maybe things have changed since then though.

I know that webdriver (the backend to selenium) has a native events module that works with Firefox. Maybe it would be possible to re-use their implementation with Gaia. Here's the best article I could find on it, though it hasn't been updated since 2013:
https://code.google.com/p/selenium/wiki/NativeEventsOnLinux

Comment 194

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Sorry for the delay. I'll try to check the patch during business trip from tomorrow.

Comment 195

[C.M.Chang[:chunmin]]

(In reply to Andrew Halberstadt [:ahal] from comment #193)
Thank you for your information. 
Well, I'd like to know how to make a bash-script-file executable on try-server?
I might need to use command "$ chmod +x filename" on our testing environment to make this script executable.
Is it possible to package this bash-script-file into our testing environment?

Comment 196

[Andrew Halberstadt [:ahal]]

Yeah you could either bake it into the docker image, or have the test machines download it from hg.mozilla.org. See either one of the images in testing/docker or testing/taskcluster/scripts for examples of other jobs that do this.

Comment 197

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Chun-Min Chang[:chunmin] from comment #190)
> Comment on attachment 8707271 [details] [diff] [review]
> [WIP] Simulate hardware keystroke in mochitest
> 
> Review of attachment 8707271 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Hi Masayuki-san,
> We need to use a new method to simulate native keyboard event after adding a
> flag |IsSynthesizedByTIP| in |KeyboardEvent|. The |mozbowserbeforekeyXX| and
> |mozbowserafterkeyXX| won't be fired when |KeyboardEvent.IsSynthesizedByTIP|
> is true, so some old tests like
> |test_dom_before_after_keyboard_event_remote.html| should not use
> |synthesizeKey| to send key event.

If the input transaction of TIP is for tests, shouldn't before and after key events be fired?
http://mxr.mozilla.org/mozilla-central/source/widget/TextEventDispatcher.cpp?rev=63101b99fa95&mark=136-136#131

Comment 198

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #197)
> http://mxr.mozilla.org/mozilla-central/source/widget/TextEventDispatcher.
> cpp?rev=63101b99fa95&mark=136-136#131
Hi Masayuki-san,
I may not understand your point right. Do you mean that we can route the event dispatching path depends on |event.mFlags.mIsSynthesizedForTests| in our code? I personally think the testing part should be split from the executing code. Instead of adding a testing-specific flag into code, I prefer to find a way to simulate the hardware input event. If I misunderstand what you mean, please feel free to correct me.

Comment 199

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Chun-Min Chang[:chunmin] from comment #198)
> (In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #197)
> > http://mxr.mozilla.org/mozilla-central/source/widget/TextEventDispatcher.
> > cpp?rev=63101b99fa95&mark=136-136#131
> Hi Masayuki-san,
> I may not understand your point right. Do you mean that we can route the
> event dispatching path depends on |event.mFlags.mIsSynthesizedForTests| in
> our code?

Yes, I guess that you are trying to change PresShell to prevent to fire before/after key events if key events came from TIP. However, I don't think that the change should be applied when mIsSynthesizedForTests is true because automated tests are testing current code path. And if automated tests need to emulate VKB's keyboard events, they can use TIP with beginInputTransaction() instead of beginInputTransactionForTests().

> I personally think the testing part should be split from the
> executing code. Instead of adding a testing-specific flag into code, I
> prefer to find a way to simulate the hardware input event. If I
> misunderstand what you mean, please feel free to correct me.

If you need to dispatch hardware keyboard event, you should implement nsIDOMWindowUtils::SendNativeKeyEvent() in Gonk widget. However, I'm not sure if it's available with multi-process.

Comment 200

[C.M.Chang[:chunmin]]

Attachment: part1 - add flags:IsSynthesizedByTIP and InputMethodAppState in WidgetKeyboardEvent

This patch is mostly copied from the old one.

Comment 201

[C.M.Chang[:chunmin]]

Attachment: part2 - Expose IsSynthesizedByTIP to KeyboardEvent

Comment 202

[C.M.Chang[:chunmin]]

Attachment: part3 - HardwareKeyHandler component

Comment 203

[C.M.Chang[:chunmin]]

Attachment: part4 - Interface between PresShell and HardwareKeyHandler

Comment 204

[C.M.Chang[:chunmin]]

Attachment: part5 - Prevent PostHandleKeyboardEvent if it's handling by IME

Comment 205

[C.M.Chang[:chunmin]]

Attachment: part6 - Interface between HardwareKeyHandler and Keyboard App

Hi Masayuki-san,
Could you review these patches when you're available?

Comment 206

[C.M.Chang[:chunmin]]

Comment on attachment 8712081 [details] [diff] [review]
part6 - Interface between HardwareKeyHandler and Keyboard App

Review of attachment 8712081 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/inputmethod/Keyboard.jsm
@@ +156,5 @@
> +                               type: 'hardwareinput',
> +                               keyDict: keyboardEventDict
> +                              });
> +  },
> +

In the previous review, to keep good maintenance of the code, you'd like to see that event's information can be passed by flags.

If I understange corretly, |onHardwareKey| in |nsIHardwareKeyHandler.idl| should be redefined, like:

bool onHardwareKey(in string aType, /* keydown, keypress, keyup */
                   in unsigned long keyPacket,
                   in boolean aRepeat);

the |keyPacket| is an unsigned 32 bits value, its format is:

| codeNameIndex | keyNameIndex  | 
(16 bits + 16 bits)

We can use |keyNameIndex| to get the DOMKeyName and keyCode and |codeNameIndex| to get the DOMCodeName, then we can get |event.key|, |event.keyCode| and |event.code|.

To create a keyboard event with flags in javascript like:

new KeyboardEvent(type, keyPacket, repeat)

we need to add a new function in dom/keyboardevent.h/cpp

For me, the advantage to use bit-flag here is that we can pass other information(e.g., isComposing) without changing the interface. We only need to adjust the format of |keyPacket|. For example, 

| repeat  | isComposing | codeNameIndex | keyNameIndex  |
(1 bit + 1 bit + 15 bits + 15 bits)

However, the drawback of using flag is un-readable. Thus, I think some information like |type| should not be compressed into the flags

Comment 207

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8712074 [details] [diff] [review]
part1 - add flags:IsSynthesizedByTIP and InputMethodAppState in WidgetKeyboardEvent

>@@ -845,17 +847,17 @@ TextInputProcessor::KeydownInternal(const WidgetKeyboardEvent& aKeyboardEvent,
>     return NS_OK;
>   }
>
>   aConsumedFlags |=
>     (status == nsEventStatus_eConsumeNoDefault) ? KEYDOWN_IS_CONSUMED :
>                                                   KEYEVENT_NOT_CONSUMED;
>
>   if (aAllowToDispatchKeypress &&
>-      mDispatcher->MaybeDispatchKeypressEvents(keyEvent, status,
>+      mDispatcher->MaybeDispatchKeypressEvents(keyEvent, status,

What's changing this line??

>   nsString mCodeValue;
>   // OS-specific native event can optionally be preserved
>   void* mNativeKeyEvent;
>   // Unique id associated with a keydown / keypress event. Used in identifing
>   // keypress events for removal from async event dispatch queue in metrofx
>   // after preventDefault is called on keydown events. It's ok if this wraps
>   // over long periods.
>   uint32_t mUniqueId;
>+  // Indicates whether the event is synthesized from Text Input Processor
>+  // or an actual event from nsAppShell.
>+  bool mIsSynthesizedByTIP;
>+
>+  // Indicates that the event is being handled by input method app
>+  typedef uint8_t InputMethodAppStateType;
>+  enum InputMethodAppState : InputMethodAppStateType
>+  {
>+    eNotHandled, // not yet handled by intput method app
>+    eHandling,   // being handled by intput method app
>+    eHandled     // handled by input method app
>+  };
>+  InputMethodAppState mInputMethodAppState;

nit: For memory alignment, all bool members should be defined at the end of each class/struct. However, WidgetWheelEvent is already messy. So, I'll sort it out later. Don't you mind this issue for now.

>+  /**
>+   * Reset the flags related to event flow control for redispatch
>+   */
>+  void ResetForReuse()
>+  {
>+    mFlags.mPropagationStopped = false;
>+    mFlags.mImmediatePropagationStopped = false;
>+    mFlags.mDefaultPrevented = false;
>+    mFlags.mDefaultPreventedByContent = false;
>+    mFlags.mDefaultPreventedByChrome = false;
>+    mFlags.mIsBeingDispatched = false;
>+    mFlags.mDispatchedAtLeastOnce = false;
>+  }

I don't think that it's good idea to reuse events because we haven't done that. And I think that this method should be in BaseEventFlags.


Let me check following patches for deciding if we should add ResetForReuse().

Comment 208

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8712075 [details] [diff] [review]
part2 - Expose IsSynthesizedByTIP to KeyboardEvent

Looks okay to me, but I'd like smaug to review this too.

Comment 209

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8712076 [details] [diff] [review]
part3 - HardwareKeyHandler component

>+++ b/dom/inputmethod/HardwareKeyHandler.cpp
>@@ -0,0 +1,434 @@
>+/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
>+/* vim: set ts=8 sts=2 et sw=2 tw=80: */
>+/* This Source Code Form is subject to the terms of the Mozilla Public
>+ * License, v. 2.0. If a copy of the MPL was not distributed with this
>+ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
>+
>+#include "HardwareKeyHandler.h"
>+#include "mozilla/BasicEvents.h"
>+#include "mozilla/ClearOnShutdown.h"
>+#include "mozilla/dom/TabParent.h"
>+#include "mozilla/EventDispatcher.h"
>+#include "mozilla/TextEvents.h"
>+#include "nsDeque.h"
>+#include "nsFocusManager.h"
>+#include "nsFrameLoader.h"
>+#include "nsIContent.h"
>+#include "nsPIDOMWindow.h"
>+#include "nsPresShell.h"
>+#include "mozilla/EventStateManager.h"

nit: please move this line after EventDispatcher.h?

>+NS_IMETHODIMP
>+HardwareKeyHandler::RegisterListener(nsIHardwareKeyEventListener* aListener)
>+{
>+  // Make sure that listenr is not nullptr and there is no availabe

s/listenr/listener
s/availabe/available

>+  // hardwareKeyEventListener now
>+  if (!aListener) {

NS_WARN_IF()?

>+    return NS_ERROR_NULL_POINTER;
>+  }
>+
>+  if (mHardwareKeyEventListener) {

NS_WARN_IF()?

>+    return NS_ERROR_ALREADY_INITIALIZED;
>+  }
>+
>+  mHardwareKeyEventListener = aListener;
>+  return NS_OK;
>+}
>+
>+NS_IMETHODIMP
>+HardwareKeyHandler::ForwardKeyToInputMethodApp(nsINode* aTarget,
>+                                               WidgetKeyboardEvent* aEvent,
>+                                               nsEventStatus* aEventStatus,
>+                                               bool* aResult)
>+{
>+  MOZ_ASSERT(aTarget, "No target provided");
>+  MOZ_ASSERT(aEvent, "No event provided");
>+
>+  // No need to forward hardware key event to IME if:
>+  // 1. IME is inactive

If you mean, no editor has focus,
s/inactive/disabled

>+  // 2. There is no nsIHardwareKeyEventListener in use
>+  // 3. This key event is genereated IME(by nsITextInputProcessor)

s/genereated/generated by

>+  // 4. The key event is already handling
>+  if (!mInputMethodAppConnected ||
>+      !mHardwareKeyEventListener ||
>+      aEvent->mIsSynthesizedByTIP ||
>+      aEvent->mInputMethodAppState != WidgetKeyboardEvent::eNotHandled) {
>+    *aResult = false;
>+    return NS_OK;
>+  }
>+
>+  // We only forward keydown and keyup event to input-method-app
>+  // because input-method-app will generate keypress by itself.
>+  // The keypress event will be pushed into queue without forwarding.
>+  if (aEvent->mMessage != eKeyPress) {
>+    // Retrieve key for the event
>+    aEvent->GetDOMKeyName(aEvent->mKeyValue);
>+
>+    // Retrieve code for the event
>+    aEvent->GetDOMCodeName(aEvent->mCodeValue);
>+
>+    // Try forwarding this key event to input-method-app
>+    bool isSent = false;
>+    mHardwareKeyEventListener->OnHardwareKey(((aEvent->mMessage == eKeyDown)?
>+                                               "keydown" : "keyup"),
>+                                             NS_ConvertUTF16toUTF8(aEvent->mKeyValue).get(), //keyValue,
>+                                             NS_ConvertUTF16toUTF8(aEvent->mCodeValue).get(),//codeValue,
>+                                             aEvent->mIsRepeat,
>+                                             aEvent->keyCode,
>+                                             aEvent->charCode,
>+                                             aEvent->time,
>+                                             &isSent);

Looks like creating dom::KeyboardEvent instance use it as the argument of OnHardwareKey. See also my comment at the definition of this.

>+NS_IMETHODIMP
>+HardwareKeyHandler::OnHandledByInputMethodApp(const nsAString& aType,
>+                                              bool aDefaultPrevented)
>+{
>+  // We can not handle this reply because the pending event had been already
>+  // removed from the forwarding queue before this reply arrives.
>+  if (!mEventQueue.GetSize()) {
>+    return NS_OK;
>+  }
>+
>+  nsAutoPtr<KeyboardInfo>
>+    keyInfo(static_cast<KeyboardInfo*>(mEventQueue.PopFront()));
>+
>+  // Only allow keydown and keyup to call this method
>+  if (NS_WARN_IF(aType.EqualsLiteral("keydown") &&
>+                 keyInfo->event->mMessage != eKeyDown) ||
>+      NS_WARN_IF(aType.EqualsLiteral("keyup") &&
>+                 keyInfo->event->mMessage != eKeyUp)) {
>+    return NS_ERROR_INVALID_ARG;
>+  }
>+
>+  // Reset the flags to dispatch this event again.
>+  keyInfo->event->ResetForReuse();

I wonder, why don't you just create new WidgetKeyboardEvent instance in the stack? As I said, like ResetForReuse() approach hasn't been used and hard to maintain when new flags are added. So, I strongly recommend that you should create WidgetKeyboardEvent again.

>+  keyInfo->event->mFlags.mDefaultPrevented = aDefaultPrevented;

I wonder, how about mDefaultPrevetnedBy*?

>+  // Set the flag to specify the reply phase.
>+  keyInfo->event->mInputMethodAppState = WidgetKeyboardEvent::eHandled;
>+
>+  // If |aDefaultPrevented| is true, it means that input-method-app has already
>+  // consumed this key, so we can dispatch |mozbrowserafterkeyXXX| now if
>+  // preference "dom.beforeAfterKeyboardEvent.enabled" is enabled.
>+  if (aDefaultPrevented) {
>+    DispatchAfterKeyEvent(keyInfo->target, keyInfo->event);
>+  // Otherwise, it means that input-method-app doesn't handle this key,
>+  // so we need to dispatch it to its original event target.
>+  } else {
>+    DispatchToTargetApp(keyInfo->target, keyInfo->event, keyInfo->status);
>+  }

Don't we need to check if the event target is valid (e.g., still in the DOM tree) if IME app can cause firing some events in the content?

>+  // Handle the pending keypress event once keydown's reply arrives:
>+  // If the event is keydown and it's consumed by input-method-app,
>+  // then there is no need to dispatch the following keypress to its
>+  // original event target. On the other hand, if the keydown event
>+  // doesn't be consumed by input-method-app, then the following
>+  // keypress need to be dispatched to its original event target.
>+  if (keyInfo->event->mMessage != eKeyDown) {
>+    return NS_OK;
>+  }

You've dispatched keyboard down event above, so, we don't know the document and event target state. Please check it first before dispatching next event.

>+
>+  nsAutoPtr<KeyboardInfo>
>+    keypressInfo(static_cast<KeyboardInfo*>(mEventQueue.PeekFront()));
>+
>+  // Make sure that the next of keydown is keypress
>+  MOZ_ASSERT(keypressInfo->event->mMessage == eKeyPress,
>+             "the keydown's next event is not keypress.");

If the keydown event is modifier key's, keypress event shouldn't be fired in Gecko. And also, for conforming UI Events spec, we shouldn't dispatch keypress events for non-printable keys in the future. So, if there is no keypress events, you can just do nothing.

And I guess that Gonk won't dispatch two or more keypress events per keydown event. But it's possible in other platforms. So, please use loop here even though this is used only on B2G now.

>+bool
>+HardwareKeyHandler::DispatchToTargetApp(nsINode* aTarget,
>+                                        WidgetKeyboardEvent* aEvent,
>+                                        nsEventStatus* aStatus)
>+{
>+  MOZ_ASSERT(aTarget, "No target provided");
>+  MOZ_ASSERT(aEvent, "No event provided");
>+
>+  // Get current focus content as the event target
>+  nsCOMPtr<nsIContent> currentFocusTarget = GetFocusContent();
>+  if(!currentFocusTarget) {
>+    return false;
>+  }
>+
>+  nsIContent* originalTarget = aTarget->AsContent();
>+
>+  // Check the event target is same as the current focus element or not.
>+  // The event target will change if focus is changed.
>+  // If the event target is changed, then what should we do?
>+  if (originalTarget != currentFocusTarget) {
>+    NS_WARNING("The current focus element is different from the origial event target.");
>+  }

We should fire on new focused element. That's valid behavior for UI Events spec.

>+  // After the oop target recevices the event from TabChild::RecvRealKeyEvent

s/recevices/receives

>+  // and retrun the result through TabChild::SendDispatchAfterKeyboardEvent,

s/retrun/return

>+bool
>+HardwareKeyHandler::DispatchToCrossProcess(nsINode* aTarget,
>+                                           WidgetKeyboardEvent* aEvent)
>+{
>+  nsCOMPtr<nsIFrameLoaderOwner> remoteLoaderOwner = do_QueryInterface(aTarget);
>+  if (!remoteLoaderOwner) {

NS_WARN_IF()?

>+    return false;
>+  }
>+
>+  RefPtr<nsFrameLoader> remoteFrameLoader =
>+    remoteLoaderOwner->GetFrameLoader();
>+  if (!remoteFrameLoader) {

NS_WARN_IF()?

>+    return false;
>+  }
>+
>+  uint32_t eventMode;
>+  remoteFrameLoader->GetEventMode(&eventMode);
>+  if (eventMode == nsIFrameLoader::EVENT_MODE_DONT_FORWARD_TO_CHILD) {
>+    return false;
>+  }
>+
>+  PBrowserParent* remoteBrowser = remoteFrameLoader->GetRemoteBrowser();
>+  TabParent* remote = static_cast<TabParent*>(remoteBrowser);
>+  if (!remote) {

NS_WARN_IF()?

>+already_AddRefed<nsIContent>
>+HardwareKeyHandler::GetFocusContent()
>+{
>+  nsFocusManager* fm = nsFocusManager::GetFocusManager();
>+  if (!fm) {

NS_WARN_IF()

>+    return nullptr;
>+  }
>+
>+  nsCOMPtr<nsIDOMWindow> focusedWindow;
>+  fm->GetFocusedWindow(getter_AddRefs(focusedWindow));
>+  nsCOMPtr<nsPIDOMWindow> ourWindow = do_QueryInterface(focusedWindow);
>+  if (!ourWindow) {

NS_WARN_IF()

>+    return nullptr;
>+  }
>+
>+  nsPIDOMWindow* rootWindow = ourWindow->GetPrivateRoot();
>+  if (!rootWindow) {

NS_WARN_IF()

>+    return nullptr;
>+  }
>+
>+  nsCOMPtr<nsPIDOMWindow> focusedFrame;
>+  nsCOMPtr<nsIContent> focusedContent =
>+    fm->GetFocusedDescendant(rootWindow, true, getter_AddRefs(focusedFrame));
>+
>+  return focusedContent.forget();

Hmm, this returns event target of keyboard event. So, if there is no focused element, keyboard events should be fired on <body> or something (not sure the exactly target, though).

So, I think that you should rename this method and when there is no focused element in the window, you should return <body> or something.

>+++ b/dom/inputmethod/HardwareKeyHandler.h
>@@ -0,0 +1,82 @@
>+/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
>+/* vim: set ts=8 sts=2 et sw=2 tw=80: */
>+/* This Source Code Form is subject to the terms of the Mozilla Public
>+ * License, v. 2.0. If a copy of the MPL was not distributed with this
>+ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
>+

Please add include guard like:

#ifndef mozilla_HardwareKeyHandler_h_
#define mozilla_HardwareKeyHandler_h_


And add
#endif // #ifndef mozilla_HardwareKeyHandler_h_
at the end of this file.

>+#include "mozilla/StaticPtr.h"
>+#include "nsCOMPtr.h"
>+#include "nsDeque.h"
>+#include "mozilla/EventForwards.h"  // For nsEventStatus
>+#include "nsIHardwareKeyHandler.h"

Could you sort the include order from A to Z?

>+
>+class nsDeque;
>+class nsIPresShell;
>+class nsIContent;
>+class nsINode;

And this order too.

>+
>+namespace mozilla {
>+class WidgetKeyboardEvent;
>+
>+struct KeyboardInfo {

put the |{| (of struct/class/enum) to next line.

>+  WidgetKeyboardEvent* event;
>+  nsINode* target;
>+  nsEventStatus* status;

Please add |m| prefix for member variables.

>+
>+  KeyboardInfo(WidgetKeyboardEvent* aEvent, nsINode* aTarget, nsEventStatus* aStatus)

Too long line. Breaking after each |,| must be easier to read.

>+  {
>+    event = aEvent;
>+    target = aTarget;
>+    status = aStatus;
>+  }
>+};
>+
>+class HardwareKeyHandler : public nsIHardwareKeyHandler
>+{
>+public:
>+  HardwareKeyHandler();
>+
>+  NS_DECL_ISUPPORTS
>+  NS_DECL_NSIHARDWAREKEYHANDLER

I wonder, don't you need to add this class to cycle collector? Because this class may be held by JS.

>+
>+  static already_AddRefed<HardwareKeyHandler> GetInstance();
>+
>+private:
>+  virtual ~HardwareKeyHandler();
>+
>+  // To keep tracking whether or not the input-method-app is active
>+  bool mInputMethodAppConnected;

Please move bool member to the end of this class (for memory alignment). In this case, after mHardwareKeyEventListener.

>diff --git a/dom/inputmethod/nsIHardwareKeyHandler.idl b/dom/inputmethod/nsIHardwareKeyHandler.idl
>+namespace mozilla {
>+  class WidgetKeyboardEvent;

nit: unnecessary indent.

>+}
>+
>+class nsINode;
>+
>+#include "mozilla/EventForwards.h" /* For nsEventStatus */
>+%}
>+
>+[ptr] native WidgetKeyboardEventPtr(mozilla::WidgetKeyboardEvent);
>+[ptr] native nsINodePtr(nsINode);
>+[ptr] native nsEventStatusPtr(nsEventStatus);
>+
>+/**
>+ * This interface should be registered to nsIHardwareKeyHandler through
>+ * |nsIHardwareKeyHandler.registerListener|. When nsIHardwareKeyHandler needs
>+ * to forward a hardware keyboard event to the active input method, this
>+ * listener is invoked to communicate with the active input method. Therefore,
>+ * this listener implementation should contain the communication with the
>+ * active input method.
>+ */
>+[scriptable, function, uuid(cd5aeee3-b4b9-459d-85e7-c0671c7a8a2e)]
>+interface nsIHardwareKeyEventListener : nsISupports
>+{
>+  /**
>+   * This method is called by nsIHardwareKeyHandler, when nsIHardwareKeyHandler
>+   * needs to forward a hardware keyboard event to the active input method.
>+   * Therefore, thie method implementation should forward a hardware keyboard

s/thie/this

>+   * event to the active input method.
>+   */
>+  bool onHardwareKey(in string aType,
>+                     in string aKey,
>+                     in string aCode,
>+                     in boolean aRepeat,
>+                     in unsigned long aKeyCode,
>+                     in unsigned long aCharCode,
>+                     in unsigned long long aTimeStamp);

I wonder, isn't using nsIDOMKeyEvent better for compatibility with future? Like this arguments requires to change the method when DOM Events will add new attributes to KeyboardEvent.

>+};
>+
>+/**
>+ * This interface has two main roles. One is to send a hardware keyboard event
>+ * to the active input method app and the second is to receive its result.

s/the second/the other

>+ * If a keyboard event is triggered from a hardware keyboard when an editor has
>+ * a focus, the event target should be an editor. However, the text input

s/a focus/focus
s/should be an editor/should be the editor

>+ * processor algorithm is implemented in an input method app and it should
>+ * handle the event earlier than the real event target to do the mapping such
>+ * as character conversion according to the language setting or the type of a
>+ * hardware keyboard.
>+ */
>+[scriptable, builtinclass, uuid(25b34270-caad-4d18-a910-860351690639)]
>+interface nsIHardwareKeyHandler : nsISupports
>+{
>+  /**
>+   * Registers a listener which has the communication channel with the active
>+   * input method to forward a hardware keyboard event.
>+   */
>+  void registerListener(in nsIHardwareKeyEventListener aListener);
>+
>+  /**
>+   * Sends a hardware keyboard event to the active input method app to give the
>+   * first chance to handle a hardware keyboard event. Internally, this method

s/a hardware keyboard event/the event

>+   * implementation should call |nsIHardwareKeyEventListener.onHardwareKey| of
>+   * the listener registered through |registerListener|. As a result, the input
>+   * method app having the active input method can receive a hardware keyboard
>+   * event through this method call.
>+   *
>+   * Returns true, if the event is sent to the active input method app.
>+   * Returns false, if the event is not sent.
>+   *
>+   * If it returns true, the caller should pause the further event processing
>+   * to wait for the completion of the event handling in input method app.
>+   * Once |onHandledByInputMethodApp| is called, the pending event processing
>+   * should be resumed according to the result from the active input method app.
>+   * If it returns false, the caller should continue the normal event
>+   * processing.
>+   */
>+  bool forwardKeyToInputMethodApp(in nsINodePtr aTarget,
>+                                  in WidgetKeyboardEventPtr aEvent,
>+                                  in nsEventStatusPtr aEventStatus);

Shouldn't this be [noscript]?

Comment 210

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8712074 [details] [diff] [review]
part1 - add flags:IsSynthesizedByTIP and InputMethodAppState in WidgetKeyboardEvent

Now, I think we shouldn't create ResetForReuse().

Comment 211

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8712078 [details] [diff] [review]
part4 - Interface between PresShell and HardwareKeyHandler

>--- a/layout/base/nsIPresShell.h
>+++ b/layout/base/nsIPresShell.h
>@@ -874,16 +875,21 @@ public:
>   /**
>    * Dispatch AfterKeyboardEvent with specific target.
>    */
>   virtual void DispatchAfterKeyboardEvent(nsINode* aTarget,
>                                           const mozilla::WidgetKeyboardEvent& aEvent,
>                                           bool aEmbeddedCancelled) = 0;
> 
>   /**
>+   * Return whethor or not event can be dispatched

s/whethor/whether

>+   */
>+  virtual bool CanDispatchEvent(const mozilla::WidgetGUIEvent* aEvent = nullptr) = 0;

You need to modify UUID of nsIPresShell due to adding this virtual method.

>@@ -1707,16 +1713,19 @@ protected:
> #endif
> 
>   // At least on Win32 and Mac after interupting a reflow we need to post
>   // the resume reflow event off a timer to avoid event starvation because
>   // posted messages are processed before other messages when the modal
>   // moving/sizing loop is running, see bug 491700 for details.
>   nsCOMPtr<nsITimer>        mReflowContinueTimer;
> 
>+  // Forward hardware key event to IME

s/event/events
s/IME/IME apps on B2G

>+  nsCOMPtr <nsIHardwareKeyHandler> mHardwareKeyHandler;

unnecessary whitespace after nsCOMPtr and before <.

>--- a/layout/base/nsPresShell.cpp
>+++ b/layout/base/nsPresShell.cpp
>@@ -727,16 +728,22 @@ PresShell::BeforeAfterKeyboardEventEnabled()
>   if (!sInitialized) {
>     Preferences::AddBoolVarCache(&sBeforeAfterKeyboardEventEnabled,
>       "dom.beforeAfterKeyboardEvent.enabled");
>     sInitialized = true;
>   }
>   return sBeforeAfterKeyboardEventEnabled;
> }
> 
>+/* static */ bool
>+PresShell::IsTargetIframe(nsINode* aTarget) {

Put |{| to the next line.

>@@ -776,16 +783,17 @@ PresShell::PresShell()
>                                  "dom.w3c_pointer_events.enabled", true);
>     addedPointerEventEnabled = true;
>   }
> 
>   mPaintingIsFrozen = false;
>   mHasCSSBackgroundColor = true;
>   mIsLastChromeOnlyEscapeKeyConsumed = false;
>   mHasReceivedPaintMessage = false;
>+  mHardwareKeyHandler = nullptr;

mHardwareKeyHandler is nsCOMPtr. So, you don't need to set it to nullptr explicitly.

>@@ -947,16 +955,21 @@ PresShell::Init(nsIDocument* aDocument,
> 
>   if (mPresContext->IsRootContentDocument()) {
>     mZoomConstraintsClient = new ZoomConstraintsClient();
>     mZoomConstraintsClient->Init(this, mDocument);
>     if (gfxPrefs::MetaViewportEnabled() || gfxPrefs::APZAllowZooming()) {
>       mMobileViewportManager = new MobileViewportManager(this, mDocument);
>     }
>   }
>+
>+  if (XRE_IsParentProcess()) {
>+    nsresult rv;
>+    mHardwareKeyHandler = do_GetService("@mozilla.org/HardwareKeyHandler;1", &rv);
>+  }

Hmm, this means that you create HardwareKeyHandler instances on every platform. You should restrict this. If you want to create it even on desktop for tests, perhaps, using a pref is better? Otherwise, using #ifdef? Anyway, maybe it's better GetInstance() to return nullptr if it shouldn't be created.

> bool
>-PresShell::CanDispatchEvent(const WidgetGUIEvent* aEvent) const
>+PresShell::CanDispatchEvent(const WidgetGUIEvent* aEvent)

Cannot you to define this method with const in nsIPresShell?

> void
> PresShell::HandleKeyboardEvent(nsINode* aTarget,
>                                WidgetKeyboardEvent& aEvent,
>                                bool aEmbeddedCancelled,
>                                nsEventStatus* aStatus,
>                                EventDispatchingCallback* aEventCB)
> {
>-  if (aEvent.mMessage == eKeyPress ||
>-      !BeforeAfterKeyboardEventEnabled()) {
>+  MOZ_ASSERT(aTarget);
>+
>+  if (aEvent.mIsSynthesizedByTIP) {

How about if the event was synthesized for tests?

>     EventDispatcher::Dispatch(aTarget, mPresContext,
>                               &aEvent, nullptr, aStatus, aEventCB);
>     return;
>   }
> 
>-  MOZ_ASSERT(aTarget);
>+  // return true if the real event target is in its child process
>+  bool targetIsIframe = IsTargetIframe(aTarget);
>+
>+  if (aEvent.mMessage == eKeyPress ||
>+      !BeforeAfterKeyboardEventEnabled()) {
>+    FinishDispatchingKeyboardEvent(targetIsIframe, aTarget, aEvent, aStatus, aEventCB);

too long line, please wrap after |aEvent,|.

>@@ -6802,19 +6820,19 @@ PresShell::HandleKeyboardEvent(nsINode* aTarget,
>     return;
>   }
> 
>   // Event listeners may kill nsPresContext and nsPresShell.
>   if (!CanDispatchEvent()) {
>     return;
>   }
> 
>-  // Dispatch actual key event to event target.
>-  EventDispatcher::Dispatch(aTarget, mPresContext,
>-                            &aEvent, nullptr, aStatus, aEventCB);
>+  if (FinishDispatchingKeyboardEvent(targetIsIframe, aTarget, aEvent, aStatus, aEventCB)) {

Too long line...

>+    return;
>+  }
> 
>   if (aEvent.mFlags.mDefaultPrevented) {
>     // When embedder prevents the default action of actual key event, attribute
>     // 'embeddedCancelled' of after event is false, i.e. |!targetIsIframe|.
>     // On the contrary, if the defult action is prevented by embedded iframe,
>     // 'embeddedCancelled' is true which equals to |!targetIsIframe|.
>     DispatchAfterKeyboardEventInternal(chain, aEvent, !targetIsIframe, chainIndex);
>     return;
>@@ -6825,16 +6843,74 @@ PresShell::HandleKeyboardEvent(nsINode* aTarget,
>     return;
>   }
> 
>   // Dispatch after events to all items in the chain.
>   DispatchAfterKeyboardEventInternal(chain, aEvent,
>                                      aEvent.mFlags.mDefaultPrevented);
> }
> 
>+bool
>+PresShell::FinishDispatchingKeyboardEvent(bool isTargetRemote,

Should be aIsTargetRemote.

>+                                          nsINode* aTarget,
>+                                          WidgetKeyboardEvent& aEvent,
>+                                          nsEventStatus* aStatus,
>+                                          EventDispatchingCallback* aEventCB)
>+{
>+  // In-process case: the real event target is in the current process
>+  if (!isTargetRemote) {
>+    if(ForwardKeyToInputMethodApp(aTarget, aEvent, aStatus)) {
>+      return true;
>+    }
>+
>+    // If keyboard doesn't be forward to input-method-app, then it should be

s/doesn't be/isn't

>+    // dispatched to its event target.
>+    EventDispatcher::Dispatch(aTarget, mPresContext,
>+                              &aEvent, nullptr, aStatus, aEventCB);
>+
>+    return false;
>+  }
>+
>+  // oop case: the real event target is in the child process of the current one.
>+  // Dispatch the keyboard event to the iframe that embed the remote
>+  // event target first.
>+  EventDispatcher::Dispatch(aTarget, mPresContext,
>+                            &aEvent, nullptr, aStatus, aEventCB);
>+
>+  if (aEvent.mFlags.mDefaultPrevented) {
>+    return false;
>+  }
>+
>+  // Check whethor or not the event should be forwarded to input-method-app.

s/whethor/whether

>+  return ForwardKeyToInputMethodApp(aTarget, aEvent, aStatus);
>+}
>+
>+bool
>+PresShell::ForwardKeyToInputMethodApp(nsINode* aTarget,
>+                                      WidgetKeyboardEvent& aEvent,
>+                                      nsEventStatus* aStatus)
>+{
>+  if (!mHardwareKeyHandler) {
>+    return false;
>+  }
>+
>+  bool forwarded = false;
>+  mHardwareKeyHandler->ForwardKeyToInputMethodApp(aTarget,
>+                                                  aEvent.AsKeyboardEvent(),
>+                                                  aStatus,
>+                                                  &forwarded);
>+  if (forwarded) {
>+    // No need to dispatch the forwarded keyboard event to it's child procss

s/procss/process

>+++ b/layout/base/nsPresShell.h
>@@ -747,17 +751,24 @@ protected:
>          const mozilla::WidgetKeyboardEvent& aEvent,
>          size_t& aChainIndex,
>          bool& aDefaultPrevented);
>   void DispatchAfterKeyboardEventInternal(
>          const nsTArray<nsCOMPtr<mozilla::dom::Element> >& aChain,
>          const mozilla::WidgetKeyboardEvent& aEvent,
>          bool aEmbeddedCancelled,
>          size_t aChainIndex = 0);
>-  bool CanDispatchEvent(const mozilla::WidgetGUIEvent* aEvent = nullptr) const;
>+  bool FinishDispatchingKeyboardEvent(bool isTargetRemote,
>+                                      nsINode* aTarget,
>+                                      mozilla::WidgetKeyboardEvent& aEvent,
>+                                      nsEventStatus* aStatus,
>+                                      mozilla::EventDispatchingCallback* aEventCB);
>+  bool ForwardKeyToInputMethodApp(nsINode* aTarget,
>+                                  mozilla::WidgetKeyboardEvent& aEvent,
>+                                  nsEventStatus* aStatus);

I'd like you to explain what these methods do with comment. Especially, the result is unclear.


Sorry for the delay to review. Unfortunately, time over today, I'll check remaining patches tomorrow and I should check this patch more carefully.

Comment 212

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8712075 [details] [diff] [review]
part2 - Expose IsSynthesizedByTIP to KeyboardEvent

So we don't use KeyboardEventInit.isSynthesizedByTIP for anything, yet expose it to the web? (if one passes an object which has a getter isSynthesizedByTIP to constructor, that getter gets called.)
Doesn't sounds quite right.

Drop isSynthesizedByTIP from the dictionary and just have the [ChromeOnly] attribute in the interface. With that, r+.

If you do other changes, ask review again.

Comment 213

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #209)
Hi Masayuki-san,
Thank you for sharing your time to review the code.

> Comment on attachment 8712076 [details] [diff] [review]
> part3 - HardwareKeyHandler component
> 
> >+++ b/dom/inputmethod/HardwareKeyHandler.cpp 
> >+  keyInfo->event->mFlags.mDefaultPrevented = aDefaultPrevented;
> 
> I wonder, how about mDefaultPrevetnedBy*?

Do we really need to get the result of mDefaultPrevetnedBy*? The |OnHandledByInputMethodApp| will be called by Mozkeyboard.js or Keyboard.jsm, and we can't know the the value of mDefaultPrevetnedBy* from javascript. If we really need to pass the value of mDefaultPrevetnedBy*, we might need to set it by ourselves(need find the answer of Event::IsChrome first?)

And the following dispatching won't be affected if the |event.defaultPrevented| is ture, no matter the event is prevented by chrome or content. If we need to set the value of mDefaultPrevetnedBy*, I 'll add it in next version.

> >+bool
> >+HardwareKeyHandler::DispatchToTargetApp(nsINode* aTarget,
> >+                                        WidgetKeyboardEvent* aEvent,
> >+                                        nsEventStatus* aStatus)
> >+{
> >+  MOZ_ASSERT(aTarget, "No target provided");
> >+  MOZ_ASSERT(aEvent, "No event provided");
> >+
> >+  // Get current focus content as the event target
> >+  nsCOMPtr<nsIContent> currentFocusTarget = GetFocusContent();
> >+  if(!currentFocusTarget) {
> >+    return false;
> >+  }
> >+
> >+  nsIContent* originalTarget = aTarget->AsContent();
> >+
> >+  // Check the event target is same as the current focus element or not.
> >+  // The event target will change if focus is changed.
> >+  // If the event target is changed, then what should we do?
> >+  if (originalTarget != currentFocusTarget) {
> >+    NS_WARNING("The current focus element is different from the origial event target.");
> >+  }
> 
> We should fire on new focused element. That's valid behavior for UI Events
> spec.

I have a question here:

Suppose beforekey* and afterkey* are enabled. Imagine the following situation: 
1) At time T0, we focus on a input element in a remote app A and 
   the keyboard event is forwarding to IME via IPC during T0~T2, 
   from Keyboard.jsm to MozKeyboard.js. 
2) At time T1, the focus is changed to another element in remote app B due to some reason. 
3) At time T2, IME gets the key event and decides that it doesn't want to handle this key,
   so MozKeyboard.js sends the result back to Keyboard.jsm during T2~T3.
4) At time T3, the Keyboard.jsm gets the result that the |event.defaultPrevented| is false
   and pass the result back to |HardwareKeyHandler.onHandledByInputMethodApp|
5) At time T4, HardwareKeyHandler should dispatch the key event as original path.

In this case, if we dispatch the event to the focus element, which locates in app B, then we will have following problems:

1. The beforeKey* should be dispatched to the mozbrowser-embedder, and the mozbrowser-embedder of the focus element in app A and the one in app B may be different.
  e.g., In nested-oop case: 
  if the focus element is inside a remote mozbrowser iframe in the remote app A or B, 
  then the mozbrowser-embedder of focus element in app A and B will be different.

2. Do we need to dispatch the afterKey* event for the original event target? Do we need to dispatch the afterKey* event for the new event target? Same as above, the afterKey*'s event targets may be different after focus is changed. 

If we take the latest focus element as the new event target, we might need to restart the event propagation and dispatch a new keyboard event because I think they are different events now after changing event targets. However, how to deal with those un-dispatched pending keyboard events? Drop them?

Comment 214

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Chun-Min Chang[:chunmin] from comment #213)
> (In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #209)
> Hi Masayuki-san,
> Thank you for sharing your time to review the code.
> 
> > Comment on attachment 8712076 [details] [diff] [review]
> > part3 - HardwareKeyHandler component
> > 
> > >+++ b/dom/inputmethod/HardwareKeyHandler.cpp 
> > >+  keyInfo->event->mFlags.mDefaultPrevented = aDefaultPrevented;
> > 
> > I wonder, how about mDefaultPrevetnedBy*?
> 
> Do we really need to get the result of mDefaultPrevetnedBy*?

If it's possible, I think so.

> The |OnHandledByInputMethodApp| will be called by Mozkeyboard.js or
> Keyboard.jsm, and we can't know the the value of mDefaultPrevetnedBy* from
> javascript. If we really need to pass the value of mDefaultPrevetnedBy*, we
> might need to set it by ourselves(need find the answer of Event::IsChrome
> first?)

Indeed, it doesn't make sense to expose them only for now. If there will be some bugs caused by this issue, we should do that, though.

> And the following dispatching won't be affected if the
> |event.defaultPrevented| is ture, no matter the event is prevented by chrome
> or content. If we need to set the value of mDefaultPrevetnedBy*, I 'll add
> it in next version.

Okay, keep current design. We should try to fix if that causes some trouble actually.

> > >+bool
> > >+HardwareKeyHandler::DispatchToTargetApp(nsINode* aTarget,
> > >+                                        WidgetKeyboardEvent* aEvent,
> > >+                                        nsEventStatus* aStatus)
> > >+{
> > >+  MOZ_ASSERT(aTarget, "No target provided");
> > >+  MOZ_ASSERT(aEvent, "No event provided");
> > >+
> > >+  // Get current focus content as the event target
> > >+  nsCOMPtr<nsIContent> currentFocusTarget = GetFocusContent();
> > >+  if(!currentFocusTarget) {
> > >+    return false;
> > >+  }
> > >+
> > >+  nsIContent* originalTarget = aTarget->AsContent();
> > >+
> > >+  // Check the event target is same as the current focus element or not.
> > >+  // The event target will change if focus is changed.
> > >+  // If the event target is changed, then what should we do?
> > >+  if (originalTarget != currentFocusTarget) {
> > >+    NS_WARNING("The current focus element is different from the origial event target.");
> > >+  }
> > 
> > We should fire on new focused element. That's valid behavior for UI Events
> > spec.
> 
> I have a question here:
> 
> Suppose beforekey* and afterkey* are enabled. Imagine the following
> situation: 
> 1) At time T0, we focus on a input element in a remote app A and 
>    the keyboard event is forwarding to IME via IPC during T0~T2, 
>    from Keyboard.jsm to MozKeyboard.js. 
> 2) At time T1, the focus is changed to another element in remote app B due
> to some reason. 
> 3) At time T2, IME gets the key event and decides that it doesn't want to
> handle this key,
>    so MozKeyboard.js sends the result back to Keyboard.jsm during T2~T3.
> 4) At time T3, the Keyboard.jsm gets the result that the
> |event.defaultPrevented| is false
>    and pass the result back to |HardwareKeyHandler.onHandledByInputMethodApp|
> 5) At time T4, HardwareKeyHandler should dispatch the key event as original
> path.
> 
> In this case, if we dispatch the event to the focus element, which locates
> in app B, then we will have following problems:
> 
> 1. The beforeKey* should be dispatched to the mozbrowser-embedder, and the
> mozbrowser-embedder of the focus element in app A and the one in app B may
> be different.
>   e.g., In nested-oop case: 
>   if the focus element is inside a remote mozbrowser iframe in the remote
> app A or B, 
>   then the mozbrowser-embedder of focus element in app A and B will be
> different.

I don't 100% understand this scenario, though.

beforekeydown and afterkeydown should be fired on same target as keydown. Similarly, beforekeyup and afterkeyup should be fired on same target as keyup.

On the other hand, keydown, keyup and keypress should be fired on current focused element in active app since this is defined by standard spec. If you worry about something security issues, I think it's okay not to dispatch keypress and keyup events if active app is changed. But we shouldn't dispatch key events to non-focused element. It really breaks the standard spec.

> 2. Do we need to dispatch the afterKey* event for the original event target?
> Do we need to dispatch the afterKey* event for the new event target? Same as
> above, the afterKey*'s event targets may be different after focus is
> changed. 

So, I think the original event target is the right target of afterKey*.

Note that the purpose of beforeKey* and afterKey* are internal use only for the system. So, we don't have any rules for them but we should keep its behavior to work for the purpose.

Comment 215

[C.M.Chang[:chunmin]]

Attachment: part1 - add flags:IsSynthesizedByTIP and InputMethodAppState in WidgetKeyboardEvent

Comment 216

[C.M.Chang[:chunmin]]

Attachment: part2 - Prevent PostHandleKeyboardEvent if it's handling by IME

Comment 217

[C.M.Chang[:chunmin]]

Attachment: part3 - Interface between PresShell and HardwareKeyHandler

Comment 218

[C.M.Chang[:chunmin]]

Comment on attachment 8715158 [details] [diff] [review]
part3 - Interface between PresShell and HardwareKeyHandler

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #211)
> Comment on attachment 8712078 [details] [diff] [review]
> part4 - Interface between PresShell and HardwareKeyHandler
>
> > void
> > PresShell::HandleKeyboardEvent(nsINode* aTarget,
> >                                WidgetKeyboardEvent& aEvent,
> >                                bool aEmbeddedCancelled,
> >                                nsEventStatus* aStatus,
> >                                EventDispatchingCallback* aEventCB)
> > {
> >-  if (aEvent.mMessage == eKeyPress ||
> >-      !BeforeAfterKeyboardEventEnabled()) {
> >+  MOZ_ASSERT(aTarget);
> >+
> >+  if (aEvent.mIsSynthesizedByTIP) {
>
> How about if the event was synthesized for tests?
>
I am trying to find other or create one API that doesn't use TIP to simulate the hardware key input to avoid this.

Comment 219

[C.M.Chang[:chunmin]]

Attachment: part4 - HardwareKeyHandler component

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #209)
> Comment on attachment 8712076 [details] [diff] [review]
> part3 - HardwareKeyHandler component
> 
> >+++ b/dom/inputmethod/HardwareKeyHandler.h
> 
> >+class HardwareKeyHandler : public nsIHardwareKeyHandler
> >+{
> >+public:
> >+  HardwareKeyHandler();
> >+
> >+  NS_DECL_ISUPPORTS
> >+  NS_DECL_NSIHARDWAREKEYHANDLER
> 
> I wonder, don't you need to add this class to cycle collector? Because this
> class may be held by JS.
I think nsIHardwareKeyHandler won't be part of reference cycle in current implementation.
However, I add this class into cycle collection just in case.

> Comment on attachment 8712076 [details] [diff] [review]
> part3 - HardwareKeyHandler component
> 
> >+++ b/dom/inputmethod/HardwareKeyHandler.cpp
> 
> >+bool
> >+HardwareKeyHandler::DispatchToTargetApp(nsINode* aTarget,
> >+                                        WidgetKeyboardEvent* aEvent,
> >+                                        nsEventStatus* aStatus)
> >+{
> >+  MOZ_ASSERT(aTarget, "No target provided");
> >+  MOZ_ASSERT(aEvent, "No event provided");
> >+
> >+  // Get current focus content as the event target
> >+  nsCOMPtr<nsIContent> currentFocusTarget = GetFocusContent();
> >+  if(!currentFocusTarget) {
> >+    return false;
> >+  }
> >+
> >+  nsIContent* originalTarget = aTarget->AsContent();
> >+
> >+  // Check the event target is same as the current focus element or not.
> >+  // The event target will change if focus is changed.
> >+  // If the event target is changed, then what should we do?
> >+  if (originalTarget != currentFocusTarget) {
> >+    NS_WARNING("The current focus element is different from the origial event target.");
> >+  }
> 
> We should fire on new focused element. That's valid behavior for UI Events
> spec.
Hmm...I still think this behavior is a little bit weird.
This implies that the event target can be changed during the event is dispatching.
I agreed with you that the event target should be the focused element,
but shouldn't it be decided at the moment that user press the physical keyboard?
I personally think it's weird that the event target will be set again to the new focused element
before user really do some actions to trigger the events(e.g., keypress, click, ..)?

Comment 220

[C.M.Chang[:chunmin]]

Attachment: part5 - Expose KeyboardEventInit dictionary

I expose the dictionary for keyboard event initialization to Keyboard.jsm and pass it to MozKeyboard.js to create a new keyboard event.

Comment 221

[C.M.Chang[:chunmin]]

Attachment: part6 - Expose DefaultPreventedBy*

Expose defaultPreventedBy* to MozKeyboard.js and pass its results to nsIHardwareKeyHandler

Comment 222

[C.M.Chang[:chunmin]]

Attachment: part7 - Interface between HardwareKeyHandler and Input Method App

Comment 223

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8715179 [details] [diff] [review]
part5 - Expose KeyboardEventInit dictionary

mEvent shouldn't be null ever.

I wonder how to keep KeyboardEvent::GetInitDict updated if/when new stuff is added to KeyboardEventInit or any dictionaries it inherits.
I don't really have good ideas for that.

Comment 224

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8715181 [details] [diff] [review]
part6 - Expose DefaultPreventedBy*

Interesting. I can see this being useful for UI code in FF.

Comment 225

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8715157 [details] [diff] [review]
part2 - Prevent PostHandleKeyboardEvent if it's handling by IME

The scope change is necessary for part4. So, it should've been in it. But it's okay.

Comment 226

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8715179 [details] [diff] [review]
part5 - Expose KeyboardEventInit dictionary

>+  // legacy attributes
>+  aParam.mKeyCode = internalEvent->keyCode;
>+  aParam.mCharCode = internalEvent->charCode;

I think that these are not exactly correct. However, they must work well in most cases.

You should use KeyCode() and CharCode() for initializing them if you do not change the event type. Otherwise, you need to adjust the values before creating new event with different event type.

>+  aParam.mWhich = mInitializedWhichValue;

This is incorrect. mInitializedWhichValue is valid only when the event is created by JS. You should use Which().


If you don't change the event type, please using the methods, then, r=masayuki.  Otherwise, please ask review me again.

>diff --git a/dom/webidl/KeyboardEvent.webidl b/dom/webidl/KeyboardEvent.webidl
>+
>+  [Cached, ChromeOnly, Constant]
>+  readonly attribute KeyboardEventInit initDict;

Could you add comment before here for explaining above complicated issue. This should be used only for creating event with same event type.

Comment 227

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8715158 [details] [diff] [review]
part3 - Interface between PresShell and HardwareKeyHandler

>-// f17842ee-f1f0-4193-814f-70d706b67060
>+// 3a0de342-93f6-4a40-9c77-574662bb2a43
> #define NS_IPRESSHELL_IID \
>-{ 0xf17842ee, 0xf1f0, 0x4193, \
>-  { 0x81, 0x4f, 0x70, 0xd7, 0x06, 0xb6, 0x70, 0x60 } }
>+{ 0x3a0de342, 0x93f6, 0x4a40, \
>+  { 0x9c, 0x77, 0x57, 0x46, 0x62, 0xbb, 0x2a, 0x43 } }

FYI: Now, we don't need to modify uuid only in mozilla-central.
https://groups.google.com/forum/#!topic/mozilla.dev.platform/HE1_qZhPj1I

> void
> PresShell::HandleKeyboardEvent(nsINode* aTarget,
>                                WidgetKeyboardEvent& aEvent,
>                                bool aEmbeddedCancelled,
>                                nsEventStatus* aStatus,
>                                EventDispatchingCallback* aEventCB)
> {
>+  MOZ_ASSERT(aTarget);
>+
>+  // return true if the real event target is in its child process
>+  bool targetIsIframe = IsTargetIframe(aTarget);
>+
>   if (aEvent.mMessage == eKeyPress ||
>-      !BeforeAfterKeyboardEventEnabled()) {
>+      !BeforeAfterKeyboardEventEnabled() ||
>+      aEvent.mIsSynthesizedByTIP) {
>+#ifdef MOZ_B2G
>+    FinishDispatchingKeyboardEvent(targetIsIframe, aTarget, aEvent,
>+                                   aStatus, aEventCB);
>+#else
>     EventDispatcher::Dispatch(aTarget, mPresContext,
>                               &aEvent, nullptr, aStatus, aEventCB);
>+#endif // MOZ_B2G

I don't like to insert a lot of #ifdef MOZ_B2G into normal path. I think that FinishDispatchingKeyboardEvent should have both implementation. I.e., if |#ifndef MOZ_B2G|, it just calls EventDispatcher::Dispatch().

>-  // Dispatch actual key event to event target.
>+#ifdef MOZ_B2G
>+  if (FinishDispatchingKeyboardEvent(targetIsIframe, aTarget, aEvent,
>+                                     aStatus, aEventCB)) {
>+    return;
>+  }
>+#else
>   EventDispatcher::Dispatch(aTarget, mPresContext,
>                             &aEvent, nullptr, aStatus, aEventCB);
>+#endif // MOZ_B2G

Same here.

Comment 228

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Olli Pettay [:smaug] from comment #223)
> Comment on attachment 8715179 [details] [diff] [review]
> part5 - Expose KeyboardEventInit dictionary
> 
> mEvent shouldn't be null ever.
> 
> I wonder how to keep KeyboardEvent::GetInitDict updated if/when new stuff is
> added to KeyboardEventInit or any dictionaries it inherits.
> I don't really have good ideas for that.

Anyway, I think that the patch should add warning about this to the definition of KeyboardEventInit.

Comment 229

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Chun-Min Chang[:chunmin] from comment #219)
> > >+bool
> > >+HardwareKeyHandler::DispatchToTargetApp(nsINode* aTarget,
> > >+                                        WidgetKeyboardEvent* aEvent,
> > >+                                        nsEventStatus* aStatus)
> > >+{
> > >+  MOZ_ASSERT(aTarget, "No target provided");
> > >+  MOZ_ASSERT(aEvent, "No event provided");
> > >+
> > >+  // Get current focus content as the event target
> > >+  nsCOMPtr<nsIContent> currentFocusTarget = GetFocusContent();
> > >+  if(!currentFocusTarget) {
> > >+    return false;
> > >+  }
> > >+
> > >+  nsIContent* originalTarget = aTarget->AsContent();
> > >+
> > >+  // Check the event target is same as the current focus element or not.
> > >+  // The event target will change if focus is changed.
> > >+  // If the event target is changed, then what should we do?
> > >+  if (originalTarget != currentFocusTarget) {
> > >+    NS_WARNING("The current focus element is different from the origial event target.");
> > >+  }
> > 
> > We should fire on new focused element. That's valid behavior for UI Events
> > spec.
> Hmm...I still think this behavior is a little bit weird.
> This implies that the event target can be changed during the event is
> dispatching.

Yes.

> I agreed with you that the event target should be the focused element,
> but shouldn't it be decided at the moment that user press the physical
> keyboard?

I don't think so. For example, data:text/html,<input onkeydown="event.target.blur();">, then, even if keydown is fired on the input element, keypress and keyup event will be fired on the root element (or <body>?).

If you worry about the security if a keyboard event is fired on *unexpected* application, we should stop dispatching keyboard event instead of firing on unfocused application.

> I personally think it's weird that the event target will be set again to the
> new focused element
> before user really do some actions to trigger the events(e.g., keypress,
> click, ..)?

But that's the current standard behavior...

Comment 230

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8715178 [details] [diff] [review]
part4 - HardwareKeyHandler component

>+NS_IMETHODIMP
>+HardwareKeyHandler::ForwardKeyToInputMethodApp(nsINode* aTarget,
>+                                               WidgetKeyboardEvent* aEvent,
>+                                               nsEventStatus* aEventStatus,
>+                                               bool* aResult)

Oh... Sorry, I probably misled you. Actually, I said you should use [noscript] for this method definition.

But you can define C++ method directly in xpidl. For example, see here:
http://mxr.mozilla.org/mozilla-central/source/dom/interfaces/events/nsIDOMEventTarget.idl#107

Then, you can specify the result type as bool. Then, you can make implementation of this method and the users simpler. Sorry for my mistake. I should've realized that.

>+{
>+  MOZ_ASSERT(aTarget, "No target provided");
>+  MOZ_ASSERT(aEvent, "No event provided");
>+
>+  // No need to forward hardware key event to IME if:
>+  // 0. key's defaultPrevented is true;
>+  // 1. IME is disabled
>+  // 2. There is no nsIHardwareKeyEventListener in use
>+  // 3. This key event is generated by IME(from nsITextInputProcessor)
>+  // 4. The key event is already handling
>+  if (aEvent->mFlags.mDefaultPrevented ||
>+      !mInputMethodAppConnected ||
>+      !mHardwareKeyEventListener ||
>+      aEvent->mIsSynthesizedByTIP ||
>+      aEvent->mInputMethodAppState != WidgetKeyboardEvent::eNotHandled) {
>+    *aResult = false;
>+    return NS_OK;
>+  }

Then, you can make this simpler, even though the number of lines are increased:

// No need to forward hardware key event to IME if key's defaultPrevented is true.
if (aEvent->mFlags.mDefaultPrevented) {
  return false;
}
// No need to forward hardware key event to IME if IME is disabled.
if (!mInputMethodAppConnected) {
  return false;
}
...

This style makes maintaining the if state and reading the condition easier.

>+    // Create a Keyboard event for nsIHardwareKeyEventListener.onHardwareKey
>+    nsCOMPtr<EventTarget> evtTarget = do_QueryInterface(aTarget);

nit: eventTarget is better.

>+    nsPresContext* presCtx = GetPresContext(aTarget);

nit: presContext is better.

>+    KeyboardEvent* keyEvt = new KeyboardEvent(evtTarget, presCtx, aEvent);

nit: keyboardEvent is better.

>+  nsEventStatus status = aEventStatus? *aEventStatus : nsEventStatus_eIgnore;

nit: insert a whitespace before |?|.

>+  KeyboardInfo* copiedInfo =
>+    new KeyboardInfo(new WidgetKeyboardEvent(*aEvent), aTarget, &status);

You create an instance of WidgetKeyboardEvent here. But looks like that nobody (including KeyboardInfo) doesn't destroy it.

>+  size_t sizeBefore = mEventQueue.GetSize();
>+  mEventQueue.Push(copiedInfo);
>+  NS_ENSURE_TRUE(mEventQueue.GetSize() == sizeBefore + 1,
>+                 NS_ERROR_OUT_OF_MEMORY);

Could you not use NS_ENSURE_* anymore? And returning false or calling MOZ_CRASH() may be better.

>+    // Dispatch the keypress to its original event target if:

So, we need agreement about this...

I'm thinking that if target app is changed, we should stop dispatching remaining keyboard events for preventing unexpected operation in the new app.

Otherwise, i.e., focus is changed in the app, we should fine keyboard event in new focused element for conforming to the spec.

>+void
>+HardwareKeyHandler::PostHandleKeyboardEvent(nsINode* aTarget,
>+                                            WidgetKeyboardEvent* aEvent,
>+                                            nsEventStatus* aStatus)
>+{
>+  MOZ_ASSERT(aTarget, "No target provided");
>+  MOZ_ASSERT(aEvent, "No event provided");
>+
>+  nsPresContext* presCtx = GetPresContext(aTarget);

nit: presContext is better.

>+
>+  RefPtr<mozilla::EventStateManager> manager = presCtx->EventStateManager();

Unfortunately, we don't have good agreement for this name... manager sounds not odd.

>+  if (NS_WARN_IF(!manager)) {
>+    return;
>+  }

If the method to retrieve a pointer to object isn't prefixed by "Get", the result cannot be nullptr. So, MOZ_ASSERT() may be enough here. As far as I know, nsPresContext::mEventStateManager must not be nullptr until it's being destroyed.

>+void
>+HardwareKeyHandler::SetDefaultPrevented(WidgetKeyboardEvent* aEvent,
>+                                        uint16_t aDefaultPrevented) {
>+  if (aDefaultPrevented & DEFAULT_PREENTED) {
>+    aEvent->mFlags.mDefaultPrevented = true;
>+  }
>+
>+  if (aDefaultPrevented & DEFAULT_PREENTED_BY_CHROME) {
>+    aEvent->mFlags.mDefaultPreventedByChrome = true;
>+  }
>+
>+  if (aDefaultPrevented & DEFAULT_PREENTED_BY_CONTENT) {
>+    aEvent->mFlags.mDefaultPreventedByContent = true;
>+  }
>+}

Nice!

>+struct KeyboardInfo
>+{
>+  WidgetKeyboardEvent* mEvent;
>+  nsINode* mTarget;
>+  nsEventStatus* mStatus;
>+
>+  KeyboardInfo(WidgetKeyboardEvent* aEvent,
>+               nsINode* aTarget,
>+               nsEventStatus* aStatus)
>+  {
>+    mEvent = aEvent;
>+    mTarget = aTarget;
>+    mStatus = aStatus;
>+  }
>+};

So, if you want this to *own* mEvent, there should be a flag if this owns it. If it's owning the event, mEvent should be deleted in destructor.

>+
>+class HardwareKeyHandler : public nsIHardwareKeyHandler
>+{
>+public:
>+  HardwareKeyHandler();
>+
>+  // nsIHardwareKeyHandler will be held by JS(input method app), and

nit: insert a whitespace before |(|.

>+  // it will hold a nsIHardwareKeyEventListener referenced to JS.

nit: s/a nsI/an nsI

>+  // Depending on the implementation, this class may in a reference cycle,

s/may in/may be in ?

>+  // by |nsIHardwareKeyHandler.registerListener|, that will set a

the last "a" should be "an".

>+  [noscript] bool forwardKeyToInputMethodApp(in nsINodePtr aTarget,
>+                                             in WidgetKeyboardEventPtr aEvent,
>+                                             in nsEventStatusPtr aEventStatus);

So, please define with |%{C++| and |%}|.

>+  /**
>+   * Notifies nsIHardwareKeyHandler of the communcation channel between

s/communcation/communication

>+   * b2g process and the active input method app has been estabilshed. This

s/estabilshed/established

>diff --git a/layout/build/nsLayoutModule.cpp b/layout/build/nsLayoutModule.cpp

If you won't support nsIHardwareKeyHandler on non-B2G platforms, you can support the service only with #ifdef MOZ_B2G. Then, getting service in PresShell will return nullptr, I guess.

Comment 231

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8715182 [details] [diff] [review]
part7 - Interface between HardwareKeyHandler and Input  Method App

>+  // Send the data retrieved from native keyboard events
>+  // to input-method-app for generating new events.
>+  onHardwareKey: function onHardwareKeyReceived(evt) {
>+    // |event.preventDefault()| is allowed to be called
>+    // when |event.cancelable| is set to true.
>+    if (!evt.initDict.cancelable) {
>+      hardwareKeyHandler.onHandledByInputMethodApp(evt.type,
>+        Ci.nsIHardwareKeyHandler.NO_DEFAULT_PREENTED);

NO_DEFAULT_PREVENTED. (or misspelled at defining it?)

>+    }
>+    return this.sendToKeyboard('Keyboard:ReceiveHardwareKeyEvent', {
>+                               type: evt.type,
>+                               keyDict: evt.initDict
>+                              });

Looks like odd indent, but I'm not sure the rules of this module.

>+    // Notify the communication channel with the active input method app has
>+    // been estabilshed.

s/estabilshed/established

>+      case 'Keyboard:ReceiveHardwareKeyEvent':
>+        if (!this._inputcontext) {
>+          // The event doesn't be handled yet.
>+          this._inputcontext.replyHardwareKeyEvent(data.type, false);

I don't understand, looks like this is |null.replyHardwareKeyEvent(data.type, false);|, isn't it?

>+  // Return defaultPrevented's results of the event
>+  forwardHardwareKeyEvent: function ic_forwardHardwareKeyEvent(data) {
>+    if (this._context) {
>+      let evt = new this._window.KeyboardEvent(data.type,
>+                                               Cu.cloneInto(data.keyDict,
>+                                                            this._window));
>+      if (evt) {
>+        this._hardwareinput.__DOM_IMPL__.dispatchEvent(evt);
>+        return this._getDefaultPreventedValue(evt);
>+      }
>+    }
>+    return Ci.nsIHardwareKeyHandler.NO_DEFAULT_PREENTED;

Oh, looks like you defined this with misspelled word.

>+  },
>+
>+  _getDefaultPreventedValue: function(evt) {
>+    let flags = Ci.nsIHardwareKeyHandler.NO_DEFAULT_PREENTED;
>+
>+    if (evt.defaultPrevented) {
>+      flags |= Ci.nsIHardwareKeyHandler.DEFAULT_PREENTED;
>+    }
>+
>+    if (evt.defaultPreventedByChrome) {
>+      flags |= Ci.nsIHardwareKeyHandler.DEFAULT_PREENTED_BY_CHROME;
>+    }
>+
>+    if (evt.defaultPreventedByContent) {
>+      flags |= Ci.nsIHardwareKeyHandler.DEFAULT_PREENTED_BY_CONTENT;
>+    }

And the others too.

I guess I should retry to review this or next patch. My brain is very tired today...

Comment 232

[C.M.Chang[:chunmin]]

Attachment: [v2] part3 - Interface between PresShell and HardwareKeyHandler

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #227)
> Comment on attachment 8715158 [details] [diff] [review]
> part3 - Interface between PresShell and HardwareKeyHandler
> 
> >-// f17842ee-f1f0-4193-814f-70d706b67060
> >+// 3a0de342-93f6-4a40-9c77-574662bb2a43
> > #define NS_IPRESSHELL_IID \
> >-{ 0xf17842ee, 0xf1f0, 0x4193, \
> >-  { 0x81, 0x4f, 0x70, 0xd7, 0x06, 0xb6, 0x70, 0x60 } }
> >+{ 0x3a0de342, 0x93f6, 0x4a40, \
> >+  { 0x9c, 0x77, 0x57, 0x46, 0x62, 0xbb, 0x2a, 0x43 } }
> 
> FYI: Now, we don't need to modify uuid only in mozilla-central.
> https://groups.google.com/forum/#!topic/mozilla.dev.platform/HE1_qZhPj1I
> 
This will be landed into mozilla-central(not v2.2 for red-tai), so I think I don't need to modify the UUID, right?

Comment 233

[C.M.Chang[:chunmin]]

Attachment: [v2] part3 - Interface between PresShell and HardwareKeyHandler

Comment 234

[C.M.Chang[:chunmin]]

Attachment: [v2] part5 - Expose KeyboardEventInit dictionary r=masayuki, smaug

Carry r+ after using the methods to set the properties in KeyboardEventInit

(In reply to Olli Pettay [:smaug] (high review load) from comment #223)
> Comment on attachment 8715179 [details] [diff] [review]
> part5 - Expose KeyboardEventInit dictionary
> 
> mEvent shouldn't be null ever.
> 
> I wonder how to keep KeyboardEvent::GetInitDict updated if/when new stuff is
> added to KeyboardEventInit or any dictionaries it inherits.
> I don't really have good ideas for that.
I don't have better solution now, but I'll update it when I have one.

Comment 235

[C.M.Chang[:chunmin]]

Attachment: [v2] part7 - Interface between HardwareKeyHandler and Input Method App

Comment 236

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Chun-Min Chang[:chunmin] from comment #232)
> Created attachment 8719478 [details] [diff] [review]
> [v2] part3 - Interface between PresShell and HardwareKeyHandler
> 
> (In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #227)
> > Comment on attachment 8715158 [details] [diff] [review]
> > part3 - Interface between PresShell and HardwareKeyHandler
> > 
> > >-// f17842ee-f1f0-4193-814f-70d706b67060
> > >+// 3a0de342-93f6-4a40-9c77-574662bb2a43
> > > #define NS_IPRESSHELL_IID \
> > >-{ 0xf17842ee, 0xf1f0, 0x4193, \
> > >-  { 0x81, 0x4f, 0x70, 0xd7, 0x06, 0xb6, 0x70, 0x60 } }
> > >+{ 0x3a0de342, 0x93f6, 0x4a40, \
> > >+  { 0x9c, 0x77, 0x57, 0x46, 0x62, 0xbb, 0x2a, 0x43 } }
> > 
> > FYI: Now, we don't need to modify uuid only in mozilla-central.
> > https://groups.google.com/forum/#!topic/mozilla.dev.platform/HE1_qZhPj1I
> > 
> This will be landed into mozilla-central(not v2.2 for red-tai), so I think I
> don't need to modify the UUID, right?

Yes. Only when you need to uplift the patch, you need to do that.

Comment 237

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8719482 [details] [diff] [review]
[v2] part3 - Interface between PresShell and HardwareKeyHandler

>+#ifdef MOZ_B2G
>+bool
>+PresShell::ForwardKeyToInputMethodApp(nsINode* aTarget,
>+                                      WidgetKeyboardEvent& aEvent,
>+                                      nsEventStatus* aStatus)
>+{
>+  if (!XRE_IsParentProcess() || aEvent.mIsSynthesizedByTIP) {
>+    return false;
>+  }
>+
>+  nsresult rv;
>+  mHardwareKeyHandler = do_GetService("@mozilla.org/HardwareKeyHandler;1", &rv);
>+  if (!NS_SUCCEEDED(rv) || !mHardwareKeyHandler) {
>+    return false;
>+  }

nit: Do you really need to get service even when mHardwareKeyHandler isn't nullptr?

>+
>+  if (mHardwareKeyHandler->ForwardKeyToInputMethodApp(aTarget,
>+                                                      aEvent.AsKeyboardEvent(),
>+                                                      aStatus)) {
>+    // No need to dispatch the forwarded keyboard event to it's child process
>+    aEvent.mFlags.mNoCrossProcessBoundaryForwarding = true;
>+    return true;
>+  }
>+
>+  return false;
>+}
>+#endif // MOZ_B2G
>+
>+bool
>+PresShell::FinishDispatchingKeyboardEvent(bool aIsTargetRemote,
>+                                          nsINode* aTarget,
>+                                          WidgetKeyboardEvent& aEvent,
>+                                          nsEventStatus* aStatus,
>+                                          EventDispatchingCallback* aEventCB)

I still don't like this name, but no better name idea.

E.g., MaybeForwardKeyToInputMethodApp() sounds like it won't dispatch keyboard event normally. On the other hand, MaybeForwardKeyToInputMethodAppAndDispatch() is too long and sounds like redundant...

>+{
>+#ifndef MOZ_B2G
>+  // No need to forward to input-method-app if the platform isn't run on B2G.
>+  EventDispatcher::Dispatch(aTarget, mPresContext,
>+                            &aEvent, nullptr, aStatus, aEventCB);
>+  return false;
>+#endif // MOZ_B2G

Could you use #else here?

>+  // This method is used to determine whether or not we should stop
>+  // event dispatching when input-method-editor(IME) can get the event first
>+  // when the platform is run on Firefox OS.
>+  // If the event target is in the current process, the return value
>+  // is depended on whether the event is forwarded to IME successfully.
>+  // On the other hand, when the event target is in its child process,
>+  // before IME gets the event, the event will be dispatched to
>+  // the embedding iframe first. After dispatching,
>+  // if the |event.preventDefault()| is called, then return and
>+  // do further processing. Otherwise, the return value is
>+  // depended on whether the event is forwarded to IME successfully.
>+  bool FinishDispatchingKeyboardEvent(bool aIsTargetRemote,
>+                                      nsINode* aTarget,
>+                                      mozilla::WidgetKeyboardEvent& aEvent,
>+                                      nsEventStatus* aStatus,
>+                                      mozilla::EventDispatchingCallback* aEventCB);

I hope that you explicitly explain when true and false are returned.

And this patch changes nsIPresShell. Could you request sr to smaug?

Comment 238

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8719495 [details] [diff] [review]
[v2] part7 - Interface between HardwareKeyHandler and Input Method App

>+  // Send the initialized dictionary retrieved from the native keyboard event
>+  // to input-method-app for generating a new event.
>+  onHardwareKey: function onHardwareKeyReceived(evt) {
>+    // |event.preventDefault()| is allowed to be called
>+    // when |event.cancelable| is set to true.
>+    if (!evt.initDict.cancelable) {

Why don't you just check evt.cancelable?

>+      hardwareKeyHandler.onHandledByInputMethodApp(evt.type,
>+                                                   Ci.nsIHardwareKeyHandler
>+                                                     .NO_DEFAULT_PREVENTED);
>+    }

Well, if it's not cancelable, you return the result to hardwareKeyHandler first.

However...

>+      case 'Keyboard:ReplyHardwareKeyEvent':
>+        hardwareKeyHandler.onHandledByInputMethodApp(msg.data.type,
>+                                                     msg.data.defaultPrevented);
>+        break;

Looks like you don't check if the event was cancelable.  So, it seems that if non-cencelable keyboard event is fired (although, I have no idea of such situation), hardwareKeyHandler receives the result twice?

If I'm wrong, I'll give r+. Please request r? again.

>+  // Generate a new keyboard event by the received keyboard dictionary
>+  // and return defaultPrevented's result of the event after dispatching.
>+  forwardHardwareKeyEvent: function ic_forwardHardwareKeyEvent(data) {
>+    if (this._context) {
>+      let evt = new this._window.KeyboardEvent(data.type,
>+                                               Cu.cloneInto(data.keyDict,
>+                                                            this._window));
>+      if (evt) {

Can this be null?

Comment 239

[C.M.Chang[:chunmin]]

Attachment: [v2] part4 - HardwareKeyHandler component

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #230)
> Comment on attachment 8715178 [details] [diff] [review]
> part4 - HardwareKeyHandler component
> 
> >+NS_IMETHODIMP
> >+HardwareKeyHandler::ForwardKeyToInputMethodApp(nsINode* aTarget,
> >+                                               WidgetKeyboardEvent* aEvent,
> >+                                               nsEventStatus* aEventStatus,
> >+                                               bool* aResult)
> 
> Oh... Sorry, I probably misled you. Actually, I said you should use
> [noscript] for this method definition.
> 
> But you can define C++ method directly in xpidl. For example, see here:
> http://mxr.mozilla.org/mozilla-central/source/dom/interfaces/events/
> nsIDOMEventTarget.idl#107
> 
> Then, you can specify the result type as bool. Then, you can make
> implementation of this method and the users simpler. Sorry for my mistake. I
> should've realized that.
I am very grateful for your mentorship and toleration for my mistakes to make the code more solid.
I learn much from your wise advises and guiding, so it doesn't matter to me about this.

> >+{
> >+  MOZ_ASSERT(aTarget, "No target provided");
> >+  MOZ_ASSERT(aEvent, "No event provided");
> >+
> >+  // No need to forward hardware key event to IME if:
> >+  // 0. key's defaultPrevented is true;
> >+  // 1. IME is disabled
> >+  // 2. There is no nsIHardwareKeyEventListener in use
> >+  // 3. This key event is generated by IME(from nsITextInputProcessor)
> >+  // 4. The key event is already handling
> >+  if (aEvent->mFlags.mDefaultPrevented ||
> >+      !mInputMethodAppConnected ||
> >+      !mHardwareKeyEventListener ||
> >+      aEvent->mIsSynthesizedByTIP ||
> >+      aEvent->mInputMethodAppState != WidgetKeyboardEvent::eNotHandled) {
> >+    *aResult = false;
> >+    return NS_OK;
> >+  }
> 
> Then, you can make this simpler, even though the number of lines are
> increased:
> 
> // No need to forward hardware key event to IME if key's defaultPrevented is
> true.
> if (aEvent->mFlags.mDefaultPrevented) {
>   return false;
> }
> // No need to forward hardware key event to IME if IME is disabled.
> if (!mInputMethodAppConnected) {
>   return false;
> }
> ...
> 
> This style makes maintaining the if state and reading the condition easier.
Done.

> >+    // Create a Keyboard event for nsIHardwareKeyEventListener.onHardwareKey
> >+    nsCOMPtr<EventTarget> evtTarget = do_QueryInterface(aTarget);
> 
> nit: eventTarget is better.
> 
> >+    nsPresContext* presCtx = GetPresContext(aTarget);
> 
> nit: presContext is better.
> 
> >+    KeyboardEvent* keyEvt = new KeyboardEvent(evtTarget, presCtx, aEvent);
> 
> nit: keyboardEvent is better.
Done.

> >+  nsEventStatus status = aEventStatus? *aEventStatus : nsEventStatus_eIgnore;
> 
> nit: insert a whitespace before |?|.
Done.

> >+  KeyboardInfo* copiedInfo =
> >+    new KeyboardInfo(new WidgetKeyboardEvent(*aEvent), aTarget, &status);
> 
> You create an instance of WidgetKeyboardEvent here. But looks like that
> nobody (including KeyboardInfo) doesn't destroy it.
Thank you for your reminder, this will be deleted in the deconstructor of KeyboardInfo.
 
> >+  size_t sizeBefore = mEventQueue.GetSize();
> >+  mEventQueue.Push(copiedInfo);
> >+  NS_ENSURE_TRUE(mEventQueue.GetSize() == sizeBefore + 1,
> >+                 NS_ERROR_OUT_OF_MEMORY);
> 
> Could you not use NS_ENSURE_* anymore? And returning false or calling
> MOZ_CRASH() may be better.
OK, I will return false instead.
 
> >+    // Dispatch the keypress to its original event target if:
> 
> So, we need agreement about this...
> 
> I'm thinking that if target app is changed, we should stop dispatching
> remaining keyboard events for preventing unexpected operation in the new app.
> 
> Otherwise, i.e., focus is changed in the app, we should fine keyboard event
> in new focused element for conforming to the spec.
The event dispatching will be stopped if the focused element is changed in my current implementation.

> >+void
> >+HardwareKeyHandler::PostHandleKeyboardEvent(nsINode* aTarget,
> >+                                            WidgetKeyboardEvent* aEvent,
> >+                                            nsEventStatus* aStatus)
> >+{
> >+  MOZ_ASSERT(aTarget, "No target provided");
> >+  MOZ_ASSERT(aEvent, "No event provided");
> >+
> >+  nsPresContext* presCtx = GetPresContext(aTarget);
> 
> nit: presContext is better.
> 
> >+
> >+  RefPtr<mozilla::EventStateManager> manager = presCtx->EventStateManager();
> 
> Unfortunately, we don't have good agreement for this name... manager sounds
> not odd.
OK, maybe I can use 'esm' like [0]?
[0] https://dxr.mozilla.org/mozilla-central/source/layout/base/nsPresShell.cpp#2869

> >+  if (NS_WARN_IF(!manager)) {
> >+    return;
> >+  }
> 
> If the method to retrieve a pointer to object isn't prefixed by "Get", the
> result cannot be nullptr. So, MOZ_ASSERT() may be enough here. As far as I
> know, nsPresContext::mEventStateManager must not be nullptr until it's being
> destroyed.
OK, I think I should take notes about this...

> 
> >+struct KeyboardInfo
> >+{
> >+  WidgetKeyboardEvent* mEvent;
> >+  nsINode* mTarget;
> >+  nsEventStatus* mStatus;
> >+
> >+  KeyboardInfo(WidgetKeyboardEvent* aEvent,
> >+               nsINode* aTarget,
> >+               nsEventStatus* aStatus)
> >+  {
> >+    mEvent = aEvent;
> >+    mTarget = aTarget;
> >+    mStatus = aStatus;
> >+  }
> >+};
> 
> So, if you want this to *own* mEvent, there should be a flag if this owns
> it. If it's owning the event, mEvent should be deleted in destructor.
Well, I can't get your points about the 'flag' here. 
Should I use a flag variable in KeyboardInfo to state which pointers is owned by this module itself?
Instead of using flag, could I use other variable name like 'mCopiedEvent' to make it clearer?

> >+
> >+class HardwareKeyHandler : public nsIHardwareKeyHandler
> >+{
> >+public:
> >+  HardwareKeyHandler();
> >+
> >+  // nsIHardwareKeyHandler will be held by JS(input method app), and
> 
> nit: insert a whitespace before |(|.
> 
> >+  // it will hold a nsIHardwareKeyEventListener referenced to JS.
> 
> nit: s/a nsI/an nsI
> 
> >+  // Depending on the implementation, this class may in a reference cycle,
> 
> s/may in/may be in ?
> 
> >+  // by |nsIHardwareKeyHandler.registerListener|, that will set a
> 
> the last "a" should be "an".
Done.

> >+  [noscript] bool forwardKeyToInputMethodApp(in nsINodePtr aTarget,
> >+                                             in WidgetKeyboardEventPtr aEvent,
> >+                                             in nsEventStatusPtr aEventStatus);
> 
> So, please define with |%{C++| and |%}|.
OK.

> >+  /**
> >+   * Notifies nsIHardwareKeyHandler of the communcation channel between
> 
> s/communcation/communication
> 
> >+   * b2g process and the active input method app has been estabilshed. This
> 
> s/estabilshed/established
> 
> >diff --git a/layout/build/nsLayoutModule.cpp b/layout/build/nsLayoutModule.cpp
> 
> If you won't support nsIHardwareKeyHandler on non-B2G platforms, you can
> support the service only with #ifdef MOZ_B2G. Then, getting service in
> PresShell will return nullptr, I guess.
OK.

Comment 240

[C.M.Chang[:chunmin]]

Attachment: [v3] part7 - Interface between HardwareKeyHandler and Input Method App

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #238)
> Comment on attachment 8719495 [details] [diff] [review]
> [v2] part7 - Interface between HardwareKeyHandler and Input Method App
> 
> >+  // Send the initialized dictionary retrieved from the native keyboard event
> >+  // to input-method-app for generating a new event.
> >+  onHardwareKey: function onHardwareKeyReceived(evt) {
> >+    // |event.preventDefault()| is allowed to be called
> >+    // when |event.cancelable| is set to true.
> >+    if (!evt.initDict.cancelable) {
> 
> Why don't you just check evt.cancelable?
> 
> >+      hardwareKeyHandler.onHandledByInputMethodApp(evt.type,
> >+                                                   Ci.nsIHardwareKeyHandler
> >+                                                     .NO_DEFAULT_PREVENTED);
> >+    }
> 
> Well, if it's not cancelable, you return the result to hardwareKeyHandler
> first.
> 
> However...
> 
> >+      case 'Keyboard:ReplyHardwareKeyEvent':
> >+        hardwareKeyHandler.onHandledByInputMethodApp(msg.data.type,
> >+                                                     msg.data.defaultPrevented);
> >+        break;
> 
> Looks like you don't check if the event was cancelable.
I think the |event.cancelable| can be checked in HardwareKeyHandler::ForwardKeyToInputMethodApp directly by
|aEvent->mFlags.mCancelable|. I will add this condition in the next version of HardwareKeyHandler component.

> >+  // Generate a new keyboard event by the received keyboard dictionary
> >+  // and return defaultPrevented's result of the event after dispatching.
> >+  forwardHardwareKeyEvent: function ic_forwardHardwareKeyEvent(data) {
> >+    if (this._context) {
> >+      let evt = new this._window.KeyboardEvent(data.type,
> >+                                               Cu.cloneInto(data.keyDict,
> >+                                                            this._window));
> >+      if (evt) {
> 
> Can this be null?
I think it won't be null, so I remove this line.

Comment 241

[C.M.Chang[:chunmin]]

Attachment: [v3] part4 - HardwareKeyHandler component

There are some changes in this version
1. Check |event.cancelable| in ForwardKeyToInputMethodApp
2. Add a member variable |latestKeyDownInfo| to store the latest keydown's info

The reason why I add |latestKeyDownInfo| is that:
In previous implementation, the keypress event will be withdrawn from the event queue to be handled only when their heading keydown reply arrives. However, if the keydown's reply arrives before its following keypress event comes, then there is no existing keypress in queue can be processed.

That is, if IME replies the keydown event through |OnHandledByInputMethodApp| before the following keypress is passed into |ForwardKeyToInputMethodApp|, then there is no keypress in event queue when |OnHandledByInputMethodApp| for keydown is called. In addition, the keypress comes after the |OnHandledByInputMethodApp| will never be handled.

Thus, I propose to use a variable |latestKeyDownInfo| to hold the latest keydown's data. Once keypress comes, we can use it to decide how to process.

Comment 242

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8720152 [details] [diff] [review]
[v3] part4 - HardwareKeyHandler component

>+  // No need to forward hardware key event to IME
>+  // if the event is not cancelable.
>+  // If |event.cancelable| is false, then |event.preventDefault()| isn't
>+  // allowed to be called. It means the IME won't be allowed to handle
>+  // the forwarded keys.
>+  if (!aEvent->mFlags.mCancelable) {
>+    return false;
>+  }

This sounds odd to me because IME won't be able to catch key events which are not cancelable. However, I think that this is okay because as I said, normal keyboard event should be always cancelable. So, this can be used for hack when somebody needs to do that.

>+  // For those keypress events coming after their heading keydown's reply
>+  // already arrives, they should be dispatched directly instead of
>+  // being stored into the event queue. Otherwise, without the heading keydown
>+  // in the event queue, the stored keypress never be withdrawn to be fired.
>+  if (aEvent->mMessage == eKeyPress && !mEventQueue.GetSize()) {
>+    DispatchKeyPress(aTarget, aEvent, aEventStatus);
>+    return true;

Hmm, this isn't actually forward the event to IME but returns true. I think that you need to update the document of ForwardKeyToInputMethodApp().

>+  // Push the key event into queue for reuse when its reply arrives.
>+  nsEventStatus *status =
>+    new nsEventStatus(aEventStatus ? *aEventStatus : nsEventStatus_eIgnore);
>+  KeyboardInfo* copiedInfo =
>+    new KeyboardInfo(new WidgetKeyboardEvent(*aEvent), aTarget, status);

Looks like that here is the only place to create KeyboardInfo. However, the new KeyboardInfo owns the pointers sent via arguments of the constructor:

>+struct KeyboardInfo
>+{
>+  WidgetKeyboardEvent* mEvent;
>+  nsINode* mTarget;
>+  nsEventStatus* mStatus;
>+
>+  KeyboardInfo(WidgetKeyboardEvent* aEvent,
>+               nsINode* aTarget,
>+               nsEventStatus* aStatus)
>+  {
>+    mEvent = aEvent;
>+    mTarget = aTarget;
>+    mStatus = aStatus;
>+  }
>+
>+  ~KeyboardInfo()
>+  {
>+    delete mEvent;
>+    delete mStatus;
>+  }

Looks like this is pretty dangerous. Other developers may set unsafe pointers which are owned by the others.

How about this?

KeyboardInfo(const WidgetKeyboardEvent& aEvent,
             nsINode* aTarget,
             nsEventStatus aStatus)
  : mEvent(new WidgetKeyboardEvent(aEvent))
  , mTarget(aTarget)
  , mStatus(aStatus)
{
}

And I don't understand why mStatus should be pointer and owned. Isn't it enough to be a non-pointer member of it? nsEventStatus is an enum, not struct/class.

>+  // Update the latest keydown data:
>+  //   Transfer the ownership of current keydown's data from keyInfo
>+  //   to latestKeyDownInfo and release the previous keydown's data.
>+  latestKeyDownInfo = keyInfo;

mLatestKeyDownInfo ?

keyInfo must be nullptr, no? But this is NOT looked like so. I have no idea clearer code, so, could you add a warning comment after the line?

// WARNING: keyInfo is now nullptr.

>+bool
>+HardwareKeyHandler::DispatchKeyPress(nsINode* aTarget,
>+                                     WidgetKeyboardEvent* aEvent,
>+                                     nsEventStatus* aStatus)
>+{
>+  MOZ_ASSERT(aTarget, "No target provided");
>+  MOZ_ASSERT(aEvent, "No event provided");
>+  MOZ_ASSERT(aEvent->mMessage == eKeyPress, "Event is not keypress");
>+  MOZ_ASSERT(!latestKeyDownInfo, "No data for the latest keydown");

It seems that this is currently impossible situation but I feel pretty risky. Could you remove this MOZ_ASSERT() and check mLatestKeyDownInfo before accessing its member?

>+  // No need to dispatch keypress to the event target
>+  // if the keydown event is consumed by the input-method-app.
>+  if (latestKeyDownInfo->mEvent->mFlags.mDefaultPrevented) {

So, e.g., |if (mLatestKeyDownInfo &&
               nKatestKeyDownInfo->mEvent->mFlags.mDefaultPrevented) {|

>+    return false;
>+  }
>+
>+  // No need to dispatch keypress to the event target
>+  // if the previous keydown event is modifier key's
>+  if (latestKeyDownInfo->mEvent->IsModifierKeyEvent()) {

Same. (I think that mLatestKeyDownInfo should be checked before these if blocks, though.)

>+    return false;
>+  }
>+
>+  // No need to dispatch keypress to the event target
>+  // if the key is non-printable
>+  if (!aEvent->isChar) {
>+    return false;
>+  }

This is really different behavior than other platforms. According to the current UI Events spec, *we* should not dispatch keypress events if the key doesn't input any text. However, we should change the behavior on all platforms at once. So, please just remove this check (At that time, gonk widget shouldn't dispatch keypress events for such keys).

>+  // The event target should be set to the current focused element.
>+  // However, it might have security issue if the event is dispatched to
>+  // the unexpected application, and it might cause unexpected operation
>+  // in the new app.
>+  nsIContent* originalTarget = aTarget->AsContent();
>+  if (originalTarget != currentTarget) {
>+    NS_WARNING("The focus element is changed during the event is dispatching");
>+    return false;
>+  }

Don't we need to check if originalTarget and currentTarget are hosts of difference processes? If there is no such cases, this is okay. But if it's possible, we should fire key event on current focused element if focus wasn't moved across process boundary.


>> >+struct KeyboardInfo
>> >+{
>> >+  WidgetKeyboardEvent* mEvent;
>> >+  nsINode* mTarget;
>> >+  nsEventStatus* mStatus;
>> >+
>> >+  KeyboardInfo(WidgetKeyboardEvent* aEvent,
>> >+               nsINode* aTarget,
>> >+               nsEventStatus* aStatus)
>> >+  {
>> >+    mEvent = aEvent;
>> >+    mTarget = aTarget;
>> >+    mStatus = aStatus;
>> >+  }
>> >+};
>> 
>> So, if you want this to *own* mEvent, there should be a flag if this owns
>> it. If it's owning the event, mEvent should be deleted in destructor.
> Well, I can't get your points about the 'flag' here. 
> Should I use a flag variable in KeyboardInfo to state which pointers is owned by this module itself?

I think that if you think that it can both own and not own the instance, you should manage it with a bool flag. But my above suggestion, i.e., if mEvent is always owned by KeyboardInfo, we don't need to do that. But you should prevent the creator of KeyboardInfo to set unsafe pointer to mEvent.

> Instead of using flag, could I use other variable name like 'mCopiedEvent' to make it clearer?

I don't think that it's necessary.

BTW, I worry about heap memory fragmentation of parent process by creating WidgetKeyboardEvent per hardware key event. But I have no better idea for solving this issue without complicated memory management (e.g., storing WidgetKeyboardEvent instances and reuse it).

Comment 243

[Tim Guan-tin Chien [:timdream] (please needinfo)]

I am sorry I did not have time to review this. Feel free to ni me if there are further questions.

Comment 244

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8707271 [details] [diff] [review]
[WIP] Simulate hardware keystroke in mochitest

If you still need to ask something to me about this patch, please request ni? again.

Comment 245

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #242)
> Comment on attachment 8720152 [details] [diff] [review]
> [v3] part4 - HardwareKeyHandler component
> 
> BTW, I worry about heap memory fragmentation of parent process by creating
> WidgetKeyboardEvent per hardware key event. But I have no better idea for
> solving this issue without complicated memory management (e.g., storing
> WidgetKeyboardEvent instances and reuse it).
I think memory pool might be helpful. I'll implement it in next version. Thanks for your review.

Comment 246

[C.M.Chang[:chunmin]]

Attachment: [v4] part4 - HardwareKeyHandler component

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #242)
> Comment on attachment 8720152 [details] [diff] [review]
> [v3] part4 - HardwareKeyHandler component
> 
> >+  // No need to forward hardware key event to IME
> >+  // if the event is not cancelable.
> >+  // If |event.cancelable| is false, then |event.preventDefault()| isn't
> >+  // allowed to be called. It means the IME won't be allowed to handle
> >+  // the forwarded keys.
> >+  if (!aEvent->mFlags.mCancelable) {
> >+    return false;
> >+  }
> 
> This sounds odd to me because IME won't be able to catch key events which
> are not cancelable. However, I think that this is okay because as I said,
> normal keyboard event should be always cancelable. So, this can be used for
> hack when somebody needs to do that.
OK, I move it back to IME part and let IME decides whether or not it's ok to forward the event.

> >+  // For those keypress events coming after their heading keydown's reply
> >+  // already arrives, they should be dispatched directly instead of
> >+  // being stored into the event queue. Otherwise, without the heading keydown
> >+  // in the event queue, the stored keypress never be withdrawn to be fired.
> >+  if (aEvent->mMessage == eKeyPress && !mEventQueue.GetSize()) {
> >+    DispatchKeyPress(aTarget, aEvent, aEventStatus);
> >+    return true;
> 
> Hmm, this isn't actually forward the event to IME but returns true. I think
> that you need to update the document of ForwardKeyToInputMethodApp().
OK, I change the document in idl.
 
> >+  // Push the key event into queue for reuse when its reply arrives.
> >+  nsEventStatus *status =
> >+    new nsEventStatus(aEventStatus ? *aEventStatus : nsEventStatus_eIgnore);
> >+  KeyboardInfo* copiedInfo =
> >+    new KeyboardInfo(new WidgetKeyboardEvent(*aEvent), aTarget, status);
> 
> Looks like that here is the only place to create KeyboardInfo. However, the
> new KeyboardInfo owns the pointers sent via arguments of the constructor:
> 
> >+struct KeyboardInfo
> >+{
> >+  WidgetKeyboardEvent* mEvent;
> >+  nsINode* mTarget;
> >+  nsEventStatus* mStatus;
> >+
> >+  KeyboardInfo(WidgetKeyboardEvent* aEvent,
> >+               nsINode* aTarget,
> >+               nsEventStatus* aStatus)
> >+  {
> >+    mEvent = aEvent;
> >+    mTarget = aTarget;
> >+    mStatus = aStatus;
> >+  }
> >+
> >+  ~KeyboardInfo()
> >+  {
> >+    delete mEvent;
> >+    delete mStatus;
> >+  }
> 
> Looks like this is pretty dangerous. Other developers may set unsafe
> pointers which are owned by the others.
> 
> How about this?
> 
> KeyboardInfo(const WidgetKeyboardEvent& aEvent,
>              nsINode* aTarget,
>              nsEventStatus aStatus)
>   : mEvent(new WidgetKeyboardEvent(aEvent))
>   , mTarget(aTarget)
>   , mStatus(aStatus)
> {
> }
> 
> And I don't understand why mStatus should be pointer and owned. Isn't it
> enough to be a non-pointer member of it? nsEventStatus is an enum, not
> struct/class.
mEvent and mStatus is declared as a member variable instead of member pointer in this version to avoid the memory fragmentation issue. 
 
> >+  // Update the latest keydown data:
> >+  //   Transfer the ownership of current keydown's data from keyInfo
> >+  //   to latestKeyDownInfo and release the previous keydown's data.
> >+  latestKeyDownInfo = keyInfo;
> 
> mLatestKeyDownInfo ?
> 
> keyInfo must be nullptr, no? But this is NOT looked like so. I have no idea
> clearer code, so, could you add a warning comment after the line?
> 
> // WARNING: keyInfo is now nullptr.
How about using shared_ptr instead of autoPtr? With shared_ptr, the mLatestKeyDownInfo is still be added/released automatically, and keyInfo won't be nullptr after assigning it to others.

> >+bool
> >+HardwareKeyHandler::DispatchKeyPress(nsINode* aTarget,
> >+                                     WidgetKeyboardEvent* aEvent,
> >+                                     nsEventStatus* aStatus)
> >+{
> >+  MOZ_ASSERT(aTarget, "No target provided");
> >+  MOZ_ASSERT(aEvent, "No event provided");
> >+  MOZ_ASSERT(aEvent->mMessage == eKeyPress, "Event is not keypress");
> >+  MOZ_ASSERT(!latestKeyDownInfo, "No data for the latest keydown");
> 
> It seems that this is currently impossible situation but I feel pretty
> risky. Could you remove this MOZ_ASSERT() and check mLatestKeyDownInfo
> before accessing its member?
> 
> >+  // No need to dispatch keypress to the event target
> >+  // if the keydown event is consumed by the input-method-app.
> >+  if (latestKeyDownInfo->mEvent->mFlags.mDefaultPrevented) {
> 
> So, e.g., |if (mLatestKeyDownInfo &&
>                nKatestKeyDownInfo->mEvent->mFlags.mDefaultPrevented) {|
> 
> >+    return false;
> >+  }
> >+
> >+  // No need to dispatch keypress to the event target
> >+  // if the previous keydown event is modifier key's
> >+  if (latestKeyDownInfo->mEvent->IsModifierKeyEvent()) {
> 
> Same. (I think that mLatestKeyDownInfo should be checked before these if
> blocks, though.)
> 
> >+    return false;
> >+  }
OK.

> >+  // No need to dispatch keypress to the event target
> >+  // if the key is non-printable
> >+  if (!aEvent->isChar) {
> >+    return false;
> >+  }
OK. This line is removed. 

> This is really different behavior than other platforms. According to the
> current UI Events spec, *we* should not dispatch keypress events if the key
> doesn't input any text. However, we should change the behavior on all
> platforms at once. So, please just remove this check (At that time, gonk
> widget shouldn't dispatch keypress events for such keys).
> 
> >+  // The event target should be set to the current focused element.
> >+  // However, it might have security issue if the event is dispatched to
> >+  // the unexpected application, and it might cause unexpected operation
> >+  // in the new app.
> >+  nsIContent* originalTarget = aTarget->AsContent();
> >+  if (originalTarget != currentTarget) {
> >+    NS_WARNING("The focus element is changed during the event is dispatching");
> >+    return false;
> >+  }
> 
> Don't we need to check if originalTarget and currentTarget are hosts of
> difference processes? If there is no such cases, this is okay. But if it's
> possible, we should fire key event on current focused element if focus
> wasn't moved across process boundary.
You're right. How about using outer window to check it?
Check which process the application is located might not be enough. The two different applications are possible in the same process(e.g., two in-process app).
 
> BTW, I worry about heap memory fragmentation of parent process by creating
> WidgetKeyboardEvent per hardware key event. But I have no better idea for
> solving this issue without complicated memory management (e.g., storing
> WidgetKeyboardEvent instances and reuse it).
I implement a simple memory pool to allocate memory from heap. With it, only n+2 memory chunk at most will be needed, which n is the times of keypress events.

Comment 247

[C.M.Chang[:chunmin]]

Attachment: [v4] part7 - Interface between HardwareKeyHandler and Input Method App

Add condition to check |event.cancelable|, so it's needed to be reviewed again.

Comment 248

[C.M.Chang[:chunmin]]

Attachment: [v4] part4 - HardwareKeyHandler component

Adjust the order of .h in HardwareKeyHandler.cpp

Comment 249

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #199)
> If you need to dispatch hardware keyboard event, you should implement
> nsIDOMWindowUtils::SendNativeKeyEvent() in Gonk widget. However, I'm not
> sure if it's available with multi-process.
If we use nsIDOMWindowUtils::SendNativeKeyEvent() to simulate hardware key, then the |beforeKey*| and |afterKey*| will only be tested on Mac, Windows and Gonk(if it's really implemented).

Is it possible to use the nsIDOMWindowUtils::SendKeyEvent to fire key? 
The problem of ::SendKeyEvent is that it lacks for |event.key| and |event.code|. The easiest way to repair it is to pass their value directly, or use keyCode to map their value. However, |event.keyCode| has been removed from the Web standards...

Comment 250

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8726562 [details] [diff] [review]
[v4] part4 - HardwareKeyHandler component

>+struct KeyboardInfo : public RefCounted<KeyboardInfo>,
>+                      public MemoryPool<KeyboardInfo>
>+{
>+  nsINode* mTarget;
>+  WidgetKeyboardEvent mEvent;
>+  nsEventStatus mStatus;
>+
>+  KeyboardInfo(nsINode* aTarget,
>+               WidgetKeyboardEvent& aEvent,
>+               nsEventStatus aStatus)
>+    : mTarget(aTarget)
>+    , mEvent(WidgetKeyboardEvent(aEvent))

nit: Why don't you just write |mEvent(aEvent)|?

>diff --git a/dom/inputmethod/MemoryPool.h b/dom/inputmethod/MemoryPool.h
>new file mode 100644
>index 0000000..e4ff97b
>--- /dev/null
>+++ b/dom/inputmethod/MemoryPool.h
>@@ -0,0 +1,121 @@
>+/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
>+/* vim: set ts=8 sts=2 et sw=2 tw=80: */
>+/* This Source Code Form is subject to the terms of the Mozilla Public
>+ * License, v. 2.0. If a copy of the MPL was not distributed with this
>+ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
>+
>+// This is an implementation of a memory pool mechanism.
>+// The new/delete operator of the class that inherits MemoryPool<T> will be
>+// overridden for using this mechanism.
>+// The memory policy here is quite simple: There is a queue that holds all the
>+// free memory chunks. When new operator is called, then one of the available
>+// memory spaces held by the queue will be returned. However, if queue is empty,
>+// which means that there is no free memory chunk in the qeueu, then we will
>+// allocate a new memory chunk from heap and return it.
>+// On the other hand, when delete operator is executed, the caller's
>+// deconstructor will be executed and then the obsolete memory
>+// will be reclaimed to queue.
>+//
>+// Examle:
>+// struct S: public MemoryPool<S>
>+// {
>+//   ...
>+// };
>+//
>+// S* p = new S(...); // Get a free memory space and construct p there.
>+// delete p; // Call p's deconstructor and then reclaim p's memory space.
>+
>+#include <queue>
>+#include "mozilla/Move.h"  // For Move(), which is identical to std::move()
>+
>+template<typename T>
>+class BasicPool
>+{
>+public:
>+  BasicPool() {};
>+
>+  // Disallow the copy constructor
>+  BasicPool(const BasicPool &)=delete;
>+
>+  // The move constructor
>+  BasicPool(BasicPool&& other)
>+    : mChunks(Move(other.mChunks))
>+  {
>+  }
>+
>+  ~BasicPool()
>+  {
>+    Clear();
>+  }
>+
>+  // Allocate a chunk of memory for the class object
>+  void* Allocate()
>+  {
>+    // If there is no more available memory chunk,
>+    // then we create a new one
>+    if (mChunks.empty()) {
>+      return ::operator new(sizeof(T));
>+    }
>+    // Otherwise, return a free memory chunk
>+    void* chunk = static_cast<void*>(mChunks.front());
>+    mChunks.pop();
>+    return chunk;
>+  }
>+
>+  // Reclaim the memory space from those no longer used objects
>+  void Deallocate(void* obsolete)
>+  {
>+    mChunks.push(static_cast<T*>(obsolete));
>+  }
>+
>+  // Release all of the free memory chunks
>+  void Clear()
>+  {
>+    while (!mChunks.empty()) {
>+      ::operator delete(mChunks.front());
>+      mChunks.pop();
>+    }
>+  }
>+
>+private:
>+  // Hold the pointers to the free memory chunks
>+  std::queue<T*> mChunks;
>+};
>+
>+template<typename T, template<typename = T> class Pool = BasicPool>
>+class MemoryPool
>+{
>+public:
>+  // Override operator new
>+  static void* operator new(size_t size)
>+  {
>+    return mPool.Allocate();
>+  }
>+
>+  // Override operator delete
>+  static void operator delete(void* p)
>+  {
>+    mPool.Deallocate(p);
>+  }
>+
>+  // Clear the memory pool
>+  static void Clear()
>+  {
>+    mPool.Clear();
>+  }
>+
>+private:
>+  // Allow only the inherited class to call it
>+  MemoryPool() {};
>+  MemoryPool(const MemoryPool&) {};
>+
>+  // Allow the inherited class to access its private members
>+  friend T;
>+
>+  static Pool<T> mPool;
>+};
>+
>+// Define the static pool variable that will be called
>+// in constructor and deconstructor
>+template<typename T, template<typename = T> class Pool>
>+Pool<T> MemoryPool<T, Pool>::mPool;

Wow, this is really out of scope from me. Could you separate this new file and making KeyboardInfo use MemoryPool to a new patch? Then, I'll give r+ to current part.4 without the parts. And then, could you request a review to smaug or somebody? I cannot decide if it should be under dom/inputmethod and you really need to create it.

Comment 251

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

>> >+  // Update the latest keydown data:
>> >+  //   Transfer the ownership of current keydown's data from keyInfo
>> >+  //   to latestKeyDownInfo and release the previous keydown's data.
>> >+  latestKeyDownInfo = keyInfo;
>> 
>> mLatestKeyDownInfo ?
>> 
>> keyInfo must be nullptr, no? But this is NOT looked like so. I have no idea
>> clearer code, so, could you add a warning comment after the line?
>> 
>> // WARNING: keyInfo is now nullptr.
> How about using shared_ptr instead of autoPtr? With shared_ptr, the mLatestKeyDownInfo is still be
> added/released automatically, and keyInfo won't be nullptr after assigning it to others.

Oh, wait, you've changed from AutoPtr to RefPtr, but you keep using nsDeque, this looks tricky...

>   KeyboardInfo* copiedInfo =
>     new KeyboardInfo(aTarget,
>   ...
>   mEventQueue.Push(copiedInfo);

You wrote the patch as this.

But I like this better:

>   RefPtr<KeyboardInfo> copiedInfo =
>     new KeyboardInfo(aTarget,
>   ...
>   mEventQueue.Push(copiedInfo.forget());

And I want you to rewrite following code:

> delete static_cast<KeyboardInfo*>(mEventQueue.PopFront());

> mEventQueue.PopFront();

> keyInfo(static_cast<KeyboardInfo*>(mEventQueue.PopFront()));

But looks like very unsafe... Is not there type-safe queue class? If no, I really recommend you to use nsAutoTArray<RefPtr<KeyboardInfo>> with some helper methods.

>> This is really different behavior than other platforms. According to the
>> current UI Events spec, *we* should not dispatch keypress events if the key
>> doesn't input any text. However, we should change the behavior on all
>> platforms at once. So, please just remove this check (At that time, gonk
>> widget shouldn't dispatch keypress events for such keys).
>> 
>> >+  // The event target should be set to the current focused element.
>> >+  // However, it might have security issue if the event is dispatched to
>> >+  // the unexpected application, and it might cause unexpected operation
>> >+  // in the new app.
>> >+  nsIContent* originalTarget = aTarget->AsContent();
>> >+  if (originalTarget != currentTarget) {
>> >+    NS_WARNING("The focus element is changed during the event is dispatching");
>> >+    return false;
>> >+  }
>> 
>> Don't we need to check if originalTarget and currentTarget are hosts of
>> difference processes? If there is no such cases, this is okay. But if it's
>> possible, we should fire key event on current focused element if focus
>> wasn't moved across process boundary.
> You're right. How about using outer window to check it?
> Check which process the application is located might not be enough. The two different applications
> are possible in the same process(e.g., two in-process app).

Does it work will with focus is moved from an element in an iframe element to outer element? If you need to compare DOM windows, I think that you need to retrieve "root" DOM window of the application.

Comment 252

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Chun-Min Chang[:chunmin] from comment #249)
> (In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #199)
> > If you need to dispatch hardware keyboard event, you should implement
> > nsIDOMWindowUtils::SendNativeKeyEvent() in Gonk widget. However, I'm not
> > sure if it's available with multi-process.
> If we use nsIDOMWindowUtils::SendNativeKeyEvent() to simulate hardware key,
> then the |beforeKey*| and |afterKey*| will only be tested on Mac, Windows
> and Gonk(if it's really implemented).

Yeah, I think that it's enough for now.

> Is it possible to use the nsIDOMWindowUtils::SendKeyEvent to fire key? 
> The problem of ::SendKeyEvent is that it lacks for |event.key| and
> |event.code|. The easiest way to repair it is to pass their value directly,
> or use keyCode to map their value. However, |event.keyCode| has been removed
> from the Web standards...

Please forget nsIDOMWindowUtils::SendKeyEvent() which should've already been removed.

nsITextInputProcessor supports new KeyboardEvent attributes and synthesizeKey of EventUtils.js already use it:
http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/EventUtils.js?rev=489672415fb3&mark=681-683,688-689#676

For example:
http://mxr.mozilla.org/mozilla-central/source/browser/base/content/test/general/browser_selectpopup.js?rev=4247cb5030a6#42

Comment 253

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #251)
> >> >+  // Update the latest keydown data:
> >> >+  //   Transfer the ownership of current keydown's data from keyInfo
> >> >+  //   to latestKeyDownInfo and release the previous keydown's data.
> >> >+  latestKeyDownInfo = keyInfo;
> >> 
> >> mLatestKeyDownInfo ?
> >> 
> >> keyInfo must be nullptr, no? But this is NOT looked like so. I have no idea
> >> clearer code, so, could you add a warning comment after the line?
> >> 
> >> // WARNING: keyInfo is now nullptr.
> > How about using shared_ptr instead of autoPtr? With shared_ptr, the mLatestKeyDownInfo is still be
> > added/released automatically, and keyInfo won't be nullptr after assigning it to others.
> 
> Oh, wait, you've changed from AutoPtr to RefPtr, but you keep using nsDeque,
> this looks tricky...
> 
> >   KeyboardInfo* copiedInfo =
> >     new KeyboardInfo(aTarget,
> >   ...
> >   mEventQueue.Push(copiedInfo);
> 
> You wrote the patch as this.
> 
> But I like this better:
> 
> >   RefPtr<KeyboardInfo> copiedInfo =
> >     new KeyboardInfo(aTarget,
> >   ...
> >   mEventQueue.Push(copiedInfo.forget());
> 
> And I want you to rewrite following code:
> 
> > delete static_cast<KeyboardInfo*>(mEventQueue.PopFront());
> 
> > mEventQueue.PopFront();
> 
> > keyInfo(static_cast<KeyboardInfo*>(mEventQueue.PopFront()));
> 
> But looks like very unsafe... Is not there type-safe queue class? If no, I
> really recommend you to use nsAutoTArray<RefPtr<KeyboardInfo>> with some
> helper methods.
I think I can write a type safe wrapper around nsDeque for storing events' data.

> >> This is really different behavior than other platforms. According to the
> >> current UI Events spec, *we* should not dispatch keypress events if the key
> >> doesn't input any text. However, we should change the behavior on all
> >> platforms at once. So, please just remove this check (At that time, gonk
> >> widget shouldn't dispatch keypress events for such keys).
> >> 
> >> >+  // The event target should be set to the current focused element.
> >> >+  // However, it might have security issue if the event is dispatched to
> >> >+  // the unexpected application, and it might cause unexpected operation
> >> >+  // in the new app.
> >> >+  nsIContent* originalTarget = aTarget->AsContent();
> >> >+  if (originalTarget != currentTarget) {
> >> >+    NS_WARNING("The focus element is changed during the event is dispatching");
> >> >+    return false;
> >> >+  }
> >> 
> >> Don't we need to check if originalTarget and currentTarget are hosts of
> >> difference processes? If there is no such cases, this is okay. But if it's
> >> possible, we should fire key event on current focused element if focus
> >> wasn't moved across process boundary.
> > You're right. How about using outer window to check it?
> > Check which process the application is located might not be enough. The two different applications
> > are possible in the same process(e.g., two in-process app).
> 
> Does it work will with focus is moved from an element in an iframe element
> to outer element? If you need to compare DOM windows, I think that you need
> to retrieve "root" DOM window of the application.
Yes, you're right. 
The nsIDOMWindow now is divided into nsPIDOMWindowInner and nsPIDOMWindowOuter since bug 1241764 was landed. However, I can't successfully build b2g after updating to the latest one(since it's moved to tier 3), so I need sometime to find a workaround, and then I can rebase and test the code.

Comment 254

[C.M.Chang[:chunmin]]

Attachment: [v4.1] part4 - HardwareKeyHandler component

- Modification
  - Using a type safe wrapper around nsDeque for mEventQueue
  - Replace nsIDOMWindow with nsPIDOMWindowOuter
- Need to do:
  - Try to find the existing memory pool mechanism in gecko
    (I think it might be somewhere since it's a general problem)

Comment 255

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Chun-Min Chang[:chunmin] from comment #254)
> - Need to do:
>   - Try to find the existing memory pool mechanism in gecko
>     (I think it might be somewhere since it's a general problem)

I think that it's okay to file a new bug for this. I'm not sure if it's actually necessary.

Comment 256

[C.M.Chang[:chunmin]]

Attachment: [v5] part4 - HardwareKeyHandler component

- Drop the memory pool part
- Use a type-safe wrapper around nsDeque for mEventQueue

Comment 257

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8728754 [details] [diff] [review]
[v5] part4 - HardwareKeyHandler component

Looks good to me. But I'd like smaug to review this too.

Comment 258

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #255)
> (In reply to Chun-Min Chang[:chunmin] from comment #254)
> > - Need to do:
> >   - Try to find the existing memory pool mechanism in gecko
> >     (I think it might be somewhere since it's a general problem)
> 
> I think that it's okay to file a new bug for this. I'm not sure if it's
> actually necessary.
Hi Masayuki-san,
I asked khuey about this, and he said we might no need to worry about this problem. 
Our built-in |new| and |delete| already have policy to minimize the memory fragments.
We can ignore it now until performance profiler tell us to improve it.

http://mxr.mozilla.org/mozilla-central/source/memory/mozjemalloc/jemalloc.c#31

Comment 259

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Chun-Min Chang[:chunmin] from comment #258)
> (In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #255)
> > (In reply to Chun-Min Chang[:chunmin] from comment #254)
> > > - Need to do:
> > >   - Try to find the existing memory pool mechanism in gecko
> > >     (I think it might be somewhere since it's a general problem)
> > 
> > I think that it's okay to file a new bug for this. I'm not sure if it's
> > actually necessary.
> Hi Masayuki-san,
> I asked khuey about this, and he said we might no need to worry about this
> problem. 
> Our built-in |new| and |delete| already have policy to minimize the memory
> fragments.
> We can ignore it now until performance profiler tell us to improve it.
> 
> http://mxr.mozilla.org/mozilla-central/source/memory/mozjemalloc/jemalloc.
> c#31

Thank you, that's a good news!

Comment 260

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8719482 [details] [diff] [review]
[v2] part3 - Interface between PresShell and HardwareKeyHandler

>+PresShell::IsTargetIframe(nsINode* aTarget)
>+{
>+  nsCOMPtr<nsIContent> content(do_QueryInterface(aTarget));
>+  return content && content->IsHTMLElement(nsGkAtoms::iframe);
>+}
random comment, why the QI to content.
return aTarget && aTarget->IsHTMLElement(nsGkAtoms::iframe); should work just fine.



>+PresShell::ForwardKeyToInputMethodApp(nsINode* aTarget,
>+                                      WidgetKeyboardEvent& aEvent,
>+                                      nsEventStatus* aStatus)
>+{
>+  if (!XRE_IsParentProcess() || aEvent.mIsSynthesizedByTIP) {
>+    return false;
>+  }
>+
>+  nsresult rv;
>+  mHardwareKeyHandler = do_GetService("@mozilla.org/HardwareKeyHandler;1", &rv);
This is what I was looking for for reviewing part 4.
Ok, HardwareKeyHandler is a service, so making class HardwareKeyHandler
a cycle collectable object doesn't really help much, since services will be kept alive until shutdown.
(I know, we do have some cycle collectable services, like nsFocusManager, but we should get rid of them.)

Comment 261

[Olli Pettay [:smaug][[e-mail]]]

(In reply to Olli Pettay [:smaug] from comment #260)
> >+PresShell::ForwardKeyToInputMethodApp(nsINode* aTarget,
> >+                                      WidgetKeyboardEvent& aEvent,
> >+                                      nsEventStatus* aStatus)
> >+{
> >+  if (!XRE_IsParentProcess() || aEvent.mIsSynthesizedByTIP) {
> >+    return false;
> >+  }
> >+
> >+  nsresult rv;
> >+  mHardwareKeyHandler = do_GetService("@mozilla.org/HardwareKeyHandler;1", &rv);
And why do we need to assign mHardwareKeyHandler to a new (yet same) value each time the method is called?

Comment 262

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8728754 [details] [diff] [review]
[v5] part4 - HardwareKeyHandler component

>+// For cycle collection, instead of using
>+// NS_IMPL_ISUPPORTS(HardwareKeyHandler, nsIHardwareKeyHandler),
>+// we need to use other specific methods for cycle collection to do its jobs.
>+// 'NS_IMPL_ISUPPORTS' provides implementations of 'QueryInterface' and
>+// 'Reference-Counting'. Without using it, we have to implement them by
>+// some magic macros by ourselves.
>+// See more detail in nsCycleCollectionParticipant.h
>+
>+// Reference count implementation for HardwareKeyHandler
>+NS_IMPL_CYCLE_COLLECTING_ADDREF(HardwareKeyHandler)
>+NS_IMPL_CYCLE_COLLECTING_RELEASE(HardwareKeyHandler)
>+
>+// QueryInterface implementation for HardwareKeyHandler
>+NS_INTERFACE_MAP_BEGIN_CYCLE_COLLECTION(HardwareKeyHandler)
>+  NS_INTERFACE_MAP_ENTRY(nsIHardwareKeyHandler)
>+  NS_INTERFACE_MAP_ENTRY_AMBIGUOUS(nsISupports, nsIHardwareKeyHandler)
>+NS_INTERFACE_MAP_END
>+
>+// 'NS_IMPL_CYCLE_COLLECTION' provides two methods:
>+// 1. Traverse  : Tell cycle collector all the strong pointer of C++
>+// 2. Unlink    : Clear all reference counts, including C++ and JS Objects
>+NS_IMPL_CYCLE_COLLECTION(HardwareKeyHandler, mHardwareKeyEventListener)
So technically this shouldn't really be needed since HardwareKeyHandler is a service, so it is kept alive until shutdown,
which means cycle collecting this code doesn't really help with anything else but shutdown.
So I think I'd prefer to not use cycle collection for HardwareKeyHandler.
And please test this all, with the code actually using this stuff, with debug build and XPCOM_MEM_LEAK_LOG environment variable set to
ensure we don't leak.

>+HardwareKeyHandler::~HardwareKeyHandler()
>+{
>+  // Clear the HardwareKeyEventListener
>+  mHardwareKeyEventListener = nullptr;
This is not needed. nsCOMPtr member variables are cleared 

>+HardwareKeyHandler::RegisterListener(nsIHardwareKeyEventListener* aListener)
>+{
>+  // Make sure the listener is not nullptr and there is no available
>+  // hardwareKeyEventListener now
>+  if (NS_WARN_IF(!aListener)) {
>+    return NS_ERROR_NULL_POINTER;
>+  }
>+
>+  if (NS_WARN_IF(mHardwareKeyEventListener)) {
>+    return NS_ERROR_ALREADY_INITIALIZED;
>+  }
>+
>+  mHardwareKeyEventListener = aListener;
>+  return NS_OK;
>+}
So if the cycle collection stuff was removed, I would probably add explicit UnregisterListener method which then just sets
mHardwareKeyEventListener to null.
One option is to also store the listener as an nsWeakPtr (to get such, use do_GetWeakReference on the listener)
and then do_QueryReferent to get back to the listener from that when it is needed.
To would be rather safe from memory management point of view.

>+  // being stored into the event queue. Otherwise, without the heading keydown
>+  // in the event queue, the stored keypress never be withdrawn to be fired.
is the sentence perhaps missing "will".  keypress will never be ... or something like that.


>+  // Create a keyboard event to pass into
>+  // nsIHardwareKeyEventListener.onHardwareKey
>+  nsCOMPtr<EventTarget> eventTarget = do_QueryInterface(aTarget);
>+  nsPresContext* presContext = GetPresContext(aTarget);
>+  RefPtr<KeyboardEvent> keyboardEvent =
>+    NS_NewDOMKeyboardEvent(eventTarget, presContext, aEvent->AsKeyboardEvent());
>+
>+  // Forward the created keyboard event to input-method-app
>+  bool isSent = false;
>+  mHardwareKeyEventListener->OnHardwareKey(keyboardEvent, &isSent);
Why is this safe. I mean, if aEvent is stack allocated, keyboardEvent, which is heap allocated will possibly have
a pointer to a deleted object if keyboardEvent stays alive after OnHardwareKey call (which it does because it is cycle collectable object which implies
asynchronous deletion)
So, you want to call event's DuplicatePrivateData() method here. That is what a normal event dispatch code ends up doing.
Perhaps simplest is to just call
keyboardEvent->DuplicatePrivateData(); right after NS_NewDOMKeyboardEvent call.

>+  // The value of defaultPrevented is depended on 
s/is depended on/depends on/




>+void
>+HardwareKeyHandler::PostHandleKeyboardEvent(nsINode* aTarget,
>+                                            WidgetKeyboardEvent& aEvent,
>+                                            nsEventStatus& aStatus)
>+{
>+  MOZ_ASSERT(aTarget, "No target provided");
>+  MOZ_ASSERT(aEvent, "No event provided");
>+
>+  nsPresContext* presContext = GetPresContext(aTarget);
>+
>+  RefPtr<mozilla::EventStateManager> esm = presContext->EventStateManager();
>+  bool dispatchedToChildProcess = PresShell::IsTargetIframe(aTarget);
>+  esm->PostHandleKeyboardEvent(&aEvent, aStatus, dispatchedToChildProcess);
>+}
So I don't understand this part. We some esm->PostHandle... but where do we call the relevant PreHandle*Event?

+// This module will copy the events' data into its event queue for reuse
+// after receiving input-method-app's reply, so we use the following struct
+// for storing these information.
+// RefCounted<T> is a helper class for adding reference counting mechanism.
+struct KeyboardInfo : public RefCounted<KeyboardInfo>
+{
+  nsINode* mTarget;
+  WidgetKeyboardEvent mEvent;
+  nsEventStatus mStatus;
+
+  KeyboardInfo(nsINode* aTarget,
+               WidgetKeyboardEvent& aEvent,
+               nsEventStatus aStatus)
+    : mTarget(aTarget)
+    , mEvent(aEvent)
+    , mStatus(aStatus)
+  {
+  }
+};
I don't understand what keeps mTarget alive?
The queue seems to addref KeyboardInfo objects, but mTarget is possibly just a pointer to a deleted object.


>+// A Type safe wrapper around nsDeque for storing events' data
>+// The T must be one class that supports reference counting mechanism.
>+template <class T>
>+class EventQueueDeallocator : public nsDequeFunctor {
Nit, { goes to its own line

>+  virtual void* operator() (void* aObject)
>+  {
>+    RefPtr<T> releaseMe = dont_AddRef(static_cast<T*>(aObject));
>+    return nullptr;
>+  }
>+};
This could use some comment. Why do we need this operator and when is it called?

Comment 263

[Olli Pettay [:smaug][[e-mail]]]

So, couple of tweaks and explanations, please.

Comment 264

[C.M.Chang[:chunmin]]

Attachment: part1 - add flags:IsSynthesizedByTIP and InputMethodAppState in WidgetKeyboardEvent r=masayuki

rebase and add r=masayuki

Comment 265

[C.M.Chang[:chunmin]]

Attachment: part2 - Prevent PostHandleKeyboardEvent if it's handling by IME r=masayuki

rebase and add r=masayuki

Comment 266

[C.M.Chang[:chunmin]]

Attachment: [v3] part3 - Interface between PresShell and HardwareKeyHandler

(In reply to Olli Pettay [:smaug] from comment #260)
> Comment on attachment 8719482 [details] [diff] [review]
> [v2] part3 - Interface between PresShell and HardwareKeyHandler
> 
> >+PresShell::IsTargetIframe(nsINode* aTarget)
> >+{
> >+  nsCOMPtr<nsIContent> content(do_QueryInterface(aTarget));
> >+  return content && content->IsHTMLElement(nsGkAtoms::iframe);
> >+}
> random comment, why the QI to content.
> return aTarget && aTarget->IsHTMLElement(nsGkAtoms::iframe); should work
> just fine.
OK

> >+PresShell::ForwardKeyToInputMethodApp(nsINode* aTarget,
> >+                                      WidgetKeyboardEvent& aEvent,
> >+                                      nsEventStatus* aStatus)
> >+{
> >+  if (!XRE_IsParentProcess() || aEvent.mIsSynthesizedByTIP) {
> >+    return false;
> >+  }
> >+
> >+  nsresult rv;
> >+  mHardwareKeyHandler = do_GetService("@mozilla.org/HardwareKeyHandler;1", &rv);
> This is what I was looking for for reviewing part 4.
> Ok, HardwareKeyHandler is a service, so making class HardwareKeyHandler
> a cycle collectable object doesn't really help much, since services will be
> kept alive until shutdown.
> (I know, we do have some cycle collectable services, like nsFocusManager,
> but we should get rid of them.)
OK, I'll remove it from cycle collection.

(In reply to Olli Pettay [:smaug] from comment #261)
> (In reply to Olli Pettay [:smaug] from comment #260)
> > >+PresShell::ForwardKeyToInputMethodApp(nsINode* aTarget,
> > >+                                      WidgetKeyboardEvent& aEvent,
> > >+                                      nsEventStatus* aStatus)
> > >+{
> > >+  if (!XRE_IsParentProcess() || aEvent.mIsSynthesizedByTIP) {
> > >+    return false;
> > >+  }
> > >+
> > >+  nsresult rv;
> > >+  mHardwareKeyHandler = do_GetService("@mozilla.org/HardwareKeyHandler;1", &rv);
> And why do we need to assign mHardwareKeyHandler to a new (yet same) value
> each time the method is called?
Yes, you're right. Masayuki mention this before, but I didn't upload the new patch.
Now |mHardwareKeyHandler| will be assigned only if it is nullptr.

Comment 267

[C.M.Chang[:chunmin]]

Attachment: part6 - Expose DefaultPreventedBy* r=masayuki, smaug

rebase and add r=masayuki, smaug

Comment 268

[C.M.Chang[:chunmin]]

Attachment: [v5] part7 - Interface between HardwareKeyHandler and Input Method App

|keyboard| in Keyboard.jsm now implements |nsIHardwareKeyEventListener| for receiving forwarded hardware key event and |nsISupportsWeakReference| to support nsWeakPtr. Request review again due to code change.

Comment 269

[C.M.Chang[:chunmin]]

Attachment: [v6] part4 - HardwareKeyHandler component

Modification
- Add a new method: UnregisterListener
- Type for mHardwareKeyEventListener is changed to |nsWeakPtr|

(In reply to Olli Pettay [:smaug] from comment #262)
> Comment on attachment 8728754 [details] [diff] [review]
> [v5] part4 - HardwareKeyHandler component
> 
> >+// For cycle collection, instead of using
> >+// NS_IMPL_ISUPPORTS(HardwareKeyHandler, nsIHardwareKeyHandler),
> >+// we need to use other specific methods for cycle collection to do its jobs.
> >+// 'NS_IMPL_ISUPPORTS' provides implementations of 'QueryInterface' and
> >+// 'Reference-Counting'. Without using it, we have to implement them by
> >+// some magic macros by ourselves.
> >+// See more detail in nsCycleCollectionParticipant.h
> >+
> >+// Reference count implementation for HardwareKeyHandler
> >+NS_IMPL_CYCLE_COLLECTING_ADDREF(HardwareKeyHandler)
> >+NS_IMPL_CYCLE_COLLECTING_RELEASE(HardwareKeyHandler)
> >+
> >+// QueryInterface implementation for HardwareKeyHandler
> >+NS_INTERFACE_MAP_BEGIN_CYCLE_COLLECTION(HardwareKeyHandler)
> >+  NS_INTERFACE_MAP_ENTRY(nsIHardwareKeyHandler)
> >+  NS_INTERFACE_MAP_ENTRY_AMBIGUOUS(nsISupports, nsIHardwareKeyHandler)
> >+NS_INTERFACE_MAP_END
> >+
> >+// 'NS_IMPL_CYCLE_COLLECTION' provides two methods:
> >+// 1. Traverse  : Tell cycle collector all the strong pointer of C++
> >+// 2. Unlink    : Clear all reference counts, including C++ and JS Objects
> >+NS_IMPL_CYCLE_COLLECTION(HardwareKeyHandler, mHardwareKeyEventListener)
> So technically this shouldn't really be needed since HardwareKeyHandler is a
> service, so it is kept alive until shutdown,
> which means cycle collecting this code doesn't really help with anything
> else but shutdown.
> So I think I'd prefer to not use cycle collection for HardwareKeyHandler.
> And please test this all, with the code actually using this stuff, with
> debug build and XPCOM_MEM_LEAK_LOG environment variable set to
> ensure we don't leak.
OK. I remove this part.

> >+HardwareKeyHandler::~HardwareKeyHandler()
> >+{
> >+  // Clear the HardwareKeyEventListener
> >+  mHardwareKeyEventListener = nullptr;
> This is not needed. nsCOMPtr member variables are cleared 
OK, it's removed.

> >+HardwareKeyHandler::RegisterListener(nsIHardwareKeyEventListener* aListener)
> >+{
> >+  // Make sure the listener is not nullptr and there is no available
> >+  // hardwareKeyEventListener now
> >+  if (NS_WARN_IF(!aListener)) {
> >+    return NS_ERROR_NULL_POINTER;
> >+  }
> >+
> >+  if (NS_WARN_IF(mHardwareKeyEventListener)) {
> >+    return NS_ERROR_ALREADY_INITIALIZED;
> >+  }
> >+
> >+  mHardwareKeyEventListener = aListener;
> >+  return NS_OK;
> >+}
> So if the cycle collection stuff was removed, I would probably add explicit
> UnregisterListener method which then just sets
> mHardwareKeyEventListener to null.
> One option is to also store the listener as an nsWeakPtr (to get such, use
> do_GetWeakReference on the listener)
> and then do_QueryReferent to get back to the listener from that when it is
> needed.
> To would be rather safe from memory management point of view.
Thanks for your advises. I add a new method named UnregisterListener in .idl and the type of mHardwareKeyEventListener is changed from |nsCOMPtr<nsIHardwareKeyEventListener>| to |nsWeakPtr|.

> >+  // being stored into the event queue. Otherwise, without the heading keydown
> >+  // in the event queue, the stored keypress never be withdrawn to be fired.
> is the sentence perhaps missing "will".  keypress will never be ... or
> something like that.
Yes, you're right. I lost a "will" here.

> >+  // Create a keyboard event to pass into
> >+  // nsIHardwareKeyEventListener.onHardwareKey
> >+  nsCOMPtr<EventTarget> eventTarget = do_QueryInterface(aTarget);
> >+  nsPresContext* presContext = GetPresContext(aTarget);
> >+  RefPtr<KeyboardEvent> keyboardEvent =
> >+    NS_NewDOMKeyboardEvent(eventTarget, presContext, aEvent->AsKeyboardEvent());
> >+
> >+  // Forward the created keyboard event to input-method-app
> >+  bool isSent = false;
> >+  mHardwareKeyEventListener->OnHardwareKey(keyboardEvent, &isSent);
> Why is this safe. I mean, if aEvent is stack allocated, keyboardEvent, which
> is heap allocated will possibly have
> a pointer to a deleted object if keyboardEvent stays alive after
> OnHardwareKey call (which it does because it is cycle collectable object
> which implies
> asynchronous deletion)
> So, you want to call event's DuplicatePrivateData() method here. That is
> what a normal event dispatch code ends up doing.
> Perhaps simplest is to just call
> keyboardEvent->DuplicatePrivateData(); right after NS_NewDOMKeyboardEvent
> call.
Thank you for your kindly reminder and explanation. 
|keyboardEvent->DuplicatePrivateData()| is added there now.

> >+  // The value of defaultPrevented is depended on 
> s/is depended on/depends on/
Done.

> >+void
> >+HardwareKeyHandler::PostHandleKeyboardEvent(nsINode* aTarget,
> >+                                            WidgetKeyboardEvent& aEvent,
> >+                                            nsEventStatus& aStatus)
> >+{
> >+  MOZ_ASSERT(aTarget, "No target provided");
> >+  MOZ_ASSERT(aEvent, "No event provided");
> >+
> >+  nsPresContext* presContext = GetPresContext(aTarget);
> >+
> >+  RefPtr<mozilla::EventStateManager> esm = presContext->EventStateManager();
> >+  bool dispatchedToChildProcess = PresShell::IsTargetIframe(aTarget);
> >+  esm->PostHandleKeyboardEvent(&aEvent, aStatus, dispatchedToChildProcess);
> >+}
> So I don't understand this part. We some esm->PostHandle... but where do we
> call the relevant PreHandle*Event?
Hmm... is any difference in PreHandle*Event between the forwarded keyboard and the one who follows the normal dispatch?
I mean, do we have to do additional handling in PreHandle*Event if we forward the keyboard event to IME first? 
The PreHandle*Event is called before PresShell::HandleKeyboardEvent, where we start to forward the keyboard events to IME. I think the event is same as the normal one before we forward it, so I don't add any additional handling there.

> +// This module will copy the events' data into its event queue for reuse
> +// after receiving input-method-app's reply, so we use the following struct
> +// for storing these information.
> +// RefCounted<T> is a helper class for adding reference counting mechanism.
> +struct KeyboardInfo : public RefCounted<KeyboardInfo>
> +{
> +  nsINode* mTarget;
> +  WidgetKeyboardEvent mEvent;
> +  nsEventStatus mStatus;
> +
> +  KeyboardInfo(nsINode* aTarget,
> +               WidgetKeyboardEvent& aEvent,
> +               nsEventStatus aStatus)
> +    : mTarget(aTarget)
> +    , mEvent(aEvent)
> +    , mStatus(aStatus)
> +  {
> +  }
> +};
> I don't understand what keeps mTarget alive?
> The queue seems to addref KeyboardInfo objects, but mTarget is possibly just
> a pointer to a deleted object.
keyup's target should not be destroyed after keypress's descontructor is called, so mTarget shouldn't be deleted in keypress's descontructor. Also, if we release the object pointed by mTarget once the KeyboardInfo's descontructor is called, then the target of following any sort of event will be destroyed.

> >+// A Type safe wrapper around nsDeque for storing events' data
> >+// The T must be one class that supports reference counting mechanism.
> >+template <class T>
> >+class EventQueueDeallocator : public nsDequeFunctor {
> Nit, { goes to its own line
> 
> >+  virtual void* operator() (void* aObject)
> >+  {
> >+    RefPtr<T> releaseMe = dont_AddRef(static_cast<T*>(aObject));
> >+    return nullptr;
> >+  }
> >+};
> This could use some comment. Why do we need this operator and when is it
> called?
OK, I add some comments there in the patch.

The EventQueueDeallocator will be triggered in nsDeque::~nsDeque() or nsDeque::Erase().
The nsDeque::Erase() will release all items in the queue container. 
The parameters, |void* aObject| of the |operator()| will be passed when nsDeque::ForEach() is called[1].

[1] https://dxr.mozilla.org/mozilla-central/source/xpcom/glue/nsDeque.cpp#359

Comment 270

[C.M.Chang[:chunmin]]

Comment on attachment 8731070 [details] [diff] [review]
[v6] part4 - HardwareKeyHandler component

Note:
Need to add |#ifdef MOZ_B2G| to nsLayoutModule.cpp

Comment 271

[C.M.Chang[:chunmin]]

Comment on attachment 8731054 [details] [diff] [review]
[v5] part7 - Interface between HardwareKeyHandler and Input Method App

Notes
1. Need to add |#ifdef MOZ_B2G| to Keyboard.jsm
   and move Keyboard.jsm to EXTRA_PP_JS_MODULES
2. Need to add |#ifdef MOZ_B2G| to MozKeyboard.js
   and move MozKeyboard.js to EXTRA_PP_COMPONENTS
3. Need to add |#ifdef MOZ_B2G| to InputMethod.webidl
   and move InputMethod.webidl to PREPROCESSED_WEBIDL_FILES

Comment 272

[C.M.Chang[:chunmin]]

Attachment: [v5.1] part7 - Interface between HardwareKeyHandler and Input Method App

Modification
1. |keyboard| in Keyboard.jsm now implements |nsIHardwareKeyEventListener| for receiving forwarded hardware key event and |nsISupportsWeakReference| to support nsWeakPtr. Request review again due to code change.
2. Add |#ifdef MOZ_B2G| to Keyboard.jsm and move Keyboard.jsm to EXTRA_PP_JS_MODULES
3. Add |#ifdef MOZ_B2G| to MozKeyboard.js and move MozKeyboard.js to EXTRA_PP_COMPONENTS
4. Add |#ifdef MOZ_B2G| to InputMethod.webidl and move InputMethod.webidl to PREPROCESSED_WEBIDL_FILES

Comment 273

[C.M.Chang[:chunmin]]

Attachment: [v5.1] part7 - Interface between HardwareKeyHandler and Input Method App

Sorry for updating the wrong version.
This one should be right.

Comment 274

[C.M.Chang[:chunmin]]

Attachment: [v6.1] part4 - HardwareKeyHandler component

Add |#ifdef MOZ_B2G| to nsLayoutModule.cpp

Comment 275

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8731645 [details] [diff] [review]
[v5.1] part7 - Interface between HardwareKeyHandler and Input Method App

I don't understand why the additional change from the patch which I gave r+ to is necessary, though.

I like better runtime check rather than #ifdef MOZ_B2G. If you do so, we waste a lot of CPU and/or memory resource? If so, your patch is better, though.

Comment 276

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #275)
> Comment on attachment 8731645 [details] [diff] [review]
> [v5.1] part7 - Interface between HardwareKeyHandler and Input Method App
> 
> I don't understand why the additional change from the patch which I gave r+
> to is necessary, though.
> 
> I like better runtime check rather than #ifdef MOZ_B2G. If you do so, we
> waste a lot of CPU and/or memory resource? If so, your patch is better,
> though.
The mochitest under dom/inputmethod/ can not be passed if I don't add #ifdef MOZ_B2G. 
The nsIHardwareKeyHandler will be built under the MOZ_B2G condition, so Keyboard.jsm can't find and load it successfully without this condition. 

And I don't think the memory resource is a big issue. On the contrary, I think we will use less memory resource because those functions with condition #ifdef MOZ_B2G won't be put into memory if the platform is not b2g.

Comment 277

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

So, why don't you use:

if (hardwareKeyHandler) {
  hardwareKeyHandler.foo();
}

and

if (!Ci.nsIHardwareKeyHandler) {
  return;
}

? I don't like including #ifdef in functions since it causes messy code and not easy to read.

I guess following chunks need #ifdefs.
https://bugzilla.mozilla.org/attachment.cgi?id=8731645&action=diff#a/dom/inputmethod/Keyboard.jsm_sec2
https://bugzilla.mozilla.org/attachment.cgi?id=8731645&action=diff#a/dom/inputmethod/Keyboard.jsm_sec3
But I like using runtime check for the others better.

Comment 278

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8731052 [details] [diff] [review]
[v3] part3 - Interface between PresShell and HardwareKeyHandler

>+PresShell::IsTargetIframe(nsINode* aTarget)
>+{
>+  nsCOMPtr<nsIContent> content(do_QueryInterface(aTarget));
>+  return content && content->IsHTMLElement(nsGkAtoms::iframe);
So you don't need QI here.
just return aTarget && aTarget->IsHTMLElement(nsGkAtoms::iframe);
as I commented before.

> PresShell::HandleKeyboardEvent(nsINode* aTarget,
>                                WidgetKeyboardEvent& aEvent,
>                                bool aEmbeddedCancelled,
>                                nsEventStatus* aStatus,
>                                EventDispatchingCallback* aEventCB)
> {
>+  MOZ_ASSERT(aTarget);
>+
>+  // return true if the event target is in its child process
>+  bool targetIsIframe = IsTargetIframe(aTarget);
>+
>   if (aEvent.mMessage == eKeyPress ||
>-      !BeforeAfterKeyboardEventEnabled()) {
>-    EventDispatcher::Dispatch(aTarget, mPresContext,
>-                              &aEvent, nullptr, aStatus, aEventCB);
>+      !BeforeAfterKeyboardEventEnabled() ||
>+      aEvent.mIsSynthesizedByTIP) {
Could you explain mIsSynthesizedByTIP usage here? Add some comment to the code.

>+#ifdef MOZ_B2G
>+bool
>+PresShell::ForwardKeyToInputMethodApp(nsINode* aTarget,
>+                                      WidgetKeyboardEvent& aEvent,
>+                                      nsEventStatus* aStatus)
>+{
>+  if (!XRE_IsParentProcess() || aEvent.mIsSynthesizedByTIP) {
>+    return false;
>+  }
>+
>+  if (!mHardwareKeyHandler) {
>+    nsresult rv;
>+    mHardwareKeyHandler =
>+      do_GetService("@mozilla.org/HardwareKeyHandler;1", &rv);
>+    if (!NS_SUCCEEDED(rv) || !mHardwareKeyHandler) {
>+      return false;
>+    }
>+  }
>+
>+  if (mHardwareKeyHandler->ForwardKeyToInputMethodApp(aTarget,
>+                                                      aEvent.AsKeyboardEvent(),
>+                                                      aStatus)) {
>+    // No need to dispatch the forwarded keyboard event to it's child process
>+    aEvent.mFlags.mNoCrossProcessBoundaryForwarding = true;
>+    return true;
>+  }
>+
>+  return false;
>+}
>+#endif // MOZ_B2G
>+  // In-process case: the event target is in the current process
>+  if (!aIsTargetRemote) {
>+    if(ForwardKeyToInputMethodApp(aTarget, aEvent, aStatus)) {
>+      return true;
>+    }
>+
>+    // If the keyboard event isn't forwarded to the input-method-app,
>+    // then it should be dispatched to its event target directly.
>+    EventDispatcher::Dispatch(aTarget, mPresContext,
>+                              &aEvent, nullptr, aStatus, aEventCB);
>+
>+    return false;
>+  }
>+
>+  // OOP case: the event target is in its child process.
>+  // Dispatch the keyboard event to the iframe that embeds the remote
>+  // event target first.
>+  EventDispatcher::Dispatch(aTarget, mPresContext,
>+                            &aEvent, nullptr, aStatus, aEventCB);


Why you have similar Dispatch call twice. Couldn't you return conditionally after Dispatch call.
Say,
if (!aIsTargetRemote) {
  return;
}

Comment 279

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8731646 [details] [diff] [review]
[v6.1] part4 - HardwareKeyHandler component

>+HardwareKeyHandler::RegisterListener(nsIHardwareKeyEventListener* aListener)
>+{
>+  // Make sure the listener is not nullptr and there is no available
>+  // hardwareKeyEventListener now
>+  if (NS_WARN_IF(!aListener)) {
>+    return NS_ERROR_NULL_POINTER;
>+  }
>+
>+  if (NS_WARN_IF(mHardwareKeyEventListener)) {
>+    return NS_ERROR_ALREADY_INITIALIZED;
>+  }
>+
>+  mHardwareKeyEventListener = do_GetWeakReference(aListener);
I think it would make sense to return error value if mHardwareKeyEventListener is null after this line.
>+
>+  // Re-trigger EventStateManager::PostHandleKeyboardEvent for keypress
Re-trigger? Do we really re-trigger posthandle* for this event. That sounds odd. Please explain and fix if we actually call it twice (and if we call it twice, also PreHandle* need to be called twice, IMO)
And please add a comment that PreHandleEvent has been called already.
For consistency we really should always have a PreHandle* and PostHandle* calls the same number times, hopefully just once. Anything else is odd.


>+// The following is the type-safe wrapper around nsDeque
>+// for storing events' data.
>+// The T must be one class that supports reference counting mechanism.
>+// The EventQueueDeallocator will be called in nsDeque::~nsDeque() or
>+// nsDeque::Erase() to deallocate the objects. nsDeque::Erase() will remove
>+// and delete all items in the queue. See more from nsDeque.h.
>+template <class T>
>+class EventQueueDeallocator : public nsDequeFunctor
>+{
>+  virtual void* operator() (void* aObject)
>+  {
>+    RefPtr<T> releaseMe = dont_AddRef(static_cast<T*>(aObject));
>+    return nullptr;
>+  }
So EventQueueDeallocator is actually something very generic and so is EventQueue.
They should be renamed. Perhaps RefCountedObjectQueue. A bit long but couldn't figure out anything better right now
(in principle I'd prefer moving this useful code to be in the same directory where nsQueue.h but that can be done in a separate bug)

>+   * If the input-method-app consumes the forwarded event,
>+   * then the result shoult be pulled high by DEFAULT_PREVENTED* before reply.
I don't know what "pulled high" means in this context. perhaps clarify the text a bit.

>+  /**
>+   * Unregisters a listener from input-method-app
s/a/the current/


Those fixed, r+. But the Pre/PostHandle case especially needs a good comment and explanation or possibly some fixes.

Comment 280

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8731645 [details] [diff] [review]
[v5.1] part7 - Interface between HardwareKeyHandler and Input Method App

In general masayuki's review for this should be fine, but
+[JSImplementation="@mozilla.org/b2g-hardwareinput;1",
+ Pref="dom.mozInputMethod.enabled"]
+interface MozHardwareInput: EventTarget {
doesn't look right.
Don't we end up exposing window.MozHardwareInput pretty much in all the context since prefs tend to be global. Don't you want some permission there?

Comment 281

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #277)
> So, why don't you use:
> 
> if (hardwareKeyHandler) {
>   hardwareKeyHandler.foo();
> }
> 
> and
> 
> if (!Ci.nsIHardwareKeyHandler) {
>   return;
> }
> 
> ? I don't like including #ifdef in functions since it causes messy code and
> not easy to read.
OK, I'll modify it. 
> I guess following chunks need #ifdefs.
> https://bugzilla.mozilla.org/attachment.cgi?id=8731645&action=diff#a/dom/
> inputmethod/Keyboard.jsm_sec2
It can be checked in runtime by the following:
XPCOMUtils.defineLazyGetter(this, "hardwareKeyHandler", function() {
  if (!Cc["@mozilla.org/HardwareKeyHandler;1"]) {
    return null;
  }
  return Cc["@mozilla.org/HardwareKeyHandler;1"]
         .getService(Ci.nsIHardwareKeyHandler);
});

But I am not sure if it will make code clearer. 
> https://bugzilla.mozilla.org/attachment.cgi?id=8731645&action=diff#a/dom/
> inputmethod/Keyboard.jsm_sec3
> But I like using runtime check for the others better.

Comment 282

[C.M.Chang[:chunmin]]

Attachment: [v7] part4 - HardwareKeyHandler component

(In reply to Olli Pettay [:smaug] from comment #279)
> Comment on attachment 8731646 [details] [diff] [review]
> [v6.1] part4 - HardwareKeyHandler component
> 
> >+HardwareKeyHandler::RegisterListener(nsIHardwareKeyEventListener* aListener)
> >+{
> >+  // Make sure the listener is not nullptr and there is no available
> >+  // hardwareKeyEventListener now
> >+  if (NS_WARN_IF(!aListener)) {
> >+    return NS_ERROR_NULL_POINTER;
> >+  }
> >+
> >+  if (NS_WARN_IF(mHardwareKeyEventListener)) {
> >+    return NS_ERROR_ALREADY_INITIALIZED;
> >+  }
> >+
> >+  mHardwareKeyEventListener = do_GetWeakReference(aListener);
> I think it would make sense to return error value if
> mHardwareKeyEventListener is null after this line.
OK.
> >+  // Re-trigger EventStateManager::PostHandleKeyboardEvent for keypress
> Re-trigger? Do we really re-trigger posthandle* for this event. That sounds
> odd. Please explain and fix if we actually call it twice (and if we call it
> twice, also PreHandle* need to be called twice, IMO)
> And please add a comment that PreHandleEvent has been called already.
> For consistency we really should always have a PreHandle* and PostHandle*
> calls the same number times, hopefully just once. Anything else is odd.
I explain it in HardwareKeyHandler.h.
The original EventStateManager::PostHandleKeyboardEvent will be aborted when we try to forward events to IME.
If IME consumes the key, then the abortion is OK because IME will generate a new key event by itself.
On the other hand, if IME doesn't consume the key, then we need to dispatch the key by ourselves.
Thus, we need to call EventStateManager::PostHandleKeyboardEvent again to process some special keys(e.g., NS_VK_TAB).
The EventStateManager::PostHandleKeyboardEvent is modified in Part 2.
Sorry for forgetting to mention that.
> 
> >+// The following is the type-safe wrapper around nsDeque
> >+// for storing events' data.
> >+// The T must be one class that supports reference counting mechanism.
> >+// The EventQueueDeallocator will be called in nsDeque::~nsDeque() or
> >+// nsDeque::Erase() to deallocate the objects. nsDeque::Erase() will remove
> >+// and delete all items in the queue. See more from nsDeque.h.
> >+template <class T>
> >+class EventQueueDeallocator : public nsDequeFunctor
> >+{
> >+  virtual void* operator() (void* aObject)
> >+  {
> >+    RefPtr<T> releaseMe = dont_AddRef(static_cast<T*>(aObject));
> >+    return nullptr;
> >+  }
> So EventQueueDeallocator is actually something very generic and so is
> EventQueue.
> They should be renamed. Perhaps RefCountedObjectQueue. A bit long but
> couldn't figure out anything better right now
> (in principle I'd prefer moving this useful code to be in the same directory
> where nsQueue.h but that can be done in a separate bug)
OK, I'll open another bug after this is landed. 
> >+   * If the input-method-app consumes the forwarded event,
> >+   * then the result shoult be pulled high by DEFAULT_PREVENTED* before reply.
> I don't know what "pulled high" means in this context. perhaps clarify the
> text a bit.
> 
> >+  /**
> >+   * Unregisters a listener from input-method-app
> s/a/the current/
OK

Comment 283

[C.M.Chang[:chunmin]]

Attachment: [v4] part3 - Interface between PresShell and HardwareKeyHandler r=masayuki, smaug

(In reply to Olli Pettay [:smaug] from comment #278)
> Comment on attachment 8731052 [details] [diff] [review]
> [v3] part3 - Interface between PresShell and HardwareKeyHandler
> 
> >+PresShell::IsTargetIframe(nsINode* aTarget)
> >+{
> >+  nsCOMPtr<nsIContent> content(do_QueryInterface(aTarget));
> >+  return content && content->IsHTMLElement(nsGkAtoms::iframe);
> So you don't need QI here.
> just return aTarget && aTarget->IsHTMLElement(nsGkAtoms::iframe);
> as I commented before.
Sorry for forgetting that. I modify it in this version.
> > PresShell::HandleKeyboardEvent(nsINode* aTarget,
> >                                WidgetKeyboardEvent& aEvent,
> >                                bool aEmbeddedCancelled,
> >                                nsEventStatus* aStatus,
> >                                EventDispatchingCallback* aEventCB)
> > {
> >+  MOZ_ASSERT(aTarget);
> >+
> >+  // return true if the event target is in its child process
> >+  bool targetIsIframe = IsTargetIframe(aTarget);
> >+
> >   if (aEvent.mMessage == eKeyPress ||
> >-      !BeforeAfterKeyboardEventEnabled()) {
> >-    EventDispatcher::Dispatch(aTarget, mPresContext,
> >-                              &aEvent, nullptr, aStatus, aEventCB);
> >+      !BeforeAfterKeyboardEventEnabled() ||
> >+      aEvent.mIsSynthesizedByTIP) {
> Could you explain mIsSynthesizedByTIP usage here? Add some comment to the
> code.
I add comments there.
I think only keydown/keyup fired from hardware can generate beforeKey* and afterKey*, so anything else(e.g., fired from software by nsITextInputProcessor) can just dispatch it directly.

> >+#ifdef MOZ_B2G
> >+bool
> >+PresShell::ForwardKeyToInputMethodApp(nsINode* aTarget,
> >+                                      WidgetKeyboardEvent& aEvent,
> >+                                      nsEventStatus* aStatus)
> >+{
> >+  if (!XRE_IsParentProcess() || aEvent.mIsSynthesizedByTIP) {
> >+    return false;
> >+  }
> >+
> >+  if (!mHardwareKeyHandler) {
> >+    nsresult rv;
> >+    mHardwareKeyHandler =
> >+      do_GetService("@mozilla.org/HardwareKeyHandler;1", &rv);
> >+    if (!NS_SUCCEEDED(rv) || !mHardwareKeyHandler) {
> >+      return false;
> >+    }
> >+  }
> >+
> >+  if (mHardwareKeyHandler->ForwardKeyToInputMethodApp(aTarget,
> >+                                                      aEvent.AsKeyboardEvent(),
> >+                                                      aStatus)) {
> >+    // No need to dispatch the forwarded keyboard event to it's child process
> >+    aEvent.mFlags.mNoCrossProcessBoundaryForwarding = true;
> >+    return true;
> >+  }
> >+
> >+  return false;
> >+}
> >+#endif // MOZ_B2G
> >+  // In-process case: the event target is in the current process
> >+  if (!aIsTargetRemote) {
> >+    if(ForwardKeyToInputMethodApp(aTarget, aEvent, aStatus)) {
> >+      return true;
> >+    }
> >+
> >+    // If the keyboard event isn't forwarded to the input-method-app,
> >+    // then it should be dispatched to its event target directly.
> >+    EventDispatcher::Dispatch(aTarget, mPresContext,
> >+                              &aEvent, nullptr, aStatus, aEventCB);
> >+
> >+    return false;
> >+  }
> >+
> >+  // OOP case: the event target is in its child process.
> >+  // Dispatch the keyboard event to the iframe that embeds the remote
> >+  // event target first.
> >+  EventDispatcher::Dispatch(aTarget, mPresContext,
> >+                            &aEvent, nullptr, aStatus, aEventCB);
> 
> 
> Why you have similar Dispatch call twice. Couldn't you return conditionally
> after Dispatch call.
> Say,
> if (!aIsTargetRemote) {
>   return;
> }
Hmm, ... the two cases have reverse order, so I am not sure if it can be written in early-return pattern.
{ 
  If (!aIsTargetRemote) {
    ForwardKeyToInputMethodApp
    EventDispatcher::Dispatch
    return value
  }
  EventDispatcher::Dispatch
  ForwardKeyToInputMethodApp
  return value
}

Comment 284

[C.M.Chang[:chunmin]]

Comment on attachment 8732771 [details] [diff] [review]
[v7] part4 - HardwareKeyHandler component

Carry r+ after fix nits

Comment 285

[C.M.Chang[:chunmin]]

Attachment: [v6] part7 - Interface between HardwareKeyHandler and Input Method App

(In reply to Olli Pettay [:smaug] from comment #280)
> Comment on attachment 8731645 [details] [diff] [review]
> [v5.1] part7 - Interface between HardwareKeyHandler and Input Method App
> 
> In general masayuki's review for this should be fine, but
> +[JSImplementation="@mozilla.org/b2g-hardwareinput;1",
> + Pref="dom.mozInputMethod.enabled"]
> +interface MozHardwareInput: EventTarget {
> doesn't look right.
> Don't we end up exposing window.MozHardwareInput pretty much in all the
> context since prefs tend to be global. Don't you want some permission there?
Yes, you're right. I add |CheckAnyPermissions="input"| there.

> > I guess following chunks need #ifdefs.
> > https://bugzilla.mozilla.org/attachment.cgi?id=8731645&action=diff#a/dom/
> > inputmethod/Keyboard.jsm_sec2
> It can be checked in runtime by the following:
> XPCOMUtils.defineLazyGetter(this, "hardwareKeyHandler", function() {
>   if (!Cc["@mozilla.org/HardwareKeyHandler;1"]) {
>     return null;
>   }
>   return Cc["@mozilla.org/HardwareKeyHandler;1"]
>          .getService(Ci.nsIHardwareKeyHandler);
> });
> 
> But I am not sure if it will make code clearer. 
I finally use

XPCOMUtils.defineLazyGetter(this, "hardwareKeyHandler", function() {
#ifdef MOZ_B2G
  return Cc["@mozilla.org/HardwareKeyHandler;1"]
         .getService(Ci.nsIHardwareKeyHandler);
#else
  return null;
#endif
});

|hardwareKeyHandler| will be called, so LazyGetter need to return a value.

Comment 286

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8732786 [details] [diff] [review]
[v6] part7 - Interface between HardwareKeyHandler and Input Method App

Looks better to me. Thanks!

Comment 287

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #286)
Thanks!

However, I am afraid that I need to modify a little bit part of this patch. 
I may add a property |_isActive| to Keyboard to state the status of Keyboard and prevent calling |hardwareKeyHandler.onInputMethodAppConnected()| multiple times in some cases(e.g., dom/inputmethod/mochitest/test_sync_edit.html).

The result will be:

this.Keyboard = {
  ..
  ..
  _isActive: false,
  ..
  ..
  handleFocus: function keyboardHandleFocus(msg) {
    ..
    ..
    // Notify the nsIHardwareKeyHandler that the input-method-app is active now.
    if (hardwareKeyHandler && !this._isActive) {
      this._isActive = true;
      hardwareKeyHandler.onInputMethodAppConnected();
    }
    ..
    ..
  },

  handleBlur: function keyboardHandleBlur(msg) {
    ..
    ..
    // Notify the nsIHardwareKeyHandler that
    // the input-method-app is disabled now.
    if (hardwareKeyHandler && this._isActive) {
      this._isActive = false;
      hardwareKeyHandler.onInputMethodAppDisconnected();
    }
    ..
    ..
  },
  
  ..
  ..
};

Comment 288

[C.M.Chang[:chunmin]]

(In reply to Chun-Min Chang[:chunmin] from comment #287)
It's better to rename |_isActive| to |_isConnectedToHardwareKeyHandler| or something else.

And I found that dom/inputmethod/mochitest is skipped on Gonk[0], so even I implement nsIDOMWindowUtils::SendNativeKeyEvent() for Gonk widget, I can't test it on try-server.
Other test cases such as GIP and GIJ is moved to tier 3, and almost all test cases in tier 3 is broken.
I don't know what I should do for the test cases, could you provide some suggestions?

[0] https://dxr.mozilla.org/mozilla-central/source/dom/inputmethod/mochitest/mochitest.ini#3

Comment 289

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Chun-Min Chang[:chunmin] from comment #288)
> (In reply to Chun-Min Chang[:chunmin] from comment #287)
> It's better to rename |_isActive| to |_isConnectedToHardwareKeyHandler| or
> something else.

Sounds reasonable.

> And I found that dom/inputmethod/mochitest is skipped on Gonk[0], so even I
> implement nsIDOMWindowUtils::SendNativeKeyEvent() for Gonk widget, I can't
> test it on try-server.
> Other test cases such as GIP and GIJ is moved to tier 3, and almost all test
> cases in tier 3 is broken.
> I don't know what I should do for the test cases, could you provide some
> suggestions?

If you think that this should be tested entirely, I think that you should support it even on desktop. Then, you can test it with mochitests on desktop OSes (If you need to use SendNativeKeyEvent(), you can test only on Win and Mac, though).

Comment 290

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Anyway, please close this bug for now. Please file new bugs for each additional work.

Comment 291

[C.M.Chang[:chunmin]]

Attachment: [WIP] testcase part1: enable hwKeyHandler on mac

Comment 292

[C.M.Chang[:chunmin]]

Attachment: [WIP] testcase part2: mochitest

Comment 293

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #290)
> Anyway, please close this bug for now. Please file new bugs for each
> additional work.
OK, I'll land it after finishing test on mochitest.

Comment 294

[C.M.Chang[:chunmin]]

Attachment: part 8: Use synthesizeNativeKey instead of synthesizeKey to test beforekey and afterkey

Hi Masayuki-san,
After importing the patches from part 1 to 7, the synthesizeKey is no longer to fire beforekey* and afterkey*, so it needs to be replaced with synthesizeNativeKey. 

The design of test_dom_before_after_keyboard_event.html is now same as test_dom_before_after_keyboard_event_remote.html. synthesizeNativeKey will dispatch the task(for simulating a hardware event) to main thread instead of executing task directly like synthesizeKey, so it needs time to wait for the expected events.

Comment 295

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8734257 [details] [diff] [review]
part 8: Use synthesizeNativeKey instead of synthesizeKey to test beforekey and afterkey

>+  synthesizeNativeKey(KEYBOARD_LAYOUT_EN_US, kIsWin ? WIN_VK_A : MAC_VK_ANSI_A,
>+                          {}, "a", "a");
>+  // It takes some time to receive native events fired from hardware.
>+
>+  waitAndVerifyResult(0);
>+}
>+
>+function waitAndVerifyResult(aCount) {
>+  expectedEventLength = gCurrentTest.expectedEvents[0].length +
>+                        gCurrentTest.expectedEvents[1].length;
>+  if (gCurrentTest.resultEvents.length >= expectedEventLength || aCount > 10) {
>+    classifyEvents(gCurrentTest);
>+    verifyResults(gCurrentTest);
>+    testEventOrderAndAttr();
>   }
>+  else {
>+    setTimeout(() => waitAndVerifyResult(aCount + 1),
>+               100);
>+  }
> }

Why don't you use aCallback of synthesizeNativeKey()? Isn't it useful?

Comment 296

[C.M.Chang[:chunmin]]

Attachment: [v2] part 8: Use synthesizeNativeKey instead of synthesizeKey to test beforekey and afterkey

You're right. I didn't notice that.
However, the test_bug1096146.html keep using setTimeout because it still need to wait for 'scroll' events.

Comment 297

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 8734300 [details] [diff] [review]
[v2] part 8: Use synthesizeNativeKey instead of synthesizeKey to test beforekey and afterkey

Thanks!

Comment 298

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

FYI: Your patch touches TextEvents.h. I'd like to modify it. So, could you land them as soon as possible?

Comment 299

[C.M.Chang[:chunmin]]

Yes, I want to land it ASAP but try server works slow. I haven't get my test all results of the patches[0] I tested it yesterday.

[0] https://treeherder.mozilla.org/#/jobs?repo=try&revision=5117346f6839&selectedJob=18548832

Comment 300

[C.M.Chang[:chunmin]]

Attachment: part1 - add flags:IsSynthesizedByTIP and InputMethodAppState in WidgetKeyboardEvent r=masayuki

rebase again

Comment 301

[C.M.Chang[:chunmin]]

Attachment: part2 - Prevent PostHandleKeyboardEvent if it's handling by IME r=masayuki

rebase again

Comment 302

[C.M.Chang[:chunmin]]

Attachment: part3 - Interface between PresShell and HardwareKeyHandler r=masayuki, smaug

rebase again

Comment 303

[C.M.Chang[:chunmin]]

Attachment: part4 - HardwareKeyHandler component r=masayuki, smaug

rebase again

Comment 304

[C.M.Chang[:chunmin]]

Attachment: part5 - Expose KeyboardEventInit dictionary r=masayuki, smaug

Comment 305

[C.M.Chang[:chunmin]]

Attachment: part6 - Expose DefaultPreventedBy* r=masayuki, smaug

rebase again

Comment 306

[C.M.Chang[:chunmin]]

Attachment: part7 - Interface between HardwareKeyHandler and Input Method App r=masayuki, smaug

rebase again

Comment 307

[C.M.Chang[:chunmin]]

Attachment: part8 - Use synthesizeNativeKey instead of synthesizeKey to test beforekey and afterkey r=masayuki

carry r+ and modify summary

Comment 308

[C.M.Chang[:chunmin]]

It seems that the related mochitest tests are passed, although part of tests on try is still pending/running. Let‘s try landing it.

Comment 309

[Ryan VanderMeulen [:RyanVM]]

Your Try pushes for this bug are showing OSX mochitest-e10s permafails in the dom/events tests.
https://treeherder.mozilla.org/logviewer.html#?job_id=18540511&repo=try

Comment 310

[C.M.Chang[:chunmin]]

I found the timeout error[0] is not directly caused by these patches. The reason of timeout is the testes following test_bug1096146.html, that would be changed to fire keyboard events by synthesizeNativeKey, couldn't focus on window. The problem is slimilar to bug 670053[1]. The root cause still need to be investigated.

However, the quick solution is to remove condition |aEvent.mIsSynthesizedByTIP| from PresShell::HandleKeyboardEvent. In b2g, the whole routing feature still works because |aEvent.mIsSynthesizedByTIP| will always be checked in |PresShell::ForwardKeyToInputMethodApp|. In other platform, the routing will work as usual. Another advantage is that we might not need to change the existing test cases anymore. If we don't need to check the key is from hardware or software in PresShell::HandleKeyboardEvent, we might not need part 8 for now. [3] shows that this approach can pass the previous timeout issues[0].

Is it ok for you? I'll land it ASAP if you agree with this approach.

[0] https://treeherder.mozilla.org/logviewer.html#?job_id=18680724&repo=try#L4105-L4106
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=670053
[2] https://bugzilla.mozilla.org/attachment.cgi?id=8734623&action=diff#a/layout/base/nsPresShell.cpp_sec6
[3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=65578892e53b

Comment 311

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to Chun-Min Chang[:chunmin] from comment #310)
> However, the quick solution is to remove condition
> |aEvent.mIsSynthesizedByTIP| from PresShell::HandleKeyboardEvent. In b2g,
> the whole routing feature still works because |aEvent.mIsSynthesizedByTIP|
> will always be checked in |PresShell::ForwardKeyToInputMethodApp|. In other
> platform, the routing will work as usual. Another advantage is that we might
> not need to change the existing test cases anymore. If we don't need to
> check the key is from hardware or software in
> PresShell::HandleKeyboardEvent, we might not need part 8 for now. [3] shows
> that this approach can pass the previous timeout issues[0].
> 
> Is it ok for you? I'll land it ASAP if you agree with this approach.

Sounds reasonable to me. Please test whole mochitests on all platforms on tryserver before landing. Changing input event path may cause unexpected result like the failure.

# And as I said before (comment 211), it might be better not to set mIsSynthesizedByTIP to true if TextEventDispatcher's input transaction is for tests, though...

Comment 312

[C.M.Chang[:chunmin]]

(In reply to Masayuki Nakano [:masayuki] (Mozilla Japan) from comment #311)
> Sounds reasonable to me. Please test whole mochitests on all platforms on
> tryserver before landing. Changing input event path may cause unexpected
> result like the failure.
Sure. There's still one test has failure, but this error will always show even when I push a empty patch[0] to test.

> # And as I said before (comment 211), it might be better not to set
> mIsSynthesizedByTIP to true if TextEventDispatcher's input transaction is
> for tests, though...
OK, I will change patch 1 to use |aKeyboardEvent.mIsSynthesizedByTIP = (mForTests)? false : true;| and push them to tryserver again.

[0] https://treeherder.mozilla.org/#/jobs?repo=try&revision=65578892e53b&selectedJob=18711858
[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=4cc259056e22&selectedJob=18711857

Comment 313

[C.M.Chang[:chunmin]]

Attachment: part1 - add flags:IsSynthesizedByTIP and InputMethodAppState in WidgetKeyboardEvent r=masayuki

Rebase to the newest master

Comment 314

[C.M.Chang[:chunmin]]

Attachment: part2 - Prevent PostHandleKeyboardEvent if it's handling by IME r=masayuki

Rebase to the newest master

Comment 315

[C.M.Chang[:chunmin]]

Attachment: part3 - Interface between PresShell and HardwareKeyHandler r=masayuki, smaug

Rebase to the newest master

Comment 316

[C.M.Chang[:chunmin]]

Attachment: part4 - HardwareKeyHandler component r=masayuki, smaug

Rebase to the newest master

Comment 317

[C.M.Chang[:chunmin]]

Attachment: part5 - Expose KeyboardEventInit dictionary r=masayuki, smaug

Rebase to the newest master

Comment 318

[C.M.Chang[:chunmin]]

Attachment: part6 - Expose DefaultPreventedBy* r=masayuki, smaug

Rebase to the newest master

Comment 319

[C.M.Chang[:chunmin]]

Attachment: part7 - Interface between HardwareKeyHandler and Input Method App r=masayuki, smaug

Rebase to the newest master

Comment 320

[C.M.Chang[:chunmin]]

The results of the r+ patches on try server[0] pass all, except one test, W(5), always shows error even when I push a empty patch to run tests. So, I guess it's nothing to do with these patches.

[0] https://treeherder.mozilla.org/#/jobs?repo=try&revision=b9e16193d389&selectedJob=18729041
[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=18d7ba810b42&selectedJob=18733766

Comment 321

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/341c531db142
https://hg.mozilla.org/integration/mozilla-inbound/rev/5d9e6b3810fe
https://hg.mozilla.org/integration/mozilla-inbound/rev/22f2994b6c2d
https://hg.mozilla.org/integration/mozilla-inbound/rev/d16d8002bf47
https://hg.mozilla.org/integration/mozilla-inbound/rev/2f19708ce619
https://hg.mozilla.org/integration/mozilla-inbound/rev/ebbd79395164
https://hg.mozilla.org/integration/mozilla-inbound/rev/2ba85d06b69b

Comment 322

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/341c531db142
https://hg.mozilla.org/mozilla-central/rev/5d9e6b3810fe
https://hg.mozilla.org/mozilla-central/rev/22f2994b6c2d
https://hg.mozilla.org/mozilla-central/rev/d16d8002bf47
https://hg.mozilla.org/mozilla-central/rev/2f19708ce619
https://hg.mozilla.org/mozilla-central/rev/ebbd79395164
https://hg.mozilla.org/mozilla-central/rev/2ba85d06b69b

Comment 324

[C.M.Chang[:chunmin]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4ad54b2f8a94