Last Comment Bug 737110 - (keyboard-api) Input Method API, previously known as Virtual Keyboard API
: Input Method API, previously known as Virtual Keyboard API
Status: NEW
: dev-doc-needed
Product: Core
Classification: Components
Component: DOM: Device Interfaces (show other bugs)
: Trunk
: All All
-- normal with 2 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
: Andrew Overholt [:overholt]
Depends on: 908647 941329 946582 1092122 811177 818893 842436 844716 845171 861515 861665 885692 899073 899999 901434 902562 902830 905567 912951 915570 932145 936724 941448 944397 1057898 1092549
Blocks: webapi 3rd-party-keyboard 891669
  Show dependency treegraph
Reported: 2012-03-19 11:42 PDT by [:fabrice] Fabrice Desré
Modified: 2014-11-05 00:18 PST (History)
21 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---

Extended MozKeyboard API (41.33 KB, patch)
2013-01-31 09:30 PST, Yuan Xulei [:yxl] (Leave from May 18)
no flags Details | Diff | Splinter Review

Description User image [:fabrice] Fabrice Desré 2012-03-19 11:42:36 PDT
In bug bug 736628 we turned a hack into a b2g-only API on navigator that allows virtual keyboards to send key events.

We should define a proper API that would not be specific to b2g.
Comment 1 User image Mounir Lamouri (:mounir) 2012-03-20 03:14:23 PDT
Why would that API be b2g only? We will likely want developers to write virtual keyborads which means they will have to use that API which means that API will be part of the web platform. From there, I think we should discuss the API on webapi mailing list and reach a consensus. IIRC, I wrote something about that some time ago.
Comment 2 User image [:fabrice] Fabrice Desré 2012-03-20 07:26:37 PDT
Mounir, comment #0 says "define a proper API that would not be specific to b2g". I think we're in agreement there...
Comment 3 User image Tim Guan-tin Chien [:timdream] (please needinfo) 2012-04-25 04:46:08 PDT
Is anyone working on the API design? If not I will write my proposal here.
Comment 4 User image Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please) 2012-04-25 05:47:51 PDT
(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TW) from comment #3)
> Is anyone working on the API design? If not I will write my proposal here.

Mounir is working on drafting an API. But it won't hurt if you put your proposal here or discuss with him about it.

The API would be really simple with a similar manner to sendKey than the previous API and a way to listen for focus changes and get some informations about the newly focused element.
Comment 5 User image Mounir Lamouri (:mounir) 2012-04-25 13:51:26 PDT
Proposals have been made here:

Feel free to comment in the mailing-list ;)
Comment 6 User image Tim Guan-tin Chien [:timdream] (please needinfo) 2012-04-25 19:26:16 PDT
Looking into it right now. Thanks.
Comment 7 User image Yuan Xulei [:yxl] (Leave from May 18) 2013-01-29 21:54:34 PST
A new thread based on Mounir's proposals has been created to extend current mozKeyboard API:!topic/

We're in need of comments and suggestions. ;-)
Comment 8 User image Yuan Xulei [:yxl] (Leave from May 18) 2013-01-31 09:30:47 PST
Created attachment 708641 [details] [diff] [review]
Extended MozKeyboard API

A wiki page( for the extended API was created. 
This patch implemented most methods and properties of the API.
Comment 9 User image Jonas Sicking (:sicking) No longer reading bugmail consistently 2013-03-31 23:30:59 PDT
First off, the current API looks like a great start!

However, the API seems a bit focused around text fields. The problem with this is that we need an API that works well for contentEditable as well. I.e. something that also works when doing rich-text-editing. For this I think we should avoid interacting too much with the full value of the text that is being edited since that can be a quite large amount of text.

We also should try to make it possible for the keyboard app to not have to worry about style changes, so changes in color and underlining should not be something that the keyboard app should have to worry about.

I'm not at all familiar with what the requirements of CJK text input is. I also don't know much about input of arabic text. So someone that knows more about those languages should speak up for what the needs there is.

As for latin text: When the user clicks on a paragraph of text, I think we should fire an event on the InputMethodConnection that exposes the full paragraph of text. Ideal would be if we only exposed a sentence, but it's often too hard to figure out where a sentence starts and ends, so I think exposing the full paragraph is safer.

The paragraph would simply be a string of text, i.e. all style information has been removed.

All selection manipulation functions in the current API draft would then operate relative to that paragraph.

If the user moves around within the same paragraph of text, we might want to fire a different event from the "user switched paragraph" event described above. This way the keyboard app can remember context such as spelling corrections that it did that the user might want to revert.

I'd prefer not to have the setText function since that could easily result in loss of style information for the user.

Is the intent that commitText is used for spelling corrections? I wonder if we should instead have a dedicated function for that. I.e. a function which let you specify a start index and end index, and then the set of text to replace the current text with. That could replace the need for a separate deleteSurroundingText function.
Comment 10 User image Yuan Xulei [:yxl] (Leave from May 18) 2013-04-08 01:43:15 PDT
@Jonas, I agree with you that we should pay special attention to the contentEditables and avoid interacting too much with the full value of the text.

setText is not suitable for the contentEditables and we only use setText for special purpose, such as clear the text field or set the value of a datetime input field.

For a typical keyboard, the most common work is sending string to current cursor position, and thus we defined a dedicated function commitText to do that. commitText was designed both for contentEditables and the plain text fields(input elements and text area elements). It only modifies the text at the cursor position and does not change the styles. As the strings are sent to current cursor position, commitText does not ask for start and end index.

To deal with spelling corrections, we need a replace function as you described, or the deleteSurroundingText combined with commitText. There also need a function to allow keyboard to remove string from text field. To satifying the two requirements, we can define a new replace fucntion("function which let you specify a start index and end index, and then the set of text to replace the current text with"), or continue to use deleteSurroundingText.

When dealing with user input, what the keyboard cares is context around the cursor of the current text field. It doesn't need to be aware of paragraphs. Also it isn't easy to figure out the range of a paragraph. So I don't prefer introducing extra complexity of "paragraph" events.
Comment 11 User image :Ehsan Akhgari 2013-04-15 07:26:30 PDT
I posted some feedback about the API on dev-webapi.  For the implementation, please replace the XPIDL interface with a Web IDL interface (on top of mccr8's patches on bug 851639.)  Thanks!

Note You need to log in before you can comment on or make changes to this bug.