737110 - (keyboard-api) Input Method API, previously known as Virtual Keyboard API

Status: RESOLVED INCOMPLETE

(Core :: DOM: Device Interfaces, defect)

Description

[[:fabrice] Fabrice Desré]

In bug bug 736628 we turned a hack into a b2g-only API on navigator that allows virtual keyboards to send key events.

We should define a proper API that would not be specific to b2g.

Comment 1

[Mounir Lamouri (:mounir)]

Why would that API be b2g only? We will likely want developers to write virtual keyborads which means they will have to use that API which means that API will be part of the web platform. From there, I think we should discuss the API on webapi mailing list and reach a consensus. IIRC, I wrote something about that some time ago.

Comment 2

[[:fabrice] Fabrice Desré]

Mounir, comment #0 says "define a proper API that would not be specific to b2g". I think we're in agreement there...

Comment 3

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Is anyone working on the API design? If not I will write my proposal here.

Comment 4

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TW) from comment #3)
> Is anyone working on the API design? If not I will write my proposal here.

Mounir is working on drafting an API. But it won't hurt if you put your proposal here or discuss with him about it.

The API would be really simple with a similar manner to sendKey than the previous API and a way to listen for focus changes and get some informations about the newly focused element.

Comment 5

[Mounir Lamouri (:mounir)]

Proposals have been made here:
https://groups.google.com/group/mozilla.dev.webapi/browse_thread/thread/56cdfe1c6bfd34dc#

Feel free to comment in the mailing-list ;)

Comment 6

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Looking into it right now. Thanks.

Comment 7

[Yuan Xulei [:yxl] (Leave from May 18)]

A new thread based on Mounir's proposals has been created to extend current mozKeyboard API:

https://groups.google.com/forum/?fromgroups=#!topic/mozilla.dev.webapi/A7dIBaR3lpU

We're in need of comments and suggestions. ;-)

Comment 8

[Yuan Xulei [:yxl] (Leave from May 18)]

Attachment: Extended MozKeyboard API

A wiki page(https://wiki.mozilla.org/WebAPI/KeboardIME) for the extended API was created. 
This patch implemented most methods and properties of the API.

Comment 9

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

First off, the current API looks like a great start!

However, the API seems a bit focused around text fields. The problem with this is that we need an API that works well for contentEditable as well. I.e. something that also works when doing rich-text-editing. For this I think we should avoid interacting too much with the full value of the text that is being edited since that can be a quite large amount of text.

We also should try to make it possible for the keyboard app to not have to worry about style changes, so changes in color and underlining should not be something that the keyboard app should have to worry about.

I'm not at all familiar with what the requirements of CJK text input is. I also don't know much about input of arabic text. So someone that knows more about those languages should speak up for what the needs there is.

As for latin text: When the user clicks on a paragraph of text, I think we should fire an event on the InputMethodConnection that exposes the full paragraph of text. Ideal would be if we only exposed a sentence, but it's often too hard to figure out where a sentence starts and ends, so I think exposing the full paragraph is safer.

The paragraph would simply be a string of text, i.e. all style information has been removed.

All selection manipulation functions in the current API draft would then operate relative to that paragraph.

If the user moves around within the same paragraph of text, we might want to fire a different event from the "user switched paragraph" event described above. This way the keyboard app can remember context such as spelling corrections that it did that the user might want to revert.

I'd prefer not to have the setText function since that could easily result in loss of style information for the user.

Is the intent that commitText is used for spelling corrections? I wonder if we should instead have a dedicated function for that. I.e. a function which let you specify a start index and end index, and then the set of text to replace the current text with. That could replace the need for a separate deleteSurroundingText function.

Comment 10

[Yuan Xulei [:yxl] (Leave from May 18)]

@Jonas, I agree with you that we should pay special attention to the contentEditables and avoid interacting too much with the full value of the text.

setText is not suitable for the contentEditables and we only use setText for special purpose, such as clear the text field or set the value of a datetime input field.

For a typical keyboard, the most common work is sending string to current cursor position, and thus we defined a dedicated function commitText to do that. commitText was designed both for contentEditables and the plain text fields(input elements and text area elements). It only modifies the text at the cursor position and does not change the styles. As the strings are sent to current cursor position, commitText does not ask for start and end index.

To deal with spelling corrections, we need a replace function as you described, or the deleteSurroundingText combined with commitText. There also need a function to allow keyboard to remove string from text field. To satifying the two requirements, we can define a new replace fucntion("function which let you specify a start index and end index, and then the set of text to replace the current text with"), or continue to use deleteSurroundingText.

When dealing with user input, what the keyboard cares is context around the cursor of the current text field. It doesn't need to be aware of paragraphs. Also it isn't easy to figure out the range of a paragraph. So I don't prefer introducing extra complexity of "paragraph" events.

Comment 11

[(no longer active)]

I posted some feedback about the API on dev-webapi.  For the implementation, please replace the XPIDL interface with a Web IDL interface (on top of mccr8's patches on bug 851639.)  Thanks!

Comment 12

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

More mass-incompleting of FxOS bugs in the Device Interfaces component.

Please update and let me know if any of these are still valid.