Bug 737110 (keyboard-api)

Input Method API, previously known as Virtual Keyboard API

NEW
Unassigned

Status

()

Core
DOM: Device Interfaces
6 years ago
3 years ago

People

(Reporter: fabrice, Unassigned)

Tracking

(Depends on: 4 bugs, Blocks: 1 bug, {dev-doc-needed})

Trunk
dev-doc-needed
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [FT:System-Platform], URL)

Attachments

(1 obsolete attachment)

(Reporter)

Description

6 years ago
In bug bug 736628 we turned a hack into a b2g-only API on navigator that allows virtual keyboards to send key events.

We should define a proper API that would not be specific to b2g.
Why would that API be b2g only? We will likely want developers to write virtual keyborads which means they will have to use that API which means that API will be part of the web platform. From there, I think we should discuss the API on webapi mailing list and reach a consensus. IIRC, I wrote something about that some time ago.
(Reporter)

Comment 2

6 years ago
Mounir, comment #0 says "define a proper API that would not be specific to b2g". I think we're in agreement there...
Blocks: 673923
Summary: API for a keyboard interface → Virtual Keyboard API
Version: Other Branch → Trunk
Is anyone working on the API design? If not I will write my proposal here.
(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TW) from comment #3)
> Is anyone working on the API design? If not I will write my proposal here.

Mounir is working on drafting an API. But it won't hurt if you put your proposal here or discuss with him about it.

The API would be really simple with a similar manner to sendKey than the previous API and a way to listen for focus changes and get some informations about the newly focused element.
Proposals have been made here:
https://groups.google.com/group/mozilla.dev.webapi/browse_thread/thread/56cdfe1c6bfd34dc#

Feel free to comment in the mailing-list ;)
Looking into it right now. Thanks.
Keywords: dev-doc-needed
Blocks: 816869
Depends on: 818893
A new thread based on Mounir's proposals has been created to extend current mozKeyboard API:

https://groups.google.com/forum/?fromgroups=#!topic/mozilla.dev.webapi/A7dIBaR3lpU

We're in need of comments and suggestions. ;-)
Created attachment 708641 [details] [diff] [review]
Extended MozKeyboard API

A wiki page(https://wiki.mozilla.org/WebAPI/KeboardIME) for the extended API was created. 
This patch implemented most methods and properties of the API.
Depends on: 842436
Depends on: 844716
Depends on: 845171
First off, the current API looks like a great start!

However, the API seems a bit focused around text fields. The problem with this is that we need an API that works well for contentEditable as well. I.e. something that also works when doing rich-text-editing. For this I think we should avoid interacting too much with the full value of the text that is being edited since that can be a quite large amount of text.

We also should try to make it possible for the keyboard app to not have to worry about style changes, so changes in color and underlining should not be something that the keyboard app should have to worry about.

I'm not at all familiar with what the requirements of CJK text input is. I also don't know much about input of arabic text. So someone that knows more about those languages should speak up for what the needs there is.

As for latin text: When the user clicks on a paragraph of text, I think we should fire an event on the InputMethodConnection that exposes the full paragraph of text. Ideal would be if we only exposed a sentence, but it's often too hard to figure out where a sentence starts and ends, so I think exposing the full paragraph is safer.

The paragraph would simply be a string of text, i.e. all style information has been removed.

All selection manipulation functions in the current API draft would then operate relative to that paragraph.

If the user moves around within the same paragraph of text, we might want to fire a different event from the "user switched paragraph" event described above. This way the keyboard app can remember context such as spelling corrections that it did that the user might want to revert.

I'd prefer not to have the setText function since that could easily result in loss of style information for the user.

Is the intent that commitText is used for spelling corrections? I wonder if we should instead have a dedicated function for that. I.e. a function which let you specify a start index and end index, and then the set of text to replace the current text with. That could replace the need for a separate deleteSurroundingText function.
@Jonas, I agree with you that we should pay special attention to the contentEditables and avoid interacting too much with the full value of the text.

setText is not suitable for the contentEditables and we only use setText for special purpose, such as clear the text field or set the value of a datetime input field.

For a typical keyboard, the most common work is sending string to current cursor position, and thus we defined a dedicated function commitText to do that. commitText was designed both for contentEditables and the plain text fields(input elements and text area elements). It only modifies the text at the cursor position and does not change the styles. As the strings are sent to current cursor position, commitText does not ask for start and end index.

To deal with spelling corrections, we need a replace function as you described, or the deleteSurroundingText combined with commitText. There also need a function to allow keyboard to remove string from text field. To satifying the two requirements, we can define a new replace fucntion("function which let you specify a start index and end index, and then the set of text to replace the current text with"), or continue to use deleteSurroundingText.

When dealing with user input, what the keyboard cares is context around the cursor of the current text field. It doesn't need to be aware of paragraphs. Also it isn't easy to figure out the range of a paragraph. So I don't prefer introducing extra complexity of "paragraph" events.
Depends on: 861515
Depends on: 861665
I posted some feedback about the API on dev-webapi.  For the implementation, please replace the XPIDL interface with a Web IDL interface (on top of mccr8's patches on bug 851639.)  Thanks!
Alias: keyboard-api

Updated

4 years ago
Depends on: 885692

Updated

4 years ago
Blocks: 891669
Blocks: 899999
No longer blocks: 899999
Depends on: 899999
Depends on: 901434
Depends on: 902562
Depends on: 902830
Depends on: 899073
Depends on: 811177
Depends on: 905567
Keywords: dev-doc-needed
Depends on: 908647
Keywords: dev-doc-needed
Depends on: 912951
Depends on: 915570
Attachment #708641 - Attachment is obsolete: true
Depends on: 932145
Depends on: 941329
Depends on: 941448
Depends on: 946582
Depends on: 944397
Summary: Virtual Keyboard API → Input Method API, previously known as Virtual Keyboard API
Whiteboard: [FT:System-Platform]
Depends on: 1092122
Depends on: 1057898
Depends on: 1092549
Depends on: 936724
You need to log in before you can comment on or make changes to this bug.