797170 - Add keyboard auto-correct

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Keyboard, defect)

Description

[David Flanagan [:djf]]

Currently the keyboard can display suggestions, but won't insert any of them unless you tap.  

When the highest ranked suggestion is much more highly ranked than the current input, it should be automatically inserted if the user types space.

It must also be highlighted in some way to indicate that this will happen.

This is a requirement from page 10 of https://www.dropbox.com/sh/pz8qn4023f19dhr/zLhkCsxcbB/Keyboard%20Specs.pdf

Comment 1

[David Flanagan [:djf]]

Currently we've got word suggestions that seem to work pretty well. But if you want one you have to select it manually: they won't just be inserted when you type a space like they will in Android.

I think it would take a couple of days to go from our current word suggestions to the true auto-correct feature, that we already have UX specs for. I'm nominating this as blocking to get a decision about whether I should implement that.

Comment 2

[David Flanagan [:djf]]

This doesn't block 797151. We can enable the word suggestions by default without having auto correct.  But this does depend on 797171

Comment 3

[Alex Keybl [:akeybl]]

blocking-basecamp- for now, this is a feature and may open up a can of worms. Faramarz is going to follow up with UX to double-verify this decision.

Comment 4

[Josh Carpenter [:jcarpenter]]

Flagging as UX Most Wanted. This dramatically improves the utility of Word Suggestions, to the point of being a must have.

Comment 6

[Josh Carpenter [:jcarpenter]]

UX to follow up on this. We need to spec out how the suggested word should appear when the user has Word Suggestions turned off.

Comment 7

[Gregor Wagner [:gwagner]]

(In reply to David Flanagan [:djf] from comment #1)
> Currently we've got word suggestions that seem to work pretty well. But if
> you want one you have to select it manually: they won't just be inserted
> when you type a space like they will in Android.
> 

Well I personally hate this "feature" because it doesn't let you write words in other languages and writing a whole SMS in German for example without switching the language is nearly impossible. So I really like to see a pref for this or I will turn prediction off again because its not usable for me.

Comment 8

[Josh Carpenter [:jcarpenter]]

Good call, Gregor. We'll absolutely want to make this something users can turn on / off.

Comment 9

[David Flanagan [:djf]]

Gregor,

How do Android or Apple handle this multiple-language issue, do you know? I haven't really looked at it carefully, but I think that if you backspace after an auto-correction it will restore what you had typed and won't do the auto-correction again.

As a bi-lingual texter, would that be good enough for you?  This also assumes that auto correction is only done when one of the word suggestions is very close to what is typed. It would not replace a german word with the closest english word unless the english word was actually quite close to the german word.

Comment 10

[David Flanagan [:djf]]

Did someone assign me to this recently, or did I assign myself way back in October when I filed the bug I wonder?

Josh, is this something under consideration for Leo?

Comment 11

[Josh Carpenter [:jcarpenter]]

(In reply to David Flanagan [:djf] from comment #10)
> Did someone assign me to this recently, or did I assign myself way back in
> October when I filed the bug I wonder?
> 
> Josh, is this something under consideration for Leo?

I think you were assigned way back when :)

Re: Leo, absolutely we want to have it in. How we make that happen is an open question, but we're working on that.

Comment 12

[Josh Carpenter [:jcarpenter]]

Adding Michael, Christoph and Rudy, per this morning's conversation.

Comment 13

[Gregor Wagner [:gwagner]]

(In reply to David Flanagan [:djf] from comment #9)
> Gregor,
> 
> How do Android or Apple handle this multiple-language issue, do you know? I
> haven't really looked at it carefully, but I think that if you backspace
> after an auto-correction it will restore what you had typed and won't do the
> auto-correction again.


It doesn't do it for me on android. I usually disable auto-correction. I was super happy with our current approach where we just show the prediction results.


> 
> As a bi-lingual texter, would that be good enough for you?  This also
> assumes that auto correction is only done when one of the word suggestions
> is very close to what is typed. It would not replace a german word with the
> closest english word unless the english word was actually quite close to the
> german word.

Well you don't know if the word is German or English. We don't load 2 separate word lists. So my problem is that some German words are similar to frequently used English words and the auto correction thinks its a typo and changes it automatically to the wrong word.

Comment 14

[David Flanagan [:djf]]

(In reply to Gregor Wagner [:gwagner] from comment #13)

> Well you don't know if the word is German or English. We don't load 2
> separate word lists. So my problem is that some German words are similar to
> frequently used English words and the auto correction thinks its a typo and
> changes it automatically to the wrong word.

Right, so users like you would appreciate an easy way to indicate that you really want to use what you typed. Our keyboard spec says that tapping back-space after an auto correct will restore the original text.  So that would mean one extra keystroke for each erroneous auto-correction. Would that be good enough do you think? That's what Android does on my phone.

I suppose we could go further and put a dismiss button in the word suggestion area to dismiss all suggestions and corrections for just that one interaction wit the keyboard.  So if you were going to send an English text you use autocorrect and when sending a german text you turn them off for just that one text. I don't know what UX would think of that...

Comment 15

[David Flanagan [:djf]]

In order to do auto-correction well, we're going to need:

1) the ability to highlight the suggested word, if any, that will be used for auto-correction if the user types space or punctuation as the next key. This means new code in render.js for displaying highlights, and a new data structure for passing candidates to render.js to indicate which, if any, is the auto-correct candidate.

2) some way to decide whether a word suggestion is a close enough match that it ought to be used for auto-correction. We don't want to just use the best match, we only want matches that are actually close matches

3) to remember the user's original text when it gets auto-corrected so we can restore it if the next keystroke is backspace.

4) to turn off auto correction for the current word if the user cancels a correction with backspace.

5) if one of the current candidates is going to be used as an autocorrection, we should probably always show the current literal text as a candidate too, so the user can accept it without correction by just tapping the candidate. This lets the user do what they want with one tap instead of means one space, backspace, space.

6) A setting, and UX in the settings app, for turning auto correction on and off independently of word suggestions.  Android actually has a configuration option for how aggressive auto-correct is. We could do that too, if we want, but it seems like overkill to me for now.

Comment 16

[David Flanagan [:djf]]

Josh,

Is there a UX spec for how we turn auto-correction on and off in the settings app?

I assume that if word suggestions are off, then auto-correction is automatically off.  But if word suggestions are on, then the user can choose auto correction or not.

Also, do we want auto-correction to be on/off, or do we want varying degrees of aggressiveness, like Android 4.2 has?  (I think that varying the amount of aggressiveness will be technically easy, so this is a pure UX question).

Comment 17

[David Flanagan [:djf]]

Andreas has a work-in-progress auto-correction patch at https://github.com/mozilla-b2g/gaia/pull/8962

It looks like it does not have some of the features in comment 15, but should be a good starting place, or at least somewhere to copy code from.

Comment 18

[Andreas Gal :gal]

We should do this iteratively. Lets finish the patch and add simple auto corrections. iPhone does this really well without any config switches and nobs. Backspacing and correcting automatically disables the suggestion for that word.

Comment 19

[Josh Carpenter [:jcarpenter]]

(In reply to David Flanagan [:djf] from comment #16)
> Josh,
> 
> Is there a UX spec for how we turn auto-correction on and off in the
> settings app?

Hi David, no specs yet but we're rushing to have something as it's a top priority for us. IxD and VsD for both keyboard and Settings app changes.

> I assume that if word suggestions are off, then auto-correction is
> automatically off.  But if word suggestions are on, then the user can choose
> auto correction or not.

They should be independent settings. Users may still want auto corrections without the vertical-space hogging of Word Suggestions. That's my personal setup on Android, for example.

> Also, do we want auto-correction to be on/off, or do we want varying degrees
> of aggressiveness, like Android 4.2 has?  (I think that varying the amount
> of aggressiveness will be technically easy, so this is a pure UX question).

Per Andreas, I think simple on/off is a good start, and we can iterate from there.

Comment 20

[David Flanagan [:djf]]

I don't have any code yet on this, and am going on PTO this afternoon and tomorrow, so am unlikely to make any progress until next week.

(Andreas: if that is a problem, feel free to take the bug and resume work on your github patch).

Meanwhile, I've filed three dependent bugs on the prediction engine, so there's still stuff to work on under the hood before we can get the auto-correction UX hooked up in this bug.

Comment 21

[David Flanagan [:djf]]

Andreas: one of the features you've indicated (by email) that you'd like to see for auto-correction is conversion of "s" to "a", like iOS does. Similarly, "o" to "I", I suspect.

I tried this out on my son's iPod and noticed that if I hit the s key right in the center, then I don't get an auto-correction. It is only when I hit the left side of the s key, closer to the a, that the auto-correction happens.

This is pretty awesome, actually, and something that we should aspire to, but almost certainly not a feature we will have for 1.1. The prediction engine has all of the layout data it needs (full rectangles for each key) to figure out not just what the nearest neighbors of any tap are, but also to figure out how close each one is.  I'll file a bug on the prediction engine to make this work.

Comment 22

[Josh Carpenter [:jcarpenter]]

(In reply to David Flanagan [:djf] from comment #21)
> Andreas: one of the features you've indicated (by email) that you'd like to
> see for auto-correction is conversion of "s" to "a", like iOS does.
> Similarly, "o" to "I", I suspect.
> 
> I tried this out on my son's iPod and noticed that if I hit the s key right
> in the center, then I don't get an auto-correction. It is only when I hit
> the left side of the s key, closer to the a, that the auto-correction
> happens.
> 
> This is pretty awesome, actually, and something that we should aspire to,
> but almost certainly not a feature we will have for 1.1. The prediction
> engine has all of the layout data it needs (full rectangles for each key) to
> figure out not just what the nearest neighbors of any tap are, but also to
> figure out how close each one is.  I'll file a bug on the prediction engine
> to make this work.

I believe this is an example of dynamic hit target sizing, per bug #856521. This technique has been one of the backbones of effective touch screen keyboards since iOS v1. Something our users would benefit from tremendously if we find a way to incorporate our variation on the technique into future versions of FFOS.

Comment 23

[David Flanagan [:djf]]

My work in progress (does not work yet) is on the "autocorrect" branch in my github repo: https://github.com/davidflanagan/gaia/tree/autocorrect

Comment 24

[David Flanagan [:djf]]

Its still a work in progress, but it is ready to try out: https://github.com/mozilla-b2g/gaia/pull/9052

Comment 25

[David Flanagan [:djf]]

I've created the new bug 860462 for the initial patch that turns auto-correction on.  I'll attach the pull request from comment 24 to that bug instead of here.

Comment 26

[David Flanagan [:djf]]

Complaints about bad auto-corrections are actively solicited. Please list them by commenting in bug 860550

Comment 27

[David Flanagan [:djf]]

Now that this is an umbrella bug, the [LOE:S] doesn't make sense, so removing it.

Comment 28

[David Flanagan [:djf]]

Adding bug 838227 to the list of dependencies because we won't be able to uplift any of the improvements to that patch without first uplifting that one, and I don't want to forget about it.

Comment 29

[David Flanagan [:djf]]

Adding 865484 as a dependency and moving a bunch of other bugs to be dependent on that one.

Comment 30

[David Flanagan [:djf]]

I've landed 865484 which was a huge one.  865494 may no longer be necessary, actually.

Comment 31

[Jan Jongboom [:janjongboom] (inactive)]

All depending issues have been closed, can we close the meta bug or?

Comment 32

[David Flanagan [:djf]]

QA is testing auto correct today and will file any bugs they find as blocking this bug.  So let's leave this open until we get through that.

Comment 33

[Tony Chung [:tchung]]

per discussion, if this is to get uplifted, we will need to flag if the dependent bug gets leo+'d.   Given ther'es so many, i'll request this for leo? until triage thinks meta bug should not be flagged but the individual ones do.

Comment 34

[Alex Keybl [:akeybl]]

(In reply to Tony Chung [:tchung] from comment #33)
> per discussion, if this is to get uplifted, we will need to flag if the
> dependent bug gets leo+'d.   Given ther'es so many, i'll request this for
> leo? until triage thinks meta bug should not be flagged but the individual
> ones do.

Not a blocker but we will uplift today/tomorrow - we're  going to approve the rollup patch (a=akeybl already, in case I miss the nom).

Comment 35

[David Flanagan [:djf]]

Auto correct is feature complete.  
All the blocking bugs are closed.
We are done here!
Uplift to v1-train will happen in bug 873934

Comment 36

[David Flanagan [:djf]]

Uplifted to v1-train in bug 873934