1121978 - Test test_keyboard_predictive_key.py is failing intermittently, wrong predictive word is selected

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Viorela Ioia [:viorela]]

Test test_keyboard_predictive_key.py has failed in latest mozilla-central build: http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.mozilla-central.ui.functional.smoke/262/HTML_Report/
I've been able to reproduce the failure locally, by running the automated test several times; It is failing intermittently, repro rate: 1/10. The issue is not reproducible manually.
The test failed because the wrong predictive word was selected(the 2nd word, instead of the 1st)

Traceback (most recent call last):
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.ui.functional.non-smoke.2/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/marionette_test.py", line 268, in run
testMethod()
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.ui.functional.non-smoke.2/tests/python/gaia-ui-tests/gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py", line 64, in test_keyboard_predictive_key
self.assertEqual(keyboard_page.text_input, 'keyboard Tea he ')
AssertionError: u'keyboard Tea yes ' != 'keyboard Tea he '

Device firmware (base) 	L1TC10011880
Device firmware (date) 	15 Jan 2015 01:30:42
Device firmware (incremental) 	eng.cltbld.20150115.043033
Device firmware (release) 	4.4.2
Device identifier 	flame
Gaia date 	14 Jan 2015 08:21:14
Gaia revision 	bcc76f93f565
Gecko build 	20150115010229
Gecko revision 	c1f6345f2803
Gecko version 	38.0a1

Comment 1

[Viorela Ioia [:viorela]]

I've been investigating this issue and I was able to reproduce the exact failure while using the device in landscape orientation. Typing 'ye'(mispelled word), then pressing space returns 'yes' instead of 'he' .
In test_keyboard_predictive_key.py the word we expect to get after autocorrecting 'ye' is 'he', and not 'yes': https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py#L64

This issue is also reproducible manually. 
STR from the test:
Prerequisities:
Make sure your phone is in landscape orientation.
1. Launch Ui-tests app
2. Select Keyboard option in UI tab
3. In text input field type 'ye'

#Expected results:
The mispelled word 'ye' is autocorrected and 'he' is displayed instead

#Actual results:
The mispelled word 'ye' is autocorrected and 'yes' is displayed instead

I am not sure if this is an actual bug or it is expected that writing the same misspelled word to give us different autocorrect words list in landscape and portrait.

Comment 2

[Christoph Kerschbaumer [:ckerschb]]

(In reply to Viorela Ioia [:viorela] from comment #1)
> I've been investigating this issue and I was able to reproduce the exact
> failure while using the device in landscape orientation. Typing
> 'ye'(mispelled word), then pressing space returns 'yes' instead of 'he' .
> In test_keyboard_predictive_key.py the word we expect to get after
> autocorrecting 'ye' is 'he', and not 'yes':
> https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/
> gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py#L64
> 
> This issue is also reproducible manually. 
> STR from the test:
> Prerequisities:
> Make sure your phone is in landscape orientation.
> 1. Launch Ui-tests app
> 2. Select Keyboard option in UI tab
> 3. In text input field type 'ye'
> 
> #Expected results:
> The mispelled word 'ye' is autocorrected and 'he' is displayed instead
> 
> #Actual results:
> The mispelled word 'ye' is autocorrected and 'yes' is displayed instead
> 
> I am not sure if this is an actual bug or it is expected that writing the
> same misspelled word to give us different autocorrect words list in
> landscape and portrait.

As far as I know we should provide the same prediction whether the phone is landscape orientation or not. I have not been working on the engine for quite some time - forwarding the needinfo to David.

Comment 3

[David Flanagan [:djf]]

The predictions try to correct mistyping, and assume that the most likely typing errors are when the user presses a nearby key. So predictions are based on which keys are closest. And landscape and portrait keyboards have different dimensions and different key spacings. So what you're seeing is a side effect of that. Both 'he' and 'yes' seem like good corrections for 'ye', so I don't think this is a bug. 

I'd suggest that you work around it by finding a different test case (maybe a longer word) that autocorrects the same way in both orientations.  Or maybe by locking the orientation of the app so it runs consistently in the orientation you expect. Is there any way to do that with marionette? If automated tests are running in different orientations intermittently, this seems like it could be the source of other unexpected bugs.

Comment 4

[Viorela Ioia [:viorela]]

Thanks, David! 
I will add a check that the device is in portrait mode, as expected, in the beginning of the test.

Comment 5

[GitHub Autolander Bot]

Attachment: [PullReq] viorelaioia:bug_1121978 to mozilla-b2g:master

Comment 6

[Florin Strugariu [:Bebe]]

https://github.com/mozilla-b2g/gaia/commit/6ef10ac58f5bce320d844e19aa8e0badf84568bd

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Reverted for the exact same failures your Gaia Try run hit. Please verify that the tests are actually passing before blindly landing in the future.
Master: https://github.com/mozilla-b2g/gaia/commit/0dd0dc56cce9b69f3a2b7a11a4e08b2f83d317a5

Comment 8

[GitHub Autolander Bot]

Attachment: [gaia] viorelaioia:bug-1121978 > mozilla-b2g:master

Comment 9

[Florin Strugariu [:Bebe]]

Comment on attachment 8562672 [details] [review]
[gaia] viorelaioia:bug-1121978 > mozilla-b2g:master

https://github.com/mozilla-b2g/gaia/commit/e280a660955bbdab265d50f8d9e009de34082332

Comment 10

[Oliver Nelson [:oliverthor]]

Attachment: mar03_predictivekey.html

Issue is reproducing in latest mozilla-central-nightly runs on Jenkins.

http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.1/62/testReport/junit/%28root%29/test_keyboard_predictive_key_py%20TestKeyboardPredictiveKey_test_keyboard_predictive_key/test_keyboard_predictive_key_py_TestKeyboardPredictiveKey_test_keyboard_predictive_key/history/

Traceback (most recent call last):
  File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.1/.env/local/lib/python2.7/site-packages/marionette_client-0.8.7-py2.7.egg/marionette/marionette_test.py", line 283, in run
    testMethod()
  File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.1/tests/python/gaia-ui-tests/gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py", line 67, in test_keyboard_predictive_key
    self.assertEqual(keyboard_page.text_input, 'keyboard Tea he ')

AssertionError: u'keyboard Tea ye ' != 'keyboard Tea he '



Attaching html report of run via local automation (7/10 passing)

Comment 11

[Martijn Wargers (dead)]

Apparently, the fix didn't work, I'll see if I can fix it.

Comment 12

[Martijn Wargers (dead)]

When this failure happens, the phone was in portrait mode, btw, see the report: http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.1/45/HTML_Report/

Comment 13

[Martijn Wargers (dead)]

Attachment: predictive.diff

I tried this, but I still got a failure once in 20 times, while running this:
REPEAT 1
-------
TEST-START | test_keyboard_predictive_key.py TestKeyboardPredictiveKey.test_keyboard_predictive_key
TEST-UNEXPECTED-ERROR | test_keyboard_predictive_key.py TestKeyboardPredictiveKey.test_keyboard_predictive_key | TimeoutException: TimeoutException: Timed out after 10.0 seconds


Traceback (most recent call last):
  File "/Users/mwargers/.virtualenvs/gaia-py-latest/lib/python2.7/site-packages/marionette_client-0.9-py2.7.egg/marionette/marionette_test.py", line 290, in run
    testMethod()
  File "/Users/mwargers/B2G/gaia_clean/tests/python/gaia-ui-tests/gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py", line 75, in test_keyboard_predictive_key
    Wait(self.marionette).until(lambda m: keyboard_page.text_input == 'keyboard Tea he wit.')
  File "/Users/mwargers/.virtualenvs/gaia-py-latest/lib/python2.7/site-packages/marionette_driver-0.2-py2.7.egg/marionette_driver/wait.py", line 143, in until
    cause=last_exc)
TEST-INFO took 103898ms

It makes me wonder if there isn't a genuine bug here somehow.

Comment 14

[Martijn Wargers (dead)]

Perhaps this issue happens somehow when there is some longer text in the input? (so the corrected word is scrolled).

Comment 15

[Martijn Wargers (dead)]

Comment on attachment 8572718 [details] [diff] [review]
predictive.diff

Even though this doesn't seem to fix the issue, it might still be useful to check this in. I think using Wait() is more appropriate here.

Comment 16

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

I really think it's sketchy to do a test based on assumptions about how the internal dictionary will correct. Is it context sensitive to what's already been typed in the past, making this possibly non-deterministic? Plainly it can make different decisions as we progress builds, and they're not bugs.

But if we do this, I think we need to take djf's comment 3 to heart and quit trying to do this on three letter, ambiguous-in-English words. Just making it a string of multiple small words isn't going to fix that.

I think we should change this to a single, distinct English word for which there's only one reasonable completion when most of it is typed and no near-corrections. That's difficult because you need one that has no possible suffixes which might be displayed before the main word. That more or less limits it to idiosyncratic adverbs and semi-proper names that don't take regular plurals.

Try:

"edelw" --> only result should be "edelweiss"

That tests completion. If you want misspelling + completion:

"edels" --> only result should be "edelweiss"

Just misspelling:

"edelseiss" --> only result should be "edelweiss"

There are some other words that behave similarly (backstage is one I found) but "edelweiss" is the only one I found easily that becomes absolutely unique within a few keystrokes.

Comment 17

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Should mention that I tried this manually in both orientations and it behaved as I said. 

If we pick a different word for some reason, that should be attempted manually prior to committing the test. But it really does need to be a long, complex word to make this at all robust.

Comment 18

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8572718 [details] [diff] [review]
predictive.diff

As sometimes the word takes some time to be replaced (saw it manually), this looks like a good way to fix the intermittent issue!

Agreed on the dictionary problem.

Comment 19

[Martijn Wargers (dead)]

(In reply to Johan Lorenzo [:jlorenzo] (QA) from comment #18)
> As sometimes the word takes some time to be replaced (saw it manually), this
> looks like a good way to fix the intermittent issue!

Yes, although, I still saw one time a failure with this applied, it certainly wouldn't make things worse, at least.

> Agreed on the dictionary problem.

Me too, but I think it's better to do that in a separate bug, so I filed bug 1139976 for it.

Comment 20

[GitHub Autolander Bot]

Attachment: [gaia] mwargers:predictive > mozilla-b2g:master

Comment 21

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8573333 [details] [review]
[gaia] mwargers:predictive > mozilla-b2g:master

No changes in the patch since feedback. I don't see any reason to not r+ it.

Comment 22

[No-Jun Park [:njpark]]

Comment on attachment 8573333 [details] [review]
[gaia] mwargers:predictive > mozilla-b2g:master

Simple conversion to Wait() statement, looks good to me

Comment 23

[Martijn Wargers (dead)]

(In reply to No-Jun Park [:njpark] from comment #22)
> Comment on attachment 8573333 [details] [review]
> [gaia] mwargers:predictive > mozilla-b2g:master
> 
> Simple conversion to Wait() statement, looks good to me

Merged: https://github.com/mozilla-b2g/gaia/commit/78eed3afcad407acb543682263d81b0e72ca7e69

Comment 25

[Cynthia Tang [:cynthiatang]]

This issue can be reproduced on Flame v2.2. Thanks.

Device firmware (base)	L1TC100118D0
Device firmware (date)	28 Apr 2015 16:23:30
Device firmware (incremental)	eng.cltbld.20150428.042318
Device firmware (release)	4.4.2
Device identifier	flame
Gaia date	28 Apr 2015 04:41:33
Gaia revision	9f6b1b908266
Gecko build	20150428002500
Gecko revision	e79c19bf19bf
Gecko version	37.0


Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/B2G.v2.2.pvt.flame.gaiatest.non-smoketest/tests/python/gaia-ui-tests/.env/local/lib/python2.7/site-packages/marionette_client_mozilla_b2g37_v2_2-0.1-py2.7.egg/marionette/marionette_test.py", line 268, in run
    testMethod()
  File "/var/lib/jenkins/workspace/B2G.v2.2.pvt.flame.gaiatest.non-smoketest/tests/python/gaia-ui-tests/gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py", line 69, in test_keyboard_predictive_key
    self.assertEqual(keyboard_page.text_input, 'keyboard Tea he wit.')
AssertionError: u'keyboard Tea he wot.' != 'keyboard Tea he wit.'

Comment 26

[Martijn Wargers (dead)]

Still failing in mozilla-central in case you're wondering: http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2/278/

Comment 27

[Martijn Wargers (dead)]

Here a failure when the device is in landscape mode:
http://jenkins1.qa.scl3.mozilla.com/view/Mozilla%20Lab/job/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2/443/HTML_Report/

Comment 28

[BMO Automation]

Firefox OS is not being worked on