Closed Bug 1121978 Opened 5 years ago Closed 2 years ago

Test test_keyboard_predictive_key.py is failing intermittently, wrong predictive word is selected

Categories

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

ARM
Gonk (Firefox OS)
defect
Not set

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: viorela, Unassigned)

References

()

Details

Attachments

(4 files, 1 obsolete file)

Test test_keyboard_predictive_key.py has failed in latest mozilla-central build: http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.mozilla-central.ui.functional.smoke/262/HTML_Report/
I've been able to reproduce the failure locally, by running the automated test several times; It is failing intermittently, repro rate: 1/10. The issue is not reproducible manually.
The test failed because the wrong predictive word was selected(the 2nd word, instead of the 1st)

Traceback (most recent call last):
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.ui.functional.non-smoke.2/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/marionette_test.py", line 268, in run
testMethod()
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.ui.functional.non-smoke.2/tests/python/gaia-ui-tests/gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py", line 64, in test_keyboard_predictive_key
self.assertEqual(keyboard_page.text_input, 'keyboard Tea he ')
AssertionError: u'keyboard Tea yes ' != 'keyboard Tea he '

Device firmware (base) 	L1TC10011880
Device firmware (date) 	15 Jan 2015 01:30:42
Device firmware (incremental) 	eng.cltbld.20150115.043033
Device firmware (release) 	4.4.2
Device identifier 	flame
Gaia date 	14 Jan 2015 08:21:14
Gaia revision 	bcc76f93f565
Gecko build 	20150115010229
Gecko revision 	c1f6345f2803
Gecko version 	38.0a1
Assignee: nobody → viorela.ioia
QA Whiteboard: [fxosqa-auto-s8]
I've been investigating this issue and I was able to reproduce the exact failure while using the device in landscape orientation. Typing 'ye'(mispelled word), then pressing space returns 'yes' instead of 'he' .
In test_keyboard_predictive_key.py the word we expect to get after autocorrecting 'ye' is 'he', and not 'yes': https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py#L64

This issue is also reproducible manually. 
STR from the test:
Prerequisities:
Make sure your phone is in landscape orientation.
1. Launch Ui-tests app
2. Select Keyboard option in UI tab
3. In text input field type 'ye'

#Expected results:
The mispelled word 'ye' is autocorrected and 'he' is displayed instead

#Actual results:
The mispelled word 'ye' is autocorrected and 'yes' is displayed instead

I am not sure if this is an actual bug or it is expected that writing the same misspelled word to give us different autocorrect words list in landscape and portrait.
Flags: needinfo?(mozilla)
QA Whiteboard: [fxosqa-auto-s8] → [fxosqa-auto-from-s8][fxosqa-auto-s9]
(In reply to Viorela Ioia [:viorela] from comment #1)
> I've been investigating this issue and I was able to reproduce the exact
> failure while using the device in landscape orientation. Typing
> 'ye'(mispelled word), then pressing space returns 'yes' instead of 'he' .
> In test_keyboard_predictive_key.py the word we expect to get after
> autocorrecting 'ye' is 'he', and not 'yes':
> https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/
> gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py#L64
> 
> This issue is also reproducible manually. 
> STR from the test:
> Prerequisities:
> Make sure your phone is in landscape orientation.
> 1. Launch Ui-tests app
> 2. Select Keyboard option in UI tab
> 3. In text input field type 'ye'
> 
> #Expected results:
> The mispelled word 'ye' is autocorrected and 'he' is displayed instead
> 
> #Actual results:
> The mispelled word 'ye' is autocorrected and 'yes' is displayed instead
> 
> I am not sure if this is an actual bug or it is expected that writing the
> same misspelled word to give us different autocorrect words list in
> landscape and portrait.

As far as I know we should provide the same prediction whether the phone is landscape orientation or not. I have not been working on the engine for quite some time - forwarding the needinfo to David.
Flags: needinfo?(mozilla) → needinfo?(dflanagan)
The predictions try to correct mistyping, and assume that the most likely typing errors are when the user presses a nearby key. So predictions are based on which keys are closest. And landscape and portrait keyboards have different dimensions and different key spacings. So what you're seeing is a side effect of that. Both 'he' and 'yes' seem like good corrections for 'ye', so I don't think this is a bug. 

I'd suggest that you work around it by finding a different test case (maybe a longer word) that autocorrects the same way in both orientations.  Or maybe by locking the orientation of the app so it runs consistently in the orientation you expect. Is there any way to do that with marionette? If automated tests are running in different orientations intermittently, this seems like it could be the source of other unexpected bugs.
Flags: needinfo?(dflanagan)
Thanks, David! 
I will add a check that the device is in portrait mode, as expected, in the beginning of the test.
Attachment #8562002 - Flags: review?(robert.chira)
Attachment #8562002 - Flags: review?(florin.strugariu)
Attachment #8562002 - Flags: review?(robert.chira) → review+
Attachment #8562002 - Flags: review?(florin.strugariu) → review+
Reverted for the exact same failures your Gaia Try run hit. Please verify that the tests are actually passing before blindly landing in the future.
Master: https://github.com/mozilla-b2g/gaia/commit/0dd0dc56cce9b69f3a2b7a11a4e08b2f83d317a5
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → FIXED
Issue is reproducing in latest mozilla-central-nightly runs on Jenkins.

http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.1/62/testReport/junit/%28root%29/test_keyboard_predictive_key_py%20TestKeyboardPredictiveKey_test_keyboard_predictive_key/test_keyboard_predictive_key_py_TestKeyboardPredictiveKey_test_keyboard_predictive_key/history/

Traceback (most recent call last):
  File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.1/.env/local/lib/python2.7/site-packages/marionette_client-0.8.7-py2.7.egg/marionette/marionette_test.py", line 283, in run
    testMethod()
  File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.1/tests/python/gaia-ui-tests/gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py", line 67, in test_keyboard_predictive_key
    self.assertEqual(keyboard_page.text_input, 'keyboard Tea he ')

AssertionError: u'keyboard Tea ye ' != 'keyboard Tea he '



Attaching html report of run via local automation (7/10 passing)
Flags: needinfo?(pbylenga)
Status: RESOLVED → REOPENED
QA Whiteboard: [fxosqa-auto-from-s8][fxosqa-auto-s9] → [fxosqa-auto-from-s8][fxosqa-auto-s9][QAnalyst-Triage?][fxosqa-auto-backlog?]
Resolution: FIXED → ---
QA Whiteboard: [fxosqa-auto-from-s8][fxosqa-auto-s9][QAnalyst-Triage?][fxosqa-auto-backlog?] → [fxosqa-auto-from-s8][fxosqa-auto-s9][QAnalyst-Triage+][fxosqa-auto-backlog?]
Flags: needinfo?(pbylenga)
Apparently, the fix didn't work, I'll see if I can fix it.
Assignee: viorelaioia → martijn.martijn
Attached patch predictive.diff (obsolete) — Splinter Review
I tried this, but I still got a failure once in 20 times, while running this:
REPEAT 1
-------
TEST-START | test_keyboard_predictive_key.py TestKeyboardPredictiveKey.test_keyboard_predictive_key
TEST-UNEXPECTED-ERROR | test_keyboard_predictive_key.py TestKeyboardPredictiveKey.test_keyboard_predictive_key | TimeoutException: TimeoutException: Timed out after 10.0 seconds


Traceback (most recent call last):
  File "/Users/mwargers/.virtualenvs/gaia-py-latest/lib/python2.7/site-packages/marionette_client-0.9-py2.7.egg/marionette/marionette_test.py", line 290, in run
    testMethod()
  File "/Users/mwargers/B2G/gaia_clean/tests/python/gaia-ui-tests/gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py", line 75, in test_keyboard_predictive_key
    Wait(self.marionette).until(lambda m: keyboard_page.text_input == 'keyboard Tea he wit.')
  File "/Users/mwargers/.virtualenvs/gaia-py-latest/lib/python2.7/site-packages/marionette_driver-0.2-py2.7.egg/marionette_driver/wait.py", line 143, in until
    cause=last_exc)
TEST-INFO took 103898ms

It makes me wonder if there isn't a genuine bug here somehow.
Perhaps this issue happens somehow when there is some longer text in the input? (so the corrected word is scrolled).
Comment on attachment 8572718 [details] [diff] [review]
predictive.diff

Even though this doesn't seem to fix the issue, it might still be useful to check this in. I think using Wait() is more appropriate here.
Attachment #8572718 - Flags: feedback?(jlorenzo)
I really think it's sketchy to do a test based on assumptions about how the internal dictionary will correct. Is it context sensitive to what's already been typed in the past, making this possibly non-deterministic? Plainly it can make different decisions as we progress builds, and they're not bugs.

But if we do this, I think we need to take djf's comment 3 to heart and quit trying to do this on three letter, ambiguous-in-English words. Just making it a string of multiple small words isn't going to fix that.

I think we should change this to a single, distinct English word for which there's only one reasonable completion when most of it is typed and no near-corrections. That's difficult because you need one that has no possible suffixes which might be displayed before the main word. That more or less limits it to idiosyncratic adverbs and semi-proper names that don't take regular plurals.

Try:

"edelw" --> only result should be "edelweiss"

That tests completion. If you want misspelling + completion:

"edels" --> only result should be "edelweiss"

Just misspelling:

"edelseiss" --> only result should be "edelweiss"

There are some other words that behave similarly (backstage is one I found) but "edelweiss" is the only one I found easily that becomes absolutely unique within a few keystrokes.
Should mention that I tried this manually in both orientations and it behaved as I said. 

If we pick a different word for some reason, that should be attempted manually prior to committing the test. But it really does need to be a long, complex word to make this at all robust.
Comment on attachment 8572718 [details] [diff] [review]
predictive.diff

As sometimes the word takes some time to be replaced (saw it manually), this looks like a good way to fix the intermittent issue!

Agreed on the dictionary problem.
Attachment #8572718 - Flags: feedback?(jlorenzo) → feedback+
(In reply to Johan Lorenzo [:jlorenzo] (QA) from comment #18)
> As sometimes the word takes some time to be replaced (saw it manually), this
> looks like a good way to fix the intermittent issue!

Yes, although, I still saw one time a failure with this applied, it certainly wouldn't make things worse, at least.

> Agreed on the dictionary problem.

Me too, but I think it's better to do that in a separate bug, so I filed bug 1139976 for it.
Attachment #8572718 - Attachment is obsolete: true
Attachment #8573333 - Flags: review?(jlorenzo)
Comment on attachment 8573333 [details] [review]
[gaia] mwargers:predictive > mozilla-b2g:master

No changes in the patch since feedback. I don't see any reason to not r+ it.
Attachment #8573333 - Flags: review?(jlorenzo) → review+
Attachment #8573333 - Flags: review?(npark)
Comment on attachment 8573333 [details] [review]
[gaia] mwargers:predictive > mozilla-b2g:master

Simple conversion to Wait() statement, looks good to me
Attachment #8573333 - Flags: review?(npark) → review+
(In reply to No-Jun Park [:njpark] from comment #22)
> Comment on attachment 8573333 [details] [review]
> [gaia] mwargers:predictive > mozilla-b2g:master
> 
> Simple conversion to Wait() statement, looks good to me

Merged: https://github.com/mozilla-b2g/gaia/commit/78eed3afcad407acb543682263d81b0e72ca7e69
Duplicate of this bug: 1151173
This issue can be reproduced on Flame v2.2. Thanks.

Device firmware (base)	L1TC100118D0
Device firmware (date)	28 Apr 2015 16:23:30
Device firmware (incremental)	eng.cltbld.20150428.042318
Device firmware (release)	4.4.2
Device identifier	flame
Gaia date	28 Apr 2015 04:41:33
Gaia revision	9f6b1b908266
Gecko build	20150428002500
Gecko revision	e79c19bf19bf
Gecko version	37.0


Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/B2G.v2.2.pvt.flame.gaiatest.non-smoketest/tests/python/gaia-ui-tests/.env/local/lib/python2.7/site-packages/marionette_client_mozilla_b2g37_v2_2-0.1-py2.7.egg/marionette/marionette_test.py", line 268, in run
    testMethod()
  File "/var/lib/jenkins/workspace/B2G.v2.2.pvt.flame.gaiatest.non-smoketest/tests/python/gaia-ui-tests/gaiatest/tests/functional/keyboard/test_keyboard_predictive_key.py", line 69, in test_keyboard_predictive_key
    self.assertEqual(keyboard_page.text_input, 'keyboard Tea he wit.')
AssertionError: u'keyboard Tea he wot.' != 'keyboard Tea he wit.'
Assignee: martijn.martijn → nobody
Firefox OS is not being worked on
Status: REOPENED → RESOLVED
Closed: 5 years ago2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.