1157772 - test_rocketbar_offline_behavior.py: "TimeoutException: Timed out after 29.9 seconds with message: Keyboard inputWindow not interpreted as displayed. Debug is_displayed(): False, class: inputWindow."

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Oliver Nelson [:oliverthor]]

Description:
test_rocketbar_offline_behavior appears to not be spawning the keyboard when attempting an offline search.

Repro Steps:
1) Update phone to 20150423010203
2) Enable Wi-Fi & Data
3) Enable Airplane mode (lose network connection)
4) Tap into Rocketbar Search bar: search for 'test'
5) Observe UI

Manual:
Rocketbar displays 'No Internet Connection' label; keyboard prompts and lets user type

Automated: 
No keyboard displayed, 

http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc/805/HTML_Report/

http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2/176/HTML_Report/

Traceback (most recent call last):
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2/.env/local/lib/python2.7/site-packages/marionette_client-0.11-py2.7.egg/marionette/marionette_test.py", line 296, in run
testMethod()
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2/tests/python/gaia-ui-tests/gaiatest/tests/functional/rocketbar/test_rocketbar_offline_behavior.py", line 28, in test_rocketbar_offline_behavior
search_panel.type_into_search_box(test_string)
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2/tests/python/gaia-ui-tests/gaiatest/apps/homescreen/regions/search_panel.py", line 43, in type_into_search_box
self.keyboard.send(search_term)
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2/tests/python/gaia-ui-tests/gaiatest/apps/keyboard/app.py", line 232, in send
self.switch_to_keyboard()
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2/tests/python/gaia-ui-tests/gaiatest/apps/keyboard/app.py", line 167, in switch_to_keyboard
% (input_window.is_displayed(), input_window.get_attribute('class')))
File "/var/jenkins/1/workspace/flame-kk-319.mozilla-central.nightly.ui.functional.non-smoke.2/.env/local/lib/python2.7/site-packages/marionette_driver-0.4-py2.7.egg/marionette_driver/wait.py", line 143, in until
cause=last_exc)
TimeoutException: TimeoutException: Timed out after 29.9 seconds with message: Keyboard inputWindow not interpreted as displayed. Debug is_displayed(): False, class: inputWindow.


Environmental Variables:
Device firmware (base) 	L1TC100118D0
Device firmware (date) 	23 Apr 2015 08:29:45
Device firmware (inc-remental) 	eng.cltbld.20150423.042933
Device firmware (release) 	4.4.2
Device identifier 	flame
Device memory 	219772 kB
Device serial 	e472d8d7
Device uptime 	0 days 0 hours 2 minutes 13 seconds
Gaia date 	22 Apr 2015 17:32:36
Gaia revision 	9d4f756aa35c
Gecko build 	20150423010203
Gecko revision 	0b202671c9e2
Gecko version 	40.0a1

Reproducing in automation: 2/6 passing

Reproducible manually: No
Repro frequency: 5/5 passing

Comment 1

[Martijn Wargers (dead)]

This is probably related to the workaround that was added in bug 1020974.

Comment 2

[Martijn Wargers (dead)]

We also should make sure that the keyboard is fully visible here.

Comment 3

[GitHub Autolander Bot]

Attachment: [gaia] mwargers:1157772 > mozilla-b2g:master

Comment 4

[Martijn Wargers (dead)]

Comment on attachment 8597335 [details] [review]
[gaia] mwargers:1157772 > mozilla-b2g:master

Oliver, if you could try this out and see if it fixes the intermittent failure, that would be great.
I think this will fix it. This code is horrible, but it is still necessary as bug 1020974 is still around.

Comment 5

[Oliver Nelson [:oliverthor]]

Comment on attachment 8597335 [details] [review]
[gaia] mwargers:1157772 > mozilla-b2g:master

So I adhoced this behavior via Jenkins and saw a 10/10 passing rate, so it's hard to see where this fix may ensure an intermittent failure doesn't occur.

http://jenkins1.qa.scl3.mozilla.com/view/UI/job/flame-kk.ui.adhoc/810/HTML_Report/

I also performed the test locally on a machine with and without your fix and observed 10/10 passing on both.
Not sure if something else already fixed this, or the repro rate lowered from some other cause. If there's anything else I should test around, don't hesitate to ask.

Comment 6

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8597335 [details] [review]
[gaia] mwargers:1157772 > mozilla-b2g:master

I don't minus the patch, because I don't know if this fix is worth it. I'd like to know what Geo and John think about it.

IMO, I'd prefer to have this test intermittent, rather than adding 4 more seconds to each execution of the test. The value is too magic and I think waiting 5 seconds can hide some performance issues.

Comment 7

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

(In reply to Johan Lorenzo [:jlorenzo] (QA) from comment #6)
> Comment on attachment 8597335 [details] [review]
> [gaia] mwargers:1157772 > mozilla-b2g:master
> 
> I don't minus the patch, because I don't know if this fix is worth it. I'd
> like to know what Geo and John think about it.
> 
> IMO, I'd prefer to have this test intermittent, rather than adding 4 more
> seconds to each execution of the test. The value is too magic and I think
> waiting 5 seconds can hide some performance issues.

OK, thoughts:

* I personally don't care about hiding performance and don't think that should be considered. These are functional tests, and should by and large adapt to the responsiveness of the system under test. For proper isolation, use performance timing tests to ensure performance.

* Waiting 5 seconds during one test isn't a big deal. If we were doing this in some function being used by global test startup code or something I'd care a lot more.

* That said, it sure is a magic number. At the very least I'd like some documentation in the way of a comment as to why this is being done. It should read something like "A small amount of time is required for .... to initialize" or something--don't follow the "workaround for bug foo" example, that's meaningless. has to be relevant to the operation at hand.

* Being a magic sleep, this won't be resistant to slow builds like debug builds and emulator. It would possibly be better if there was a Wait done here instead, even if it's a wait whose polling function repeatedly taps until the keyboard actually appears (keyboard appearing is the wait success condition).

Of course, at that point the test no longer tests the Rocketbar appearing after an immediate tap. But that's true of the sleep as well. Best bet is to actually figure out -why- the keyboard is nonresponsive and put a wait on that if appropriate.

* Man, not Martijn's fault, but tap_search_bar() is no longer a good name for that function. It's display_search_panel() now. A tap function should look for the element, tap, then exit. Once this kind of logic is in there, it's a "ensure something happens" function, not a "do an input" function and we should have named it after the thing that you're returning/ensuring.

TL;DR/Summary:

You should dig further and see if you can do a wait on it, though if you told me this was our only route I'd probably r+ it. And Oliver can better look to see if this improves things by going to something more like 50 tries. 10 tries isn't always enough. That said, if we're talking once in 50 tries or something for the original, I agree with Johan: maybe just leave it be.

Comment 8

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

I'll wait for a reply as to whether this should be dropped or investigated further before r+'ing.

Comment 9

[Martijn Wargers (dead)]

Comment on attachment 8597335 [details] [review]
[gaia] mwargers:1157772 > mozilla-b2g:master

Ok, I'm now using Wait calls to fix the intermittent failure.
This similar thing was used here:
http://mxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/apps/homescreen/regions/search_panel.py#53

I wasn't able to remove the last remaining time.sleep call, but I think this is an improvement of what we have.

Comment 10

[Johan Lorenzo [:jlorenzo]]

Looks great! I started [1] to weigh up the effects of your patch.

[1] http://jenkins1.qa.scl3.mozilla.com/job/flame-kk.ui.adhoc/817/

Comment 11

[Johan Lorenzo [:jlorenzo]]

It seems like we have a Marionette error. This might be caused by the upgrade that happened a couple of days ago[1]. Could you see if a rebase of your PR helps to fix the error we see in Jenkins,Martijn?

[1] https://github.com/mozilla-b2g/gaia/commit/277ff361005fc37b764e01e2d32fec072a0189af

Comment 12

[GitHub Autolander Bot]

Attachment: [gaia] mwargers:1157772_v2 > mozilla-b2g:master

Comment 13

[Martijn Wargers (dead)]

Comment on attachment 8600894 [details] [review]
[gaia] mwargers:1157772_v2 > mozilla-b2g:master

Rebasing always causing trouble here, so I just created a new branch, applied the patch and pushed that.

Comment 14

[Johan Lorenzo [:jlorenzo]]

New adhoc job: http://jenkins1.qa.scl3.mozilla.com/job/flame-kk.ui.adhoc/819

Comment 15

[Martijn Wargers (dead)]

http://jenkins1.qa.scl3.mozilla.com/job/flame-kk.ui.adhoc/820/

Comment 16

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8600894 [details] [review]
[gaia] mwargers:1157772_v2 > mozilla-b2g:master

Not a single failure in 100 tries. This patch looks like a fix! Worst case scenario, we remove a time.sleep(), which is awesome. r+!

Comment 17

[No-Jun Park [:njpark]]

Comment on attachment 8600894 [details] [review]
[gaia] mwargers:1157772_v2 > mozilla-b2g:master

Fix looks good to me.

Comment 18

[Martijn Wargers (dead)]

https://github.com/mozilla-b2g/gaia/commit/e1773d6d1014c997be4b5c4233bba3ee073b8d7b