987946 - [unittest] test_cold_launch.py failing

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

test_cold_launch.py is failing with an assertion error; I've checked the device manually while the test is running, and can physically see that the Clock app is launched.

STR:

On today's master/mozilla-central Hamachi build, run: gaiatest --testvars=/home/webqa/webqa-credentials/b2g/b2g-7.json --address=localhost:2828 --xml-output=results/results.xml --timeout=30000 --html-output=results/index.html --restart gaiatest/tests/unit/test_cold_launch.py

Traceback (most recent call last):
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.5-py2.7.egg/marionette/marionette_test.py", line 163, in run
testMethod()
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.adhoc/tests/python/gaia-ui-tests/gaiatest/tests/unit/test_cold_launch.py", line 13, in test_cold_launch
self.assertTrue('clock' in self.marionette.get_url())
AssertionError: False is not true

and then:

Traceback (most recent call last):
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.5-py2.7.egg/marionette/marionette_test.py", line 191, in run
self.tearDown()
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.adhoc/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 1067, in tearDown
MarionetteTestCase.tearDown(self)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.5-py2.7.egg/marionette/marionette_test.py", line 339, in tearDown
self.marionette.set_context("content")
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.5-py2.7.egg/marionette/marionette.py", line 860, in set_context
return self._send_message('setContext', 'ok', value=context)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.adhoc/.env/local/lib/python2.7/site-packages/marionette_client-0.7.5-py2.7.egg/marionette/marionette.py", line 589, in _send_message
raise MarionetteException("Please start a session")
MarionetteException: MarionetteException: Please start a session

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: logcat.txt, run on b2g-7

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

ro.build.display.id=msm7627a-eng 4.0.4 IMM76I 324 test-keys
ro.build.version.incremental=324
ro.build.version.release=4.0.4
ro.build.date=Thu Dec 19 14:04:55 CST 2013
ro.build.date.utc=1387433095

Can someone grab the Gecko + other needed versions for me from b2g-7?

Comment 3

[Bob Silverberg [:bsilverberg]]

What I can see when I run the test is that, immediately after launching the app, the value of self.marionette.get_url() is 'about:blank'. If I sleep for a second and check the value again it is 'app://clock.gaiamobile.org/index.html#alarm-panel', which is what we expect.

I don't know why this delay is occurring, and, based on the fact that the test just started failing, it seems like it's something new. I'm also not sure if this is an issue with gaia's launching code, our atom, or marionette itself.

I am attaching a patch to this bug which fixes the issue by waiting for the expected value, as opposed to just asserting that it's true immediately. I'm not sure if that's how we should address this issue, but I am presenting it as an option to at least allow the test to pass.

Comment 4

[Bob Silverberg [:bsilverberg]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/17604

Comment 5

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Bob Silverberg [:bsilverberg] from comment #3)
> What I can see when I run the test is that, immediately after launching the
> app, the value of self.marionette.get_url() is 'about:blank'. If I sleep for
> a second and check the value again it is
> 'app://clock.gaiamobile.org/index.html#alarm-panel', which is what we expect.
> 
> I don't know why this delay is occurring, and, based on the fact that the
> test just started failing, it seems like it's something new. I'm also not
> sure if this is an issue with gaia's launching code, our atom, or marionette
> itself.
> 
> I am attaching a patch to this bug which fixes the issue by waiting for the
> expected value, as opposed to just asserting that it's true immediately. I'm
> not sure if that's how we should address this issue, but I am presenting it
> as an option to at least allow the test to pass.

Ah, that means that the page isn't fully loaded yet. So when an app is launched, about:blank is the default url it is launched with, then it will load its app url. When you launch the app with Gaia, it doesn't wait for the app to be completely launched or anything, it just returns. Same with switch_to_frame in marionette; we don't check the status of the page after you switch frames, we just drop you into that frame since you're free to access the frame at any point.

You can go around this by waiting until get_url returns something other than "about:blank". You can use the Wait module for this.

Comment 6

[raymond [:retornam] (inactive)]

(In reply to Stephen Donner [:stephend] from comment #2)
> ro.build.display.id=msm7627a-eng 4.0.4 IMM76I 324 test-keys
> ro.build.version.incremental=324
> ro.build.version.release=4.0.4
> ro.build.date=Thu Dec 19 14:04:55 CST 2013
> ro.build.date.utc=1387433095
> 
> Can someone grab the Gecko + other needed versions for me from b2g-7?

Please tell me the files  you need and I'll grab them for you.

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

retornam: specifically, the output of https://github.com/Mozilla-TWQA/B2G-flash-tool/blob/master/check_versions.sh for b2g-7's build, would be great.  Thanks.

Comment 8

[Bob Silverberg [:bsilverberg]]

@mdas: That's exactly what I'm doing in the patch, which does solve the problem, but I'm wondering why this suddenly is an issue. Is this a performance regression?

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8396717 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/17604

This could be a performance issue, and may be related to bug 987994. We currently wait for the appopen event, so that must now be firing before the app URL is set. The fix here looks good to me, but we may also want to improve our launch method to only return once we're confident that the app is ready to use.

Could you update the commit message to indicate the change rather than just the summary from the bug? Otherwise, let's get this in.

Comment 10

[Bob Silverberg [:bsilverberg]]

Thanks davehunt. I have added some detail to the commit message and merged the code, which landed in https://github.com/mozilla-b2g/gaia/commit/947dab7ed9817643f004e1a3bb27cbb2a5eb0b2e

I guess this should be marked as fixed, but maybe someone wants to open a separate bug about either the performance issue, or changing our launch code (or both).

Comment 11

[Zac C (:zac)]

Fixed as per comment #10, this works well now.

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

Thanks; verified.