1045818 - Semi-auto tests which access WebAPI's before a user prompt seem racy

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Certification Suite, defect)

Description

[Jonathan Griffin (:jgriffin)]

Semi-auto tests which access a WebAPI before a user prompt seem racy and fail intermittently, at least on my Flame.

For example, the test webapi_tests.fm_radio.test_fm_radio_basic.TestFMRadioBasic.test_remove_antenna sometimes fails before the user prompt with this message:

webapi_tests.fm_radio.test_fm_radio_basic.TestFMRadioBasic.test_remove_antenna

Traceback (most recent call last):
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/fxos_certsuite-0.1-py2.7.egg/webapi_tests/fm_radio/test_fm_radio_basic.py", line 79, in test_remove_antenna
    self.setup_antenna_change_listener()
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/fxos_certsuite-0.1-py2.7.egg/webapi_tests/fm_radio/fm_radio_test.py", line 45, in setup_antenna_change_listener
    """)
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/marionette_client-0.8.1-py2.7.egg/marionette/marionette.py", line 1248, in execute_async_script
    filename=os.path.basename(frame[0]))
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/marionette_client-0.8.1-py2.7.egg/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/marionette_client-0.8.1-py2.7.egg/marionette/marionette.py", line 629, in _send_message
    self._handle_error(response)
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/marionette_client-0.8.1-py2.7.egg/marionette/marionette.py", line 677, in _handle_error
    raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
JavascriptException: JavascriptException: TypeError: fm is undefined
stacktrace:
	execute_async_script @fm_radio_test.py, line 45
	inline javascript, line 315
	src: "        fm.onantennaavailablechange = function() {"


I've seen many different tests of different API's fail in this way as well.  I haven't seen this on 1.3 on my hamachi, possibly because the tests run slower on that device?

This feels like a race condition in that we may not be waiting long enough after launching the test app before doing something in it, although I'm not clear what we should be waiting for.

Comment 1

[Robert Wood [:rwood]]

I just had this happen on 1.4 Flame, I skipped an MMS tests (by answering 'No' to the first prompt) and then the next test (SMS) failed with "TypeError: mm is undefined"

Comment 2

[Jonathan Griffin (:jgriffin)]

I experimented with this a bit; the problem disappears if adding a time.sleep(1) after launching the test app.

Of course, we would prefer something more deterministic.

All of the gaiatest tests wait for some UI element to appear after launching their respective apps, e.g., https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/apps/gallery/app.py#L22

We could either do something similar, or possibly better, add a wait statement in the tests to wait for the WebAPI object (like window.navigator.mozpower in the case of the mozpower tests) to be available, with some reasonable timeout of a couple of seconds for cases like the dongle in which some WebAPI objects will never be available.

Comment 3

[Robert Wood [:rwood]]

(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> I experimented with this a bit; the problem disappears if adding a
> time.sleep(1) after launching the test app.
> 
> Of course, we would prefer something more deterministic.
> 
> All of the gaiatest tests wait for some UI element to appear after launching
> their respective apps, e.g.,
> https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/
> gaiatest/apps/gallery/app.py#L22
> 
> We could either do something similar, or possibly better, add a wait
> statement in the tests to wait for the WebAPI object (like
> window.navigator.mozpower in the case of the mozpower tests) to be
> available, with some reasonable timeout of a couple of seconds for cases
> like the dongle in which some WebAPI objects will never be available.

Ok, I will give this a try (your second suggestion to wait for the WebAPI object)

Comment 4

[Jonathan Griffin (:jgriffin)]

Something like this seems to work ok locally:

https://github.com/jonallengriffin/fxos-certsuite/commit/48d7037941c7f1c988dfd86e92a2ac15592cc592

Comment 5

[Robert Wood [:rwood]]

I just finished some similar tests, and it worked too:

https://github.com/mozilla-b2g/fxos-certsuite/pull/178/files

Your solution is better though (adding it to testcase.py)!

Comment 6

[Jonathan Griffin (:jgriffin)]

https://github.com/mozilla-b2g/fxos-certsuite/commit/744a80f548d8bf213606d85d5eb22712e14e4b43