Semi-auto tests which access WebAPI's before a user prompt seem racy

RESOLVED FIXED

Status

Firefox OS
Certification Suite
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: jgriffin, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
Semi-auto tests which access a WebAPI before a user prompt seem racy and fail intermittently, at least on my Flame.

For example, the test webapi_tests.fm_radio.test_fm_radio_basic.TestFMRadioBasic.test_remove_antenna sometimes fails before the user prompt with this message:

webapi_tests.fm_radio.test_fm_radio_basic.TestFMRadioBasic.test_remove_antenna

Traceback (most recent call last):
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/fxos_certsuite-0.1-py2.7.egg/webapi_tests/fm_radio/test_fm_radio_basic.py", line 79, in test_remove_antenna
    self.setup_antenna_change_listener()
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/fxos_certsuite-0.1-py2.7.egg/webapi_tests/fm_radio/fm_radio_test.py", line 45, in setup_antenna_change_listener
    """)
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/marionette_client-0.8.1-py2.7.egg/marionette/marionette.py", line 1248, in execute_async_script
    filename=os.path.basename(frame[0]))
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/marionette_client-0.8.1-py2.7.egg/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/marionette_client-0.8.1-py2.7.egg/marionette/marionette.py", line 629, in _send_message
    self._handle_error(response)
  File "/home/jgriffin/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/marionette_client-0.8.1-py2.7.egg/marionette/marionette.py", line 677, in _handle_error
    raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
JavascriptException: JavascriptException: TypeError: fm is undefined
stacktrace:
	execute_async_script @fm_radio_test.py, line 45
	inline javascript, line 315
	src: "        fm.onantennaavailablechange = function() {"


I've seen many different tests of different API's fail in this way as well.  I haven't seen this on 1.3 on my hamachi, possibly because the tests run slower on that device?

This feels like a race condition in that we may not be waiting long enough after launching the test app before doing something in it, although I'm not clear what we should be waiting for.

Comment 1

4 years ago
I just had this happen on 1.4 Flame, I skipped an MMS tests (by answering 'No' to the first prompt) and then the next test (SMS) failed with "TypeError: mm is undefined"
(Reporter)

Comment 2

4 years ago
I experimented with this a bit; the problem disappears if adding a time.sleep(1) after launching the test app.

Of course, we would prefer something more deterministic.

All of the gaiatest tests wait for some UI element to appear after launching their respective apps, e.g., https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/apps/gallery/app.py#L22

We could either do something similar, or possibly better, add a wait statement in the tests to wait for the WebAPI object (like window.navigator.mozpower in the case of the mozpower tests) to be available, with some reasonable timeout of a couple of seconds for cases like the dongle in which some WebAPI objects will never be available.

Comment 3

4 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> I experimented with this a bit; the problem disappears if adding a
> time.sleep(1) after launching the test app.
> 
> Of course, we would prefer something more deterministic.
> 
> All of the gaiatest tests wait for some UI element to appear after launching
> their respective apps, e.g.,
> https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/
> gaiatest/apps/gallery/app.py#L22
> 
> We could either do something similar, or possibly better, add a wait
> statement in the tests to wait for the WebAPI object (like
> window.navigator.mozpower in the case of the mozpower tests) to be
> available, with some reasonable timeout of a couple of seconds for cases
> like the dongle in which some WebAPI objects will never be available.

Ok, I will give this a try (your second suggestion to wait for the WebAPI object)

Comment 5

4 years ago
I just finished some similar tests, and it worked too:

https://github.com/mozilla-b2g/fxos-certsuite/pull/178/files

Your solution is better though (adding it to testcase.py)!
(Reporter)

Comment 6

4 years ago
https://github.com/mozilla-b2g/fxos-certsuite/commit/744a80f548d8bf213606d85d5eb22712e14e4b43
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.