948193 - Gaia UI Tests - Wi-Fi unittest is failing since Dec. 6th, 2:41 pm PDT

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Wifi, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Our Wi-Fi unittest is failing, like so:

Traceback (most recent call last):
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.wifi/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette_test.py", line 143, in run
testMethod()
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.wifi/tests/python/gaia-ui-tests/gaiatest/tests/unit/settings/test_wifi_settings.py", line 11, in test_set_wifi
self.data_layer.enable_wifi()
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.wifi/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 294, in enable_wifi
result = self.marionette.execute_async_script("return GaiaDataLayer.enableWiFi()", special_powers=True)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.wifi/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 1073, in execute_async_script
filename=os.path.basename(frame[0]))
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.wifi/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 577, in _send_message
self._handle_error(response)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests.wifi/.env/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette.py", line 626, in _handle_error
raise ScriptTimeoutException(message=message, status=status, stacktrace=stacktrace)
ScriptTimeoutException: timed out

:vchang, any ideas here?  Thanks!

The function lives here: https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/atoms/gaia_data_layer.js#L257

Regression range is:

Failed > Console Output  #1030 	Dec 6, 2013 2:41:53 PM	 
Success > Console Output  #102  Dec 6, 2013 11:55:01 AM

http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Hamachi/job/b2g.hamachi.mozilla-central.unittests.wifi/1039/

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

https://pastebin.mozilla.org/3745078 is the logcat, Vincent/all; hope that helps!

Comment 2

[Vincent Chang[:vchang][changyihsin]]

I am going to check this soon.

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

FWIW, this is also affected the *.perf.fps tests too:

http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Hamachi/job/b2g.hamachi.mozilla-central.master.perf.fps/

http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Hamachi/job/b2g.hamachi.mozilla-central.master.perf.fps/852/console

Appears to be the same regression range.

Comment 4

[Chuck Lee [:chucklee, Triggered by NI, Review, and Feedback requests]]

I got "JavascriptException: Error: Expected IccHelper to be enabled." and can't run the test at all.
Did you encounter this?



TEST-START test_wifi_settings.py
test_set_wifi (test_wifi_settings.TestWiFiSettings) ... ERROR

======================================================================
ERROR: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chucklee/work/github/gaia/tests/python/gaia-ui-tests/env/local/lib/python2.7/site-packages/marionette_client-0.6.1-py2.7.egg/marionette/marionette_test.py", line 127, in run
    self.setUp()
  File "/home/chucklee/work/github/gaia/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 769, in setUp
    self.device.start_b2g()
  File "/home/chucklee/work/github/gaia/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 722, in start_b2g
    });""", script_timeout=60000)
  File "/home/chucklee/work/github/gaia/tests/python/gaia-ui-tests/env/local/lib/python2.7/site-packages/marionette_client-0.6.1-py2.7.egg/marionette/marionette.py", line 1073, in execute_async_script
    filename=os.path.basename(frame[0]))
  File "/home/chucklee/work/github/gaia/tests/python/gaia-ui-tests/env/local/lib/python2.7/site-packages/marionette_client-0.6.1-py2.7.egg/marionette/marionette.py", line 577, in _send_message
    self._handle_error(response)
  File "/home/chucklee/work/github/gaia/tests/python/gaia-ui-tests/env/local/lib/python2.7/site-packages/marionette_client-0.6.1-py2.7.egg/marionette/marionette.py", line 612, in _handle_error
    raise JavascriptException(message=message, status=status, stacktrace=stacktrace)
JavascriptException: Error: Expected IccHelper to be enabled.
	stacktrace:
	@app://system.gaiamobile.org/shared/js/operator_variant_helper.js, line 43
TEST-UNEXPECTED-FAIL | test_wifi_settings.py test_wifi_settings.TestWiFiSettings.test_set_wifi | 
----------------------------------------------------------------------
Ran 1 test in 25.181s

FAILED (errors=1)

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

The above stack isn't what we're seeing on:

  <!-- Mercurial-Information: <project name="mozilla-central" path="gecko" remote="hgmozillaorg" revision="9f12a9fab080"/> -->
  <project name="gecko.git" path="gecko" remote="mozillaorg" revision="a0bb585098cc89c454fac0297b5ef748d5cab82c"/>

Comment 6

[Zac C (:zac)]

Chuck are you running without a SIM in the device?

Comment 7

[Chuck Lee [:chucklee, Triggered by NI, Review, and Feedback requests]]

It happened with or without SIM in device.
But now I borrow another buri, and it doesn't happen on this one.

Comment 8

[Mike Pennisi [:jugglinmike]]

This is odd. This test fails for me in two different ways (both different than the reported failure).

When run with all the Settings application tests, it fails with the follow error (5 other tests fail as well):


TEST-START test_settings_wifi.py
test_connect_to_wifi_via_settings_app (test_settings_wifi.TestSettingsWifi)
Connect to a wifi network via the Settings app ... ERROR

======================================================================
ERROR: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mike/.virtualenvs/gaia-ui/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette_test.py", line 143, in run
    testMethod()
  File "/home/mike/projects/mozilla/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/settings/test_settings_wifi.py", line 26, in test_connect_to_wifi_via_settings_app
    wifi_settings.connect_to_network(self.testvars['wifi'])
TEST-UNEXPECTED-FAIL | test_settings_wifi.py test_settings_wifi.TestSettingsWifi.test_connect_to_wifi_via_settings_app | KeyError: 'wifi'
----------------------------------------------------------------------
Ran 1 test in 45.893s

FAILED (errors=1)
TEST-START test_settings_power_save_mode.py
test_power_save_mode (test_settings_power_save_mode.TestPowerSaveMode) ... ERROR



When run in isolation:



TEST-START test_settings_wifi.py
test_connect_to_wifi_via_settings_app (test_settings_wifi.TestSettingsWifi)
Connect to a wifi network via the Settings app ... ERROR

======================================================================
ERROR: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mike/.virtualenvs/gaia-ui/local/lib/python2.7/site-packages/marionette_client-0.6.2-py2.7.egg/marionette/marionette_test.py", line 143, in run
    testMethod()
  File "/home/mike/projects/mozilla/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/settings/test_settings_wifi.py", line 26, in test_connect_to_wifi_via_settings_app
    wifi_settings.connect_to_network(self.testvars['wifi'])
TEST-UNEXPECTED-FAIL | test_settings_wifi.py test_settings_wifi.TestSettingsWifi.test_connect_to_wifi_via_settings_app | KeyError: 'wifi'
----------------------------------------------------------------------
Ran 1 test in 45.843s

FAILED (errors=1)

Comment 9

[Mike Pennisi [:jugglinmike]]

Ignore Comment 8--I was not running the tests with the correct `--testvars`.

Unfortunately, my devices cannot find any wireless networks, even when manually navigating to the Settings application and enabling Wifi.

Comment 10

[Chuck Lee [:chucklee, Triggered by NI, Review, and Feedback requests]]

(In reply to Stephen Donner [:stephend] from comment #5)
> The above stack isn't what we're seeing on:
> 
>   <!-- Mercurial-Information: <project name="mozilla-central" path="gecko"
> remote="hgmozillaorg" revision="9f12a9fab080"/> -->
>   <project name="gecko.git" path="gecko" remote="mozillaorg"
> revision="a0bb585098cc89c454fac0297b5ef748d5cab82c"/>

Note: this is built on 2013-12-09-05-34-02.

Comment 11

[Chuck Lee [:chucklee, Triggered by NI, Review, and Feedback requests]]

I updated gaia and gecko to this version, and run gaia ui test by command:
gaiatest --address=localhost:2828 --testvars=testvars.json gaiatest/tests/unit/settings/test_wifi_settings.py --repeat=10 --restart

Nothing goes wrong.

The wifi settings in my testvars.json points to an AP with PSK encryption, while APN settings are left empty.
I don't think these settings affects the test result though.

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Chuck Lee [:chucklee] from comment #11)
> I updated gaia and gecko to this version, and run gaia ui test by command:
> gaiatest --address=localhost:2828 --testvars=testvars.json
> gaiatest/tests/unit/settings/test_wifi_settings.py --repeat=10 --restart
> 
> Nothing goes wrong.
> 
> The wifi settings in my testvars.json points to an AP with PSK encryption,
> while APN settings are left empty.
> I don't think these settings affects the test result though.

Are you on the Hamachi 1.2...11-05 base build (vendor version?)

Is there anything about our config or in our logcat that points to something we can try to address?

Comment 13

[Henry Chang [:hchang]]

I can reproduce this bug and it is just because wifi took too much time to get disabled. 

Changing

https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/gaia_test.py#L323

to

result = self.marionette.execute_async_script("return GaiaDataLayer.disableWiFi()", special_powers=True, script_timeout=20000) 

solves this issue.

Comment 14

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Henry Chang [:henry] from comment #13)
> I can reproduce this bug and it is just because wifi took too much time to
> get disabled. 
> 
> Changing
> 
> https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/
> gaiatest/gaia_test.py#L323
> 
> to
> 
> result = self.marionette.execute_async_script("return
> GaiaDataLayer.disableWiFi()", special_powers=True, script_timeout=20000) 
> 
> solves this issue.

Would you mind please doing a GitHub pull request with those changes, so we can discuss/review/potentially merge?  Thanks!!

Comment 15

[Henry Chang [:hchang]]

https://github.com/mozilla-b2g/gaia/pull/14555

Are you able to try to reproduce this issue with this PR?
I personally couldn't reproduce it after applying this patch.

Thanks.

Comment 16

[Stephen Donner [:stephend] Not actively reading bugmail]

So, with Dave's switch to use the non-commercial-RIL builds in our Jenkins, as well as some tweaks such as sleep()'ing after "adb stop b2g", etc. we're no longer seeing this issue -- tests have been running fine all day today.  (See screenshot of Jenkins.)

Comment 17

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Screen Shot 2013-12-16 at 4.48.36 PM.png

Comment 18

[Stephen Donner [:stephend] Not actively reading bugmail]

Marking WFM, as, other than the Jenkins config-change (which /technically/ isn't a code commit), we don't have a corresponding Gaia/Gecko/Gonk commit, and it does seem to clearly have fixed the issue!

Thanks for your investigation and patience, all!