997552 - [unittest] Investigate and/or fix test_cleanup_gaia

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

On master, test_cleanup_gaia (unittest) is failing pretty regularly.

Gaia      3d4b4b06475d2376bad23ac46da185cd48a68d17
Gecko     https://hg.mozilla.org/mozilla-central/rev/dd50745d7f35
BuildID   20140416040204
Version   31.0a1
ro.build.version.incremental=324
ro.build.date=Thu Dec 19 14:04:55 CST 2013

19:27:21 TEST-START test_cleanup_gaia.py

19:28:24 test_cleanup_gaia (test_cleanup_gaia.TestCleanupGaia) ... FAIL

19:28:24 
19:28:24 ======================================================================
19:28:24 FAIL: None
19:28:24 ----------------------------------------------------------------------
19:28:24 Traceback (most recent call last):
19:28:24   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests/.env/local/lib/python2.7/site-packages/marionette_client-0.7.5-py2.7.egg/marionette/marionette_test.py", line 163, in run
19:28:24     testMethod()
19:28:24   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests/tests/python/gaia-ui-tests/gaiatest/tests/unit/test_cleanup_gaia.py", line 15, in test_cleanup_gaia
19:28:24     self.check_initial_state()
19:28:24   File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests/tests/python/gaia-ui-tests/gaiatest/tests/unit/test_cleanup_gaia.py", line 54, in check_initial_state
19:28:24     self.assertEqual(len(self.data_layer.known_networks), 0)
19:28:24 TEST-UNEXPECTED-FAIL | test_cleanup_gaia.py test_cleanup_gaia.TestCleanupGaia.test_cleanup_gaia | AssertionError: 1 != 0
19:28:24 ----------------------------------------------------------------------
19:28:24 Ran 1 test in 62.773s

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: logcat.txt

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

No-Jun, is this something you can look into, perhaps?

Comment 3

[No-Jun Park [:njpark]]

When locally run, this causes the homescreen to freeze (bug 991262) consistently.  This is with time.sleep(10) in line 878, so there may be something different going on here.  The test hangs when the test is executed with --restart flag, like:

gaiatest --address=localhost:2828 --testvars=gaiatest/b2g-7.json --restart --type=b2g gaiatest/tests/unit/test_cleanup_gaia.py

When --restart flag is not used, the test executes normally.

The test hangs even before entering the test, so something is happening at the setup() i assume.

I'll link the homescreen bug.

Comment 4

[Zac C (:zac)]

NJ, can we close this now?

It seemed to have been related to the startup regression issues we had around this time.

Comment 5

[No-Jun Park [:njpark]]

(In reply to Zac C (:zac) from comment #4)
> NJ, can we close this now?
> 
> It seemed to have been related to the startup regression issues we had
> around this time.

Yes I think it's gone for good.  I'll mark this one as resolved.

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

Reopening; still showing up in http://selenium.qa.mtv2.mozilla.com:8080/job/b2g.hamachi.mozilla-central.unittests/535/console, at least -- sorry!

Comment 7

[No-Jun Park [:njpark]]

But in this case, it's due to the wifi connection issue:

File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests/tests/python/gaia-ui-tests/gaiatest/tests/unit/test_cleanup_gaia.py", line 24, in test_cleanup_gaia
    self.data_layer.connect_to_wifi()
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 273, in connect_to_wifi
    script_timeout = max(self.marionette.timeout, 60000))

Aren't we tracking this separately?

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to No-Jun Park [:njpark] from comment #7)
> But in this case, it's due to the wifi connection issue:
> 
> File
> "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests/tests/python/
> gaia-ui-tests/gaiatest/tests/unit/test_cleanup_gaia.py", line 24, in
> test_cleanup_gaia
>     self.data_layer.connect_to_wifi()
>   File
> "/var/jenkins/workspace/b2g.hamachi.mozilla-central.unittests/tests/python/
> gaia-ui-tests/gaiatest/gaia_test.py", line 273, in connect_to_wifi
>     script_timeout = max(self.marionette.timeout, 60000))
> 
> Aren't we tracking this separately?

Perhaps?  I say that tentatively since I'm not sure if the functions called are the same.  Is that something you can investigate, No-Jun?

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Taipei is actually also seeing this too, at times:

http://release1-qa.corp.tpe1.mozilla.com:8080/view/B2G.Test.Hamachi/job/B2G.master.pvt.hamachi.gaiatest.unit/295/testReport/junit/test_cleanup_gaia/TestCleanupGaia/test_cleanup_gaia/

Eric, can you help investigate?  The failure is in connect_to_wifi(), just like ours; Dimi, can you look into this test, run multiple times, there?  Since your CI can reproduce it even sometimes, there, it might be easier to find a potential problem than over in bug 987760.

Comment 10

[Eric Chang [:ericcc] [:echang]]

Initial testing, looks okay, let me dig more.
http://release1-qa.corp.tpe1.mozilla.com:8080/job/B2G.master.pvt.hamachi.gaiatest.unit.cleanup.gaia/1/

Comment 11

[Eric Chang [:ericcc] [:echang]]

Fire new tests on jenkins and local machine.

Running test_cleanup_gaia.py with --restart or --repeat are okay.
Running gaiatest/tests/unit has more chances to hit errors in my setup.

Comment 12

[Eric Chang [:ericcc] [:echang]]

But all from test_cell_data_settings.TestCellDataSettings.
(In reply to Eric Chang [:ericcc] [:echang] from comment #11)
> Running gaiatest/tests/unit has more chances to hit errors in my setup.

Comment 13

[Eric Chang [:ericcc] [:echang]]

http://release1-qa.corp.tpe1.mozilla.com:8080/job/B2G.master.pvt.hamachi.gaiatest.unit.cleanup.gaia/4/console

Comment 14

[Eric Chang [:ericcc] [:echang]]

Works okay on Jenkins

Comment 15

[Florin Strugariu [:Bebe]]

Stephen is this still reproducing on master unit tests?

Comment 16

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Florin Strugariu [:Bebe] from comment #15)
> Stephen is this still reproducing on master unit tests?

Not sure why you need me, specifically, to answer that :-)

But, yes, it still fails, however due to connect_to_wifi(), which we're having problems with in the lab, so I think that's a separate issue:

http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Hamachi/job/b2g.hamachi.mozilla-central.unittests/769/testReport/test_cleanup_gaia/history/

I think because we still have clear, open Wi-Fi bugs in B2G itself, as well as known lab issues, we can close it.