755127 - mozmill python tests broken on HEAD

Status: RESOLVED FIXED

(Testing Graveyard :: Mozmill, defect)

Description

[Jeff Hammel]

Now that the python tests actually take the binary, running them is giving me lots of errors: 

FAILED (failures=4, errors=1)
INFO | Some tests were unsuccessful.
INFO | 
INFO | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
INFO | 
INFO | Python Failures:
INFO | 
INFO | (<testapi.TestMozmillAPI testMethod=test_api>, 'Traceback (most recent call last):\n  File "/home/jhammel/mozmill/src/mozmill/mutt/mutt/tests/python/testapi.py", line 24, in test_api\n    self.assertTrue(len(results.passes) == passes)\nAssertionError: False is not true\n')
INFO | 
INFO | (<testpersisted.TestMozmillPersisted testMethod=test_persisted>, 'Traceback (most recent call last):\n  File "/home/jhammel/mozmill/src/mozmill/mutt/mutt/tests/python/testpersisted.py", line 38, in test_persisted\n    self.assertTrue(len(results.passes) == 1)\nAssertionError: False is not true\n')
INFO | 
INFO | (<testpersisted.TestMozmillPersisted testMethod=test_persisted_shutdown>, 'Traceback (most recent call last):\n  File "/home/jhammel/mozmill/src/mozmill/mutt/mutt/tests/python/testpersisted.py", line 48, in test_persisted_shutdown\n    self.assertTrue(len(results.passes) == 1)\nAssertionError: False is not true\n')
INFO | 
INFO | (<testmultiplerun.TestMozmillAPI testMethod=test_runtwice>, 'Traceback (most recent call last):\n  File "/home/jhammel/mozmill/src/mozmill/mutt/mutt/tests/python/testmultiplerun.py", line 24, in test_runtwice\n    results = m.run(dict(path=path))\n  File "/home/jhammel/mozmill/src/mozmill/mozmill/mozmill/__init__.py", line 352, in run\n    self.run_tests(*tests)\n  File "/home/jhammel/mozmill/src/mozmill/mozmill/mozmill/__init__.py", line 319, in run_tests\n    frame = self.start_runner()\n  File "/home/jhammel/mozmill/src/mozmill/mozmill/mozmill/__init__.py", line 251, in start_runner\n    self.runner.start()\n  File "/home/jhammel/mozmill/src/mozbase/mozrunner/mozrunner/runner.py", line 122, in start\n    self.profile.reset()\n  File "/home/jhammel/mozmill/src/mozbase/mozprofile/mozprofile/profile.py", line 98, in reset\n    proxy = self._proxy)\n  File "/home/jhammel/mozmill/src/mozbase/mozprofile/mozprofile/profile.py", line 78, in __init__\n    self.addon_manager.install_addons(addons, addon_manifests)\n  File "/home/jhammel/mozmill/src/mozbase/mozprofile/mozprofile/addons.py", line 49, in install_addons\n    self.install_from_path(addon)\n  File "/home/jhammel/mozmill/src/mozbase/mozprofile/mozprofile/addons.py", line 177, in install_from_path\n    assert os.path.isdir(path), "Addon \'%s\' cannot be installed" % path\nAssertionError: Addon \'/tmp/tmpqPD6Yg.mozrunner/extensions/[e-mail]\' cannot be installed\n')
INFO | 
INFO | (<test_bug690154.TestBug690154 testMethod=test_JSON_structure>, 'Traceback (most recent call last):\n  File "/home/jhammel/mozmill/src/mozmill/mutt/mutt/tests/python/test_bug690154.py", line 34, in test_JSON_structure\n    self.assertTrue(fails[\'name\'] == \'<TOP_LEVEL>\')\nKeyError: \'name\'\n')
INFO | 
INFO | ===========================================================================
INFO | 
INFO | No Javascript Failures
INFO | 

FWIW, I am getting massive failures on the JS tests too (not shown here), but one thing at at time

This could be https://bugzilla.mozilla.org/show_bug.cgi?id=696468 but I'm not sure yet.  Will dig in as I am able

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2] (away July 24th - August 16th)]

Just to be curious, why are failures on Python shown via the INFO logger level? Those should be ERROR.

Comment 2

[Jeff Hammel]

The output is from the summary at the end, as the entire output is very long.  I didn't design mutt, though I assume the rationale is that the final summary should be printed at the INFO level.  As to whether this is "correct" or not, I don't really have a strong opinion and consider the issue orthogonal to the failures themselves

Comment 3

[Jeff Hammel]

Attachment: sample manifest

Comment 4

[Jeff Hammel]

So I've been playing with a reduced manifest as given in attachment 624221 [details] to try to debug these one by one.  The trick is....the failures are related.  If I run usemozmill.py or testapi.py, they pass.  Albeit, I get 

INFO | Timeout: bridge.execFunction("39e0c3b2-9ede-11e1-a11c-0024d7065b60", bridge.registry["{6d856535-6cdc-4ced-ad6d-145ca11cacc7}"]["cleanQuit"], [])

for these and any other test I run o_O.  This is probably the actual problem.  However, if I run testapi.py following usemozmill.py, as given in the manifest, I get (output truncated):

INFO | Timeout: bridge.execFunction("13518506-9ede-11e1-a11c-0024d7065b60", bridge.registry["{8d5da637-544d-416a-b8ef-e7e7a84136f4}"]["cleanQuit"], [])
INFO | 
INFO | Passed: 1
INFO | Failed: 0
INFO | Skipped: 0
.INFO | Timeout: bridge.set("38532af8-9ede-11e1-a11c-0024d7065b60", Components.utils.import('resource://mozmill/driver/mozmill.js'))
INFO | 
INFO | Timeout: bridge.execFunction("39e0c3b2-9ede-11e1-a11c-0024d7065b60", bridge.registry["{6d856535-6cdc-4ced-ad6d-145ca11cacc7}"]["cleanQuit"], [])
INFO | 
F
======================================================================
FAIL: test_api (testapi.TestMozmillAPI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jhammel/mozmill/src/mozmill/mutt/mutt/tests/python/testapi.py", line 24, in test_api
    self.assertTrue(len(results.passes) == passes)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 2 tests in 137.264s

FAILED (failures=1)
INFO | Running JS Tests
INFO | 
INFO | Some tests were unsuccessful.
INFO | 
INFO | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
INFO | 
INFO | Python Failures:
INFO | 
INFO | (<testapi.TestMozmillAPI testMethod=test_api>, 'Traceback (most recent call last):\n  File "/home/jhammel/mozmill/src/mozmill/mutt/mutt/tests/python/testapi.py", line 24, in test_api\n    self.assertTrue(len(results.passes) == passes)\nAssertionError: False is not true\n')
INFO | 

Note the timeout in loading resource://mozmill/driver/mozmill.js . Seems common sense that if we can't load this file that horrible things will happen.

I'm guessing that this is due to the timeout on cleanQuit in the previous test.  To date, I have no idea what causes it.  To repeat, running testapi.py in isolation does pass, albeit with the same timeout on cleanQuit

Comment 5

[Jeff Hammel]

:ctalbert, do you get similar? Or anyone else running linux?

Comment 6

[Jeff Hammel]

On a computer where sample.ini does run correctly (see comments in bug 696468) the only failure I see is:

======================================================================
FAIL: test_runtwice (testmultiplerun.TestMozmillAPI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jhammel/mozmill/src/mozmill/mutt/mutt/tests/python/testmultiplerun.py", line 24, in test_runtwice
    results = m.run(dict(path=path))
  File "/home/jhammel/mozmill/src/mozmill/mozmill/mozmill/__init__.py", line 352, in run
    self.run_tests(*tests)
  File "/home/jhammel/mozmill/src/mozmill/mozmill/mozmill/__init__.py", line 319, in run_tests
    frame = self.start_runner()
  File "/home/jhammel/mozmill/src/mozmill/mozmill/mozmill/__init__.py", line 251, in start_runner
    self.runner.start()
  File "/home/jhammel/mozmill/src/mozbase/mozrunner/mozrunner/runner.py", line 162, in start
    self.profile.reset()
  File "/home/jhammel/mozmill/src/mozbase/mozprofile/mozprofile/profile.py", line 98, in reset
    proxy = self._proxy)
  File "/home/jhammel/mozmill/src/mozbase/mozprofile/mozprofile/profile.py", line 78, in __init__
    self.addon_manager.install_addons(addons, addon_manifests)
  File "/home/jhammel/mozmill/src/mozbase/mozprofile/mozprofile/addons.py", line 49, in install_addons
    self.install_from_path(addon)
  File "/home/jhammel/mozmill/src/mozbase/mozprofile/mozprofile/addons.py", line 177, in install_from_path
    assert os.path.isdir(path), "Addon '%s' cannot be installed" % path
AssertionError: Addon '/tmp/tmpzGIyYb.mozrunner/extensions/[e-mail]' cannot be installed

----

I recall having seen a bug to this effect.  Will look into it

Comment 7

[Jeff Hammel]

Since Bug 753190, things seem to be working.  I see an intermittent failure in expectstacktest.py but cannot reliably reproduce.  Will close this and subsequent issues can be ticketed subsequently

Comment 8

[Jeff Hammel]

this also doesn't account for bug 696468 to be tracked there