980959 - Jetpack test runner doesn't handle assertions correctly

Status: RESOLVED DUPLICATE of bug 873204

(Add-on SDK Graveyard :: General, defect)

Description

[Ed Morley [:emorley]]

eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=35792361&tree=Mozilla-Inbound

{
TEST-PASS | tests/test-tabs.testTabPropertiesInNewWindow | favicon of the new tab is not empty
TEST-PASS | tests/test-tabs.testTabPropertiesInNewWindow | style of the new tab matches
TEST-PASS | tests/test-tabs.testTabPropertiesInNewWindow | index of the new tab matches
Assertion failure: mIsMapped, at c:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/gfx/2d/SourceSurfaceD2DTarget.cpp:283
TEST-UNEXPECTED-FAIL | tests/test-tabs.testTabPropertiesInNewWindow | Test output exceeded timeout (300s).
Error Cleaning up: [Error 5] Access is denied: 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpzf284e.mozrunner\\sdk-fixture-chmod'
Error Cleaning up: [Error 145] The directory is not empty: 'c:\\users\\cltbld~1.t-w\\appdata\\local\\temp\\tmpzf284e.mozrunner'
No handlers could be found for logger "mozrunner"
Done.
Clean the profile.
Traceback (most recent call last):
  File "jetpack/bin/cfx", line 33, in <module>
    cuddlefish.run()
  File "C:\slave\test\build\jetpack\python-lib\cuddlefish\__init__.py", line 603, in run
    test_all_packages(env_root, defaults=options.__dict__)
  File "C:\slave\test\build\jetpack\python-lib\cuddlefish\__init__.py", line 485, in test_all_packages
    env_root=env_root)
  File "C:\slave\test\build\jetpack\python-lib\cuddlefish\__init__.py", line 925, in run
    pkgdir=options.pkgdir)
  File "C:\slave\test\build\jetpack\python-lib\cuddlefish\runner.py", line 743, in run_app
    OUTPUT_TIMEOUT, test_name, parseable)
Exception: Test output exceeded timeout (300s).
program finished with exit code 1
elapsedTime=765.309000
TinderboxPrint:testpkgs<br/>0/0
Unknown Error: command finished with exit code: 1
========= Finished 'python jetpack/bin/cfx ...' warnings (results: 1, elapsed: 12 mins, 49 secs) (at 2014-03-07 07:02:34.603850) =========
}

Some issues:
* No assertion stack
* Log says that a test timed out rather than we crashed
* Cleanup failed
* We shouldn't output a timeout as an exception

Please could we get some of these covered to make sure that jetpack continues to meet https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy ? 

Thanks :-)

Comment 1

[Dave Townsend [:mossop]]

Is this the same as bug 873204?

Comment 2

[Ed Morley [:emorley]]

(In reply to Dave Townsend [:mossop] from comment #1)
> Is this the same as bug 873204?

Since comment 0 is of a fatal assertion, I think bug 873204 should fix this. Though would be good to see stacks for non-fatal assertions too, but I don't have an example of where that is/isn't working, so let's forget about that part for now.