944364 - Windows mochitest runs using mozharness always returning code 1

Status: RESOLVED INCOMPLETE

(Testing :: Mochitest, defect)

Description

[Ed Morley [:emorley]]

In bug 928887, we're trying to fix the fact that mozharness no longer uses the return code to determine the pass/fail state of a run. In order to land that patch, we first need to fix windows mochitest runs, so they don't return code 1 even when no (visible) error occurred.

eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=31217020&tree=Mozilla-Central#error0
(shows as green on TBPL at present)

Comment 1

[Ed Morley [:emorley]]

{
04:21:17     INFO -  runtests.py | Running tests: end.
04:21:17     INFO -  0
04:21:17    ERROR - Return code: 1
04:21:17     INFO - TinderboxPrint: mochitest-plain1<br/>189339/0/870
04:21:17     INFO - # TBPL SUCCESS #
04:21:17     INFO - The mochitest suite: plain1 ran with return status: SUCCESS
04:21:17     INFO - Running post-action listener: _resource_record_post_action
04:21:17     INFO - Running post-run listener: _resource_record_post_run
04:21:18     INFO - Total resource usage - Wall time: 1117s; CPU: 9.0%; Read bytes: 62393856; Write bytes: 560351232; Read time: 2218575; Write time: 2254835
04:21:18     INFO - pull - Wall time: 15s; CPU: 10.0%; Read bytes: 1126400; Write bytes: 52653056; Read time: 226170; Write time: 1174085
04:21:18     INFO - install - Wall time: 2s; CPU: 11.0%; Read bytes: 135168; Write bytes: 17359360; Read time: 523; Write time: 18135
04:21:18     INFO - run-tests - Wall time: 1100s; CPU: 9.0%; Read bytes: 61025792; Write bytes: 436747776; Read time: 1988338; Write time: 1008185
04:21:18     INFO - Running post-run listener: _upload_blobber_files
04:21:18  WARNING - Blob upload gear skipped. Missing cmdline options.
04:21:18     INFO - Copying logs to upload dir...
04:21:18     INFO - mkdir: C:\slave\test\build\upload\logs
program finished with exit code 0
elapsedTime=1283.182000
========= Finished 'c:/mozilla-build/python27/python -u ...' (results: 0, elapsed: 21 mins, 19 secs) (at 2013-11-28 04:21:46.277568) =========
}

> runtests.py | Running tests: end. 
Comes from:
http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#1044

We then return |status|, which is what we use to sys.exit via:
http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#1392

> Return code: 1

Appears to come from:
https://hg.mozilla.org/build/mozharness/file/14ea70995c0c/mozharness/base/script.py#l714

Which is the direct result of the return code from the subprocess.Popen() call (no parsing for failures has occurred at this point):
https://hg.mozilla.org/build/mozharness/file/14ea70995c0c/mozharness/base/script.py#l700

tl;dr this seems like a problem in runTests.py, not mozharness, I've pushed to try with some extra logging:
remote:   https://tbpl.mozilla.org/?tree=Try&rev=a83b5cb96433

Comment 2

[Ed Morley [:emorley]]

More logging:
https://tbpl.mozilla.org/?tree=Try&rev=918577e0ee27

Comment 3

[Ed Morley [:emorley]]

Mass-closing old bugs I filed that have not had recent activity/no longer affect me.