1033074 - webapprt-test-chrome run fails on warning despite passing all tests

Status: RESOLVED FIXED

(Firefox Graveyard :: Webapp Runtime, defect)

Description

[Myk Melez [:myk] [@mykmelez]]

All webapprt-test-chrome tests passed in this Linux test run <https://tbpl.mozilla.org/php/getParsedLog.php?id=42812763&full=1&branch=cedar>, but the run still failed due to some warning:

19:00:10     INFO -  Browser Chrome Test Summary
19:00:10     INFO -  	Passed: 243
19:00:10     INFO -  	Failed: 0
19:00:10     INFO -  	Todo: 14
19:00:10     INFO -  *** End BrowserChrome Test Results ***
…
========= Finished '/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py ...' warnings (results: 1, elapsed: 1 mins, 53 secs) (at 2014-06-30 19:00:12.668425) =========


But it isn't clear to me which warning is causing the failure.  Searching the log for "warning" turns up these lines (excerpted with a bit of context):

18:59:11     INFO -    Running setup.py (path:/builds/slave/test/build/venv/build/mock/setup.py) egg_info for package mock
18:59:11     INFO -      warning: no files found matching '*.png' under directory 'docs'
18:59:11     INFO -      warning: no files found matching '*.css' under directory 'docs'
18:59:11     INFO -      warning: no files found matching '*.html' under directory 'docs'
18:59:11     INFO -      warning: no files found matching '*.js' under directory 'docs'
18:59:11     INFO -  Installing collected packages: mock
18:59:11     INFO -    Running setup.py install for mock
18:59:11     INFO -      warning: no files found matching '*.png' under directory 'docs'
18:59:11     INFO -      warning: no files found matching '*.css' under directory 'docs'
18:59:11     INFO -      warning: no files found matching '*.html' under directory 'docs'
18:59:11     INFO -      warning: no files found matching '*.js' under directory 'docs'
18:59:11     INFO -  Successfully installed mock
…
18:59:56     INFO - Calling ['/builds/slave/test/build/venv/bin/python', '-u', '/builds/slave/test/build/tests/mochitest/runtests.py', '--app=/builds/slave/test/build/application/firefox/webapprt-stub', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/cedar-linux64/1404177415/firefox-33.0a1.en-US.linux-x86_64.crashreporter-symbols.zip', '--certificate-path=tests/certs', '--autorun', '--close-when-done', '--console-level=INFO', '--testing-modules-dir=tests/modules', '--quiet', '--webapprt-chrome', '--browser-arg=-test-mode'] with output_timeout 1000
18:59:57     INFO -  /builds/slave/test/build/venv/local/lib/python2.7/site-packages/mozrunner/utils.py:20: UserWarning: Module moznetwork was already imported from /builds/slave/test/build/tests/mochitest/moznetwork.py, but /builds/slave/test/build/venv/lib/python2.7/site-packages is being added to sys.path
18:59:57     INFO -    import pkg_resources
18:59:57     INFO -  /builds/slave/test/build/venv/local/lib/python2.7/site-packages/mozrunner/utils.py:20: UserWarning: Module manifestparser was already imported from /builds/slave/test/build/tests/mochitest/manifestparser.py, but /builds/slave/test/build/venv/lib/python2.7/site-packages is being added to sys.path
18:59:57     INFO -    import pkg_resources
…
19:00:02     INFO -  *** Start BrowserChrome Test Results ***
19:00:02     INFO -  System JS : WARNING resource://gre/modules/Preferences.jsm:378 - mutating the [[Prototype]] of an object will cause your code to run very slowly; instead create the object with the correct initial [[Prototype]] value using Object.create
19:00:02     INFO -  TEST-INFO | checking window state
19:00:02     INFO -  TEST-INFO | unknown test url | must wait for focus
19:00:02     INFO -  TEST-INFO | (browser-test.js) | Console message: [JavaScript Warning: "mutating the [[Prototype]] of an object will cause your code to run very slowly; instead create the object with the correct initial [[Prototype]] value using Object.create" {file: "resource://gre/modules/Preferences.jsm" line: 378}]
…
19:00:11     INFO -  WARNING | leakcheck | refcount logging is off, so leaks can't be detected!
…
19:00:11  WARNING - # TBPL WARNING #
19:00:11  WARNING - The webapprt suite: chrome ran with return status: WARNING


jgriffin: any chance you know which of these lines is triggering the failure?

Comment 1

[Jonathan Griffin (:jgriffin)]

It's actually because mozharness isn't parsing the passed/todo/failed lines correctly.  I'll have a patch shortly.

Comment 2

[Jonathan Griffin (:jgriffin)]

Attachment: Use 'mochitest' category for handling the webapprt parser,

Webapprt tests use the same test reporting as mochitest, so we can safely do this.

Comment 3

[Jordan Lund (:jlund)]

Comment on attachment 8449059 [details] [diff] [review]
Use 'mochitest' category for handling the webapprt parser,

Review of attachment 8449059 [details] [diff] [review]:
-----------------------------------------------------------------

looks like that will accomplish what we want. I just took a look at what we use the suite_category for and it just points to here: http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/testing/errors.py#21

I think rather than hacking the instance for webapprt, we should add webapprt_summary to this dict explicitly. It looks like that's how we added robocop anyway (robocop's summary appears to be based off mochitest_summary too).

jgriffin, what do you think about doing something like: https://pastebin.mozilla.org/5510169

I'm happy to r+ yours or you r+ mine depending on what you think :)

Comment 4

[Jonathan Griffin (:jgriffin)]

That makes sense, I'll r+ yours and land it.  :)  Thanks.

Comment 5

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/build/mozharness/rev/141595cb922b

Comment 6

[Chris Cooper [:coop] (he/him)]

Merged to production, and deployed.