Closed Bug 1174901 Opened 10 years ago Closed 10 years ago

Intermittent Gij "Tests exited with return code 2: harness failures" only when tests were run but the harness "Couldn't find failed count in marionette-mocha output:"

Categories

(Testing Graveyard :: JSMarionette, defect)

Product:
Testing Graveyard
Component:
JSMarionette
Version:
34 Branch
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Assigned: aus)

References

Details

Attachments

(2 files, 3 obsolete files)

[gaia] nullaus:bug1174901 > mozilla-b2g:master
46 bytes, text/x-github-pull-request
gaye
: review+
Details | Review
[gaia] nullaus:bug1174901 > mozilla-b2g:master
46 bytes, text/x-github-pull-request
Eli
: review+
Details | Review
https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/cJ4qV897Qi-mPhn4lLr3Eg/0/public/logs/live_backing.log 20:34:07 INFO - ...... 20:34:07 INFO - TEST-START | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync 20:34:07 INFO - TEST-PASS | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync 20:34:07 INFO - TEST-END | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync took 39 ms 20:34:07 INFO - TEST-START | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test 20:34:07 INFO - TEST-PASS | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test 20:34:07 INFO - TEST-END | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test took 20 ms[TypeError: Cannot read property '1' of null] 20:34:07 INFO - Couldn't find passed count in marionette-mocha output: 20:34:07 INFO - TEST-START | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync 20:34:07 INFO - TEST-PASS | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync 20:34:07 INFO - TEST-END | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync took 39 ms 20:34:07 INFO - TEST-START | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test 20:34:07 INFO - TEST-PASS | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test 20:34:07 INFO - TEST-END | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test took 20 ms 20:34:07 INFO - [TypeError: Cannot read property '1' of null] 20:34:07 INFO - Couldn't find failed count in marionette-mocha output: 20:34:07 INFO - TEST-START | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync 20:34:07 INFO - TEST-PASS | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync 20:34:07 INFO - TEST-END | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync took 39 ms 20:34:07 INFO - TEST-START | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test 20:34:07 INFO - TEST-PASS | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test 20:34:07 INFO - TEST-END | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test took 20 ms 20:34:07 INFO - [TypeError: Cannot read property '1' of null] 20:34:07 INFO - Couldn't find todo count in marionette-mocha output: 20:34:07 INFO - TEST-START | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync 20:34:07 INFO - TEST-PASS | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync 20:34:07 INFO - TEST-END | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client is sync took 39 ms 20:34:07 INFO - TEST-START | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test 20:34:07 INFO - TEST-PASS | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test 20:34:07 INFO - TEST-END | tests/jsmarionette/runner/marionette-js-runner/test/integration/client_test.js | default client host is reset is each test took 20 ms 20:34:15 INFO - ...
Summary: Intermittent Gij-20 "Tests exited with return code 2: harness failures" when tests were run but the harness "Couldn't find failed count in marionette-mocha output:" → Intermittent Gij-20 and only Gij-20 "Tests exited with return code 2: harness failures" only when tests were run but the harness "Couldn't find failed count in marionette-mocha output:"
Aus, can you please look into this?
Flags: needinfo?(aus)
Yep, I'm on it! Leaving the needinfo? set until I figure out what's going on.
Flags: needinfo?(aus)
I'm not sure what's going on, I'm going to have to push some of my own logging up to try and see what's going on. I haven't been able to reproduce this locally yet.
Depends on: 1186235
I moved the mozharness tag on b2g-inbound to the latest (includes fix in bug 1186235). I'm hoping it will take care of this intermittent weirdness.
Assignee: nobody → aus
Attachment #8639401 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #8640801 - Attachment is obsolete: true
Attachment #8640815 - Flags: review?(gaye)
Commit (master): https://github.com/mozilla-b2g/gaia/commit/389bbb06f1a0de5c027f1446a7bc2c0a4e3260dd ran the fix by :gaye already via IRC.
Comment on attachment 8640815 [details] [review] [gaia] nullaus:bug1174901 > mozilla-b2g:master lgtm
Attachment #8640815 - Flags: review?(gaye) → review+
Summary: Intermittent Gij-20 and only Gij-20 "Tests exited with return code 2: harness failures" only when tests were run but the harness "Couldn't find failed count in marionette-mocha output:" → Intermittent Gij "Tests exited with return code 2: harness failures" only when tests were run but the harness "Couldn't find failed count in marionette-mocha output:"
Another fix to make things a better: https://github.com/mozilla-b2g/gaia/commit/c25f1e8ccfdc4feff96236f9d3ebe01b29232b46 still seems to fail sometimes, but with a lot less frequency.
Attachment #8644616 - Flags: review?(gaye)
Attachment #8643321 - Attachment is obsolete: true
Attachment #8644616 - Flags: review?(gaye) → review?(eperelman)
Attachment #8644616 - Flags: review?(eperelman) → review+
Commit (master): https://github.com/mozilla-b2g/gaia/commit/6d66a9eff9fd33fafc97bc75753558969de48cc3 Hopefully this will resolve this issue, I certainly didn't see it happen in any of the many test runs I did. If we could instead open a new bug if this resurfaces, that'd be much appreciated. The list of TBPL robot entries is so huge that it takes minutes to load this bug. :(
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
FWIW, I think this bug gets blamed for various Gij test problems. It shows up in the "Failure summary" any time the test spits out something like "Tests exited with return code 2: harness failures", which is pretty much all failures.
Yes, you can see from the summary that I knew pretty much from the start that it was going to be a dumping ground. The first step in fixing this bug should have been to fix whatever harness that is that outputs a fatal error without making it tbpl-parseable to do a better job with its output. From what I've seen over the last couple of days, Gij currently silently fails for timeouts and for crashes, both of which are clear https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy violations, so the bugs (if they exist) about them can be blocker-severity, once someone notices that and hides Gij because of it.
Keywords: intermittent-failure
This bug is further obscure by the fact we don't get error messages when the harness detects a crash, so I'm adding that as a dependency.
Depends on: 1175116
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: