1364030 - Travis JS job is green even though the log shows tests are failing

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

There are multiple failures visible in the logs of the JS run on Travis, however `yarn test` is still returning zero, which means the overall job shows as passing/green:
https://travis-ci.org/mozilla/treeherder/jobs/230892456

We need to:
1) Figure out why `yarn test` isn't exiting non-zero
2) Fix the test failures that have crept in whilst this has been broken

I'm guessing this is a regression from the Neutrino switch.

Comment 1

[Ed Morley [:emorley]]

Travis reports 'The command "yarn test" exited with 0.'.

`yarn test` runs `neutrino test --presets ./neutrino-custom/test.js`:
https://github.com/mozilla/treeherder/blob/d9045d01b1dd14a61e5f003649b3cb51fac40ba7/neutrino-custom/test.js

...which makes use of neutrino-preset-karma v4.2.0:
https://github.com/mozilla-neutrino/neutrino-dev/blob/85ed67f035de81a861d49a74f97f6dbede61c004/packages/neutrino-preset-karma/src/index.js

Whilst we're on an old version of Neutrino, it doesn't look like that preset has changed much between v4 and master:
https://github.com/mozilla-neutrino/neutrino-dev/blob/master/packages/neutrino-preset-karma/index.js

...so I've filed an issue against Neutrino to check it's working as expected:
https://github.com/mozilla-neutrino/neutrino-dev/issues/268

Comment 2

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:fix-travis-js-tests > mozilla:master

Comment 3

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/72a3ce8140600b6fa4712cf6e60a2fb98c707808
Bug 1364030 - Update to neutrino-preset-karma 4.2.1

To pick up the fix for failing JS tests not making the Travis run
fail (mozilla-neutrino/neutrino-dev#268).