1380731 - CI sometimes succeeds even though there are test failures

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P1)

Description

[[e-mail] (:bhearsum)]

Eg: https://public-artifacts.taskcluster.net/MRW75V5zRoOgpFAb_OgxDw/0/public/logs/live_backing.log

==================== 8 failed, 881 passed in 256.01 seconds ====================
ERROR: InvocationError: '/app/.tox/py27/bin/py.test -n2 --cov=. --doctest-modules auslib'
py27 runtests: commands[2] | coverage run -a scripts/test-rules.py
Starting /app/scripts/../aus-data-snapshots/3.6.27-10.0.2-major
PASS   (108 PASS, 0 FAIL, 0.57 sec, 188.1 tests/second)
Starting /app/scripts/../aus-data-snapshots/rule-priorities
PASS   (2 PASS, 0 FAIL)
Starting /app/scripts/../aus-data-snapshots/version-matching
PASS   (8 PASS, 0 FAIL)
Starting /app/scripts/../aus-data-snapshots/alias
PASS   (8 PASS, 0 FAIL)
Starting /app/scripts/../aus-data-snapshots/locale-level-overrides
PASS   (2 PASS, 0 FAIL)
Starting /app/scripts/../aus-data-snapshots/xml-attributes-v2
PASS   (8 PASS, 0 FAIL)
Starting /app/scripts/../aus-data-snapshots/fallback-override
PASS   (1 PASS, 0 FAIL)
Starting /app/scripts/../aus-data-snapshots/fallback-falls-back
PASS   (2 PASS, 0 FAIL)
Starting /app/scripts/../aus-data-snapshots/3.6.12-3.6.13
PASS   (324 PASS, 0 FAIL, 2.38 sec, 136.2 tests/second)
Coverage.py warning: No data was collected. (no-data-collected)
___________________________________ summary ____________________________________
ERROR:   py27: commands failed
[taskcluster 2017-07-13 15:16:27.533Z] === Task Finished ===
[taskcluster 2017-07-13 15:16:27.534Z] Successful task run with exit code: 0 completed in 420.198 seconds


Most likely broken by the new coveralls code at https://github.com/mozilla/balrog/blob/master/scripts/run.sh#L83 -- the tests are no longer the last command we run, so their exit code is not the exit code of the overall script.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: fix test failure detection

Comment 2

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8886356 [details] [review]
fix test failure detection

LGTM. More details in PR.

Comment 3

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/60c089482c0ecc41a0bf396c9462bbb4c7b36c1a
bug 1380731: Fix test failure detection. (#353). r=jlorenzo

Comment 4

[[e-mail] (:bhearsum)]

This is in production