I see a failure  when looking at failures on taskcluster  and while looking at a buildbot  log for comparison, I see... the same failure. Odd, the buildbot job is green . How is this possibly you say? I don't know, so I filed a bug. on buildbot, I see this summary: 06:23:04 INFO - REFTEST INFO | Result summary: 06:23:04 INFO - REFTEST INFO | Successful: 3101 (4 pass, 3097 load only) 06:23:04 INFO - REFTEST INFO | Unexpected: 1 (0 unexpected fail, 0 unexpected pass, 1 unexpected asserts, 0 failed load, 0 exception) 06:23:04 INFO - REFTEST INFO | Known problems: 52 (0 known fail, 40 known asserts, 0 random, 12 skipped, 0 slow) 06:23:04 INFO - REFTEST SUITE-END | Shutdown on taskcluster, I see this summary: 3:15:38 INFO - REFTEST INFO | Result summary: 23:15:38 INFO - REFTEST INFO | Successful: 3101 (4 pass, 3097 load only) 23:15:38 WARNING - REFTEST INFO | Unexpected: 1 (0 unexpected fail, 0 unexpected pass, 1 unexpected asserts, 0 failed load, 0 exception) 23:15:38 WARNING - One or more unittests failed. 23:15:38 INFO - REFTEST INFO | Known problems: 51 (0 known fail, 39 known asserts, 0 random, 12 skipped, 0 slow) 23:15:38 INFO - REFTEST SUITE-END | Shutdown the test in question is bug 1319318: 06:17:52 INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///c:/slave/test/build/tests/reftest/tests/layout/generic/crashtests/1271765.html | assertion count 3 is more than expected 0 assertions this is the case for win7/win8 debug crashtests[-e10s]. it seems as though mozharness is not seeing the error: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/unittest.py?q=One+or+more+unittests+failed&redirect_type=single#141 of course, this leads me to question... 1) what other failures are we not seeing 2) why is this failing specifically with the taskcluster works vs buildbot 3) this issue has happened before on other jobs, why do we keep having problems  https://public-artifacts.taskcluster.net/NarjWbHJQSiJEVV0fHGImg/0/public/logs/live_backing.log  https://firstname.lastname@example.org&selectedJob=32273017  https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=win%20crashtest&selectedJob=7702729  https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-win32-debug/1481107298/autoland_win7_vm-debug_test-crashtest-bm139-tests1-windows-build354.txt.gz
here is a case on windows where it did show up as orange: https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-win64-debug/1480706564/autoland_win8_64-debug_test-crashtest-e10s-bm110-tests1-windows-build14.txt.gz and I believe the reason why it posted an orange result is because there was a crash as well so that triggered the full failure summary.
what is even more maddening is that we have the failure listed in the failure summary, but the job is green. Not sure if this is a treeherder issue or if it is really a mozharness issue and treeherder just doesn't see the proper message/return code.
Dup of bug 1321127?
Treeherder displays jobs as passing/failing based on what buildbot/taskcluster reported the job status as, and not what the arbitrary regex may or may not have matched in the raw text log. So this needs to be fixed in the harness/buildbot/taskcluster layer.
I agree this is a duplicate of bug 1321127
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1321127
You need to log in before you can comment on or make changes to this bug.