1322353 - crashtests are perma fail on win7 debug, but showing green

Status: RESOLVED DUPLICATE of bug 1321127

(Testing :: Reftest, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I see a failure [1] when looking at failures on taskcluster [2] and while looking at a buildbot [3] log for comparison, I see... the same failure.  Odd, the buildbot job is green [4].  How is this possibly you say?  I don't know, so I filed a bug.

on buildbot, I see this summary:
06:23:04     INFO - REFTEST INFO | Result summary:
06:23:04     INFO - REFTEST INFO | Successful: 3101 (4 pass, 3097 load only)
06:23:04     INFO - REFTEST INFO | Unexpected: 1 (0 unexpected fail, 0 unexpected pass, 1 unexpected asserts, 0 failed load, 0 exception)
06:23:04     INFO - REFTEST INFO | Known problems: 52 (0 known fail, 40 known asserts, 0 random, 12 skipped, 0 slow)
06:23:04     INFO - REFTEST SUITE-END | Shutdown

on taskcluster, I see this summary:
3:15:38     INFO -  REFTEST INFO | Result summary:
23:15:38     INFO -  REFTEST INFO | Successful: 3101 (4 pass, 3097 load only)
23:15:38  WARNING -  REFTEST INFO | Unexpected: 1 (0 unexpected fail, 0 unexpected pass, 1 unexpected asserts, 0 failed load, 0 exception)
23:15:38  WARNING -  One or more unittests failed.
23:15:38     INFO -  REFTEST INFO | Known problems: 51 (0 known fail, 39 known asserts, 0 random, 12 skipped, 0 slow)
23:15:38     INFO -  REFTEST SUITE-END | Shutdown

the test in question is bug 1319318:
06:17:52     INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///c:/slave/test/build/tests/reftest/tests/layout/generic/crashtests/1271765.html | assertion count 3 is more than expected 0 assertions

this is the case for win7/win8 debug crashtests[-e10s].

it seems as though mozharness is not seeing the error:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/unittest.py?q=One+or+more+unittests+failed&redirect_type=single#141

of course, this leads me to question...
1) what other failures are we not seeing
2) why is this failing specifically with the taskcluster works vs buildbot
3) this issue has happened before on other jobs, why do we keep having problems


[1] https://public-artifacts.taskcluster.net/NarjWbHJQSiJEVV0fHGImg/0/public/logs/live_backing.log
[2] https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&selectedJob=32273017
[3] https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=win%20crashtest&selectedJob=7702729
[4] https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-win32-debug/1481107298/autoland_win7_vm-debug_test-crashtest-bm139-tests1-windows-build354.txt.gz

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

here is a case on windows where it did show up as orange:
https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-win64-debug/1480706564/autoland_win8_64-debug_test-crashtest-e10s-bm110-tests1-windows-build14.txt.gz

and I believe the reason why it posted an orange result is because there was a crash as well so that triggered the full failure summary.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

what is even more maddening is that we have the failure listed in the failure summary, but the job is green.  Not sure if this is a treeherder issue or if it is really a mozharness issue and treeherder just doesn't see the proper message/return code.

Comment 3

[Geoff Brown [:gbrown]]

Dup of bug 1321127?

Comment 4

[Ed Morley [:emorley]]

Treeherder displays jobs as passing/failing based on what buildbot/taskcluster reported the job status as, and not what the arbitrary regex may or may not have matched in the raw text log. So this needs to be fixed in the harness/buildbot/taskcluster layer.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I agree this is a duplicate of bug 1321127