crashtests are perma fail on win7 debug, but showing green

RESOLVED DUPLICATE of bug 1321127

Status

Testing
Reftest
RESOLVED DUPLICATE of bug 1321127
a year ago
a year ago

People

(Reporter: jmaher, Unassigned)

Tracking

Trunk
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

I see a failure [1] when looking at failures on taskcluster [2] and while looking at a buildbot [3] log for comparison, I see... the same failure.  Odd, the buildbot job is green [4].  How is this possibly you say?  I don't know, so I filed a bug.

on buildbot, I see this summary:
06:23:04     INFO - REFTEST INFO | Result summary:
06:23:04     INFO - REFTEST INFO | Successful: 3101 (4 pass, 3097 load only)
06:23:04     INFO - REFTEST INFO | Unexpected: 1 (0 unexpected fail, 0 unexpected pass, 1 unexpected asserts, 0 failed load, 0 exception)
06:23:04     INFO - REFTEST INFO | Known problems: 52 (0 known fail, 40 known asserts, 0 random, 12 skipped, 0 slow)
06:23:04     INFO - REFTEST SUITE-END | Shutdown

on taskcluster, I see this summary:
3:15:38     INFO -  REFTEST INFO | Result summary:
23:15:38     INFO -  REFTEST INFO | Successful: 3101 (4 pass, 3097 load only)
23:15:38  WARNING -  REFTEST INFO | Unexpected: 1 (0 unexpected fail, 0 unexpected pass, 1 unexpected asserts, 0 failed load, 0 exception)
23:15:38  WARNING -  One or more unittests failed.
23:15:38     INFO -  REFTEST INFO | Known problems: 51 (0 known fail, 39 known asserts, 0 random, 12 skipped, 0 slow)
23:15:38     INFO -  REFTEST SUITE-END | Shutdown

the test in question is bug 1319318:
06:17:52     INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///c:/slave/test/build/tests/reftest/tests/layout/generic/crashtests/1271765.html | assertion count 3 is more than expected 0 assertions

this is the case for win7/win8 debug crashtests[-e10s].

it seems as though mozharness is not seeing the error:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/unittest.py?q=One+or+more+unittests+failed&redirect_type=single#141

of course, this leads me to question...
1) what other failures are we not seeing
2) why is this failing specifically with the taskcluster works vs buildbot
3) this issue has happened before on other jobs, why do we keep having problems


[1] https://public-artifacts.taskcluster.net/NarjWbHJQSiJEVV0fHGImg/0/public/logs/live_backing.log
[2] https://treeherder.mozilla.org/#/jobs?repo=try&author=pmoore@mozilla.com&selectedJob=32273017
[3] https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=win%20crashtest&selectedJob=7702729
[4] https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-win32-debug/1481107298/autoland_win7_vm-debug_test-crashtest-bm139-tests1-windows-build354.txt.gz
here is a case on windows where it did show up as orange:
https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-win64-debug/1480706564/autoland_win8_64-debug_test-crashtest-e10s-bm110-tests1-windows-build14.txt.gz

and I believe the reason why it posted an orange result is because there was a crash as well so that triggered the full failure summary.
what is even more maddening is that we have the failure listed in the failure summary, but the job is green.  Not sure if this is a treeherder issue or if it is really a mozharness issue and treeherder just doesn't see the proper message/return code.
Treeherder displays jobs as passing/failing based on what buildbot/taskcluster reported the job status as, and not what the arbitrary regex may or may not have matched in the raw text log. So this needs to be fixed in the harness/buildbot/taskcluster layer.
I agree this is a duplicate of bug 1321127
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1321127
You need to log in before you can comment on or make changes to this bug.