Closed Bug 1526777 Opened 6 years ago Closed 6 years ago

Something's funny about how we parse success/failure for runs with no tests on Windows vs. mac/linux

Categories

(Testing :: Mochitest, defect)

Product:
Testing
Component:
Mochitest
Version:
Version 3
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox67 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla67
Tracking Flags:
Tracking Status
firefox67 --- fixed

People

(Reporter: Gijs, Assigned: ahal)

References

Details

Attachments

(1 file)

Bug 1526777 - [mozharness] Restrict success_codes hack in desktop_unittest.py to Windows 7 reftests, r?jmaher
47 bytes, text/x-phabricator-request
Details | Review

Observe this trypush:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4fce3df81ff12d6cc889f423936800dc1adfc41a&selectedJob=227417751

Try syntax:

try: -b od -t none -p win64,macosx64,linux64,linux64-asan -u mochitest-browser-chrome-e10s-1,mochitest-devtools-chrome-e10s-1 --try-test-path browser-chrome:browser/base/content/test/static/ devtools-chrome:browser/base/content/test/static/

Note that asan/debug linux and mac are orange. This is because that manifest has skip-ifs for asan || debug, and so they're running no tests at all, skipping everything.

I would expect debug Windows to be orange for the same reason, but it isn't. This is pretty odd. Compare:

Windows:

16:15:46     INFO - SUITE-START | Running 4 tests
16:15:46     INFO - TEST-START | browser/base/content/test/static/browser_all_files_referenced.js
16:15:46     INFO - TEST-SKIP | browser/base/content/test/static/browser_all_files_referenced.js | took 0ms
16:15:46     INFO - TEST-START | browser/base/content/test/static/browser_misused_characters_in_strings.js
16:15:46     INFO - TEST-SKIP | browser/base/content/test/static/browser_misused_characters_in_strings.js | took 0ms
16:15:46     INFO - TEST-START | browser/base/content/test/static/browser_parsable_css.js
16:15:46     INFO - TEST-SKIP | browser/base/content/test/static/browser_parsable_css.js | took 0ms
16:15:46     INFO - TEST-START | browser/base/content/test/static/browser_parsable_script.js
16:15:46     INFO - TEST-SKIP | browser/base/content/test/static/browser_parsable_script.js | took 0ms
16:15:46     INFO -  TEST-INFO | checking window state
16:15:46     INFO -  Browser Chrome Test Summary
16:15:46     INFO -  	Passed: 0
16:15:46     INFO -  	Failed: 0
16:15:46     INFO -  	Todo: 0
16:15:46     INFO -  	Mode: e10s
16:15:46     INFO -  *** End BrowserChrome Test Results ***
16:15:46     INFO - Buffered messages finished
16:15:46     INFO - SUITE-END | took 0s
16:15:46    ERROR - Return code: 1
16:15:46     INFO - TinderboxPrint: mochitest-browser-chrome-chunked<br/>4/0/0
16:15:46     INFO - # TBPL SUCCESS #
16:15:46     INFO - The mochitest suite: browser-chrome-chunked ran with return status: SUCCESS

Mac:

07:36:56     INFO - SUITE-START | Running 4 tests
07:36:56     INFO - TEST-START | browser/base/content/test/static/browser_all_files_referenced.js
07:36:56     INFO - TEST-SKIP | browser/base/content/test/static/browser_all_files_referenced.js | took 0ms
07:36:56     INFO - TEST-START | browser/base/content/test/static/browser_misused_characters_in_strings.js
07:36:56     INFO - TEST-SKIP | browser/base/content/test/static/browser_misused_characters_in_strings.js | took 0ms
07:36:56     INFO - TEST-START | browser/base/content/test/static/browser_parsable_css.js
07:36:56     INFO - TEST-SKIP | browser/base/content/test/static/browser_parsable_css.js | took 0ms
07:36:56     INFO - TEST-START | browser/base/content/test/static/browser_parsable_script.js
07:36:56     INFO - TEST-SKIP | browser/base/content/test/static/browser_parsable_script.js | took 1ms
07:36:56     INFO -  TEST-INFO | checking window state
07:36:56     INFO -  Browser Chrome Test Summary
07:36:56     INFO -  	Passed: 0
07:36:56     INFO -  	Failed: 0
07:36:56     INFO -  	Todo: 0
07:36:56     INFO -  	Mode: e10s
07:36:56     INFO -  *** End BrowserChrome Test Results ***
07:36:56     INFO - Buffered messages finished
07:36:56     INFO - SUITE-END | took 0s
07:36:56    ERROR - Return code: 1
07:36:56     INFO - TinderboxPrint: mochitest-browser-chrome-chunked<br/>4/0/0
07:36:56    ERROR - # TBPL FAILURE #
07:36:56  WARNING - setting return code to 2
07:36:56    ERROR - The mochitest suite: browser-chrome-chunked ran with return status: FAILURE

I dunno where this stuff lives off-hand, but it should really be consistent, and it's code-smell-y that it's not.

Easy link for Windows:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=227419840&repo=try&lineNumber=1059-1067

I agree - it seems wrong that the job is not orange.

:jmaher - Do you have an idea why the Windows task did not fail? (More specifically, I think the issue is, why "TBPL SUCCESS" after a non-zero return code from the mochitest harness.)

Flags: needinfo?(jmaher)

this is concerning, the code that prints tbpl status is here:
https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/automation.py#61

that is in mozharness and IIRC we did some hacking in related mozharness code to make it work for test-verify last summer or a year ago. Given that I am still baffled why we get different treatment on windows.

here is a reference to where we define the status from mochitest and oddly there is a special case for windows:
https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/desktop_unittest.py#934

here is a link to evaluate_parser that would make the decision:
https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/structuredlog.py#132

I suspect this is an edge case when we have no tests run, but I would rather remove "I suspect" from my statement.

Flags: needinfo?(jmaher)

Yeah we should definitely remove that if there aren't any more false positives, here's a quick try run to test:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=07e5d35a464514b41b379ded1ee6d60fc4f71438

That bug was filed 4 years ago, so I imagine it got fixed by the taskcluster migration.

Flags: needinfo?(ahal)

Looks like reftests (incl. crashtest + jsreftest + gpu + no-accel reftests) on windows 7 are still returning 1, but not on win10. No idea what to make of that off-hand...

Barring someone having the time to actually dig into the reasons behind this discrepency, we should at least make that exception more restrictive by limiting it to Win7 reftests.

Since this is filed under mochitest and the other bug is already in the comment, I'm going to resolve this bug by making the exception much more specific (Windows 7 reftest only).

We can use bug 1120644 to look into why the tasks are returning 1 on Windows 7 reftests (though I wouldn't be surprised if we end up ignoring it until we stop supporting Windows 7).

Assignee: nobody → ahal
Status: NEW → ASSIGNED

Bug 1120644 will be used to look into why Windows 7 reftests are still returning 1.

Try looks good:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c2c960b5a5963a2938c83f515fe1d2529156a086

Bustage is from artifact builds. There's a chance that we missed something and this will be backed out, but probably better to risk it than run every task.

Pushed by ahalberstadt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/679b06a8a1aa [mozharness] Restrict success_codes hack in desktop_unittest.py to Windows 7 reftests, r=jmaher

https://hg.mozilla.org/mozilla-central/rev/679b06a8a1aa

Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox67: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: