1772754 - A task with only skipped test is marked as failing

Status: NEW

(Testing :: Mochitest, defect, P3)

Description

[Calixte Denizet (:calixte)]

The following task is marked as failing:
https://treeherder.mozilla.org/jobs?repo=try&revision=9ca2e69d891bdeffc76dfa9b4fc72648aa698942&selectedTaskRun=df1y1Dy0TTGtHm8rNeBvBA.0

There are only two tests ran in this task:
https://hg.mozilla.org/try/rev/9ca2e69d891bdeffc76dfa9b4fc72648aa698942#l1.7

and they're skipped on some platforms:
https://searchfox.org/mozilla-central/rev/9dc5ffe42635b602d4ddfc9a4b8ea0befc94975a/devtools/client/framework/test/browser.ini#182-186

Comment 1

[Marco Castelluccio [:marco]]

Is this effectively the same as bug 1667271?

Comment 2

[Julian Descottes [:jdescottes]]

(In reply to Marco Castelluccio [:marco] from comment #1)

Is this effectively the same as bug 1667271?

Not sure. In our case some valid tests are selected for the query, but they are skipped because of the platform they run on. It looks like a different failure.

Comment 3

[Marco Castelluccio [:marco]]

Isn't it what bug 1667271 is about too?

Comment 4

[Julian Descottes [:jdescottes]]

Maybe, I was saying this because the logs for the other bug have more errors in it.

For the other bug the logs show more errors before the end of the test:

[task 2022-05-30T21:15:14.479Z] 21:15:14    ERROR - No tests were found for flavor 'browser' and the following manifest filters:
[task 2022-05-30T21:15:14.479Z] 21:15:14    ERROR - skip_if, run_if, fail_if, subsuite(name=None), pathprefix(['security/sandbox/test/browser_bug1717599_XDG-CONFIG-DIRS.ini'])
[task 2022-05-30T21:15:14.479Z] 21:15:14    ERROR - 
[task 2022-05-30T21:15:14.479Z] 21:15:14    ERROR - Make sure the test paths (if any) are spelt correctly and the corresponding
[task 2022-05-30T21:15:14.479Z] 21:15:14    ERROR - --flavor and --subsuite are being used. See `mach mochitest --help` for a
[task 2022-05-30T21:15:14.479Z] 21:15:14    ERROR - list of valid flavors.
[task 2022-05-30T21:15:14.479Z] 21:15:14    ERROR - 
[task 2022-05-30T21:15:14.481Z] 21:15:14     INFO - SUITE-START | Running 0 tests
[task 2022-05-30T21:15:14.481Z] 21:15:14     INFO -  TEST-INFO | checking window state
[task 2022-05-30T21:15:14.482Z] 21:15:14     INFO -  Browser Chrome Test Summary
[task 2022-05-30T21:15:14.482Z] 21:15:14     INFO -  	Passed: 0
[task 2022-05-30T21:15:14.482Z] 21:15:14     INFO -  	Failed: 0
[task 2022-05-30T21:15:14.482Z] 21:15:14     INFO -  	Todo: 0
[task 2022-05-30T21:15:14.482Z] 21:15:14     INFO -  	Mode: e10s
[task 2022-05-30T21:15:14.483Z] 21:15:14     INFO -  *** End BrowserChrome Test Results ***
[task 2022-05-30T21:15:14.483Z] 21:15:14     INFO - Buffered messages finished
[task 2022-05-30T21:15:14.484Z] 21:15:14     INFO - SUITE-END | took 0s
[task 2022-05-30T21:15:14.524Z] 21:15:14    ERROR - Return code: 1
[task 2022-05-30T21:15:14.525Z] 21:15:14    ERROR - No checks run.
[task 2022-05-30T21:15:14.525Z] 21:15:14     INFO - TinderboxPrint: mochitest-mochitest-browser-chrome<br/><em class="testfail">T-FAIL</em>
[task 2022-05-30T21:15:14.525Z] 21:15:14    ERROR - # TBPL FAILURE #
[task 2022-05-30T21:15:14.525Z] 21:15:14  WARNING - setting return code to 2
[task 2022-05-30T21:15:14.525Z] 21:15:14    ERROR - The mochitest suite: mochitest-browser-chrome ran with return status: FAILURE

Whereas here, we don't have any error before "SUITE-END".

[task 2022-06-03T18:35:00.559Z] 18:35:00     INFO -  TEST-INFO | checking window state
[task 2022-06-03T18:35:00.560Z] 18:35:00     INFO -  Browser Chrome Test Summary
[task 2022-06-03T18:35:00.561Z] 18:35:00     INFO -  	Passed: 0
[task 2022-06-03T18:35:00.562Z] 18:35:00     INFO -  	Failed: 0
[task 2022-06-03T18:35:00.564Z] 18:35:00     INFO -  	Todo: 0
[task 2022-06-03T18:35:00.565Z] 18:35:00     INFO -  	Mode: e10s
[task 2022-06-03T18:35:00.566Z] 18:35:00     INFO -  *** End BrowserChrome Test Results ***
[task 2022-06-03T18:35:00.567Z] 18:35:00     INFO - Buffered messages finished
[task 2022-06-03T18:35:00.568Z] 18:35:00     INFO - SUITE-END | took 0s
[task 2022-06-03T18:35:00.589Z] 18:35:00    ERROR - Return code: 1
[task 2022-06-03T18:35:00.591Z] 18:35:00     INFO - TinderboxPrint: mochitest-mochitest-devtools-chrome<br/>2/0/0
[task 2022-06-03T18:35:00.591Z] 18:35:00    ERROR - # TBPL FAILURE #
[task 2022-06-03T18:35:00.591Z] 18:35:00  WARNING - setting return code to 2
[task 2022-06-03T18:35:00.591Z] 18:35:00    ERROR - The mochitest suite: mochitest-devtools-chrome ran with return status: FAILURE

Comment 5

[Andrew Halberstadt [:ahal]]

I'd say this is a separate bug.

The fact that the tests are included via ../../../ in the manifest is likely the catalyst for this bug. I don't know exactly what's going wrong, but there's a mismatch between manifests and dirs and this is tripping up the logic in Taskgraph that is supposed to ensure test chunks run actual tests that don't get skipped.

Possible solutions (in order of my preference):

1. Avoid ../../ in the manifest (dupe tests?)
2. Have updatebot pass in manifests rather than paths (unsure if will work or if acceptable?)
3. Try to fix this in test resolving logic (unsure how difficult this is)
4. Make the test harness go green if this happens (but then we'd miss problems in the resolving logic)

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:ahal, could you have a look please?

For more information, please visit auto_nag documentation.