Closed Bug 1667271 Opened 5 years ago Closed 2 months ago

Intermittent No tests were found for flavor '*' and the following manifest filters:

Categories

(Firefox Build System :: Task Configuration, defect, P3)

Product:
Firefox Build System
Component:
Task Configuration
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

References

(Regression)

Details

(Keywords: intermittent-failure, regression, Whiteboard: [stockwell infra])

Filed by: rmaries [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=316657113&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZSt8ySGfRUWwllSTNrDTNw/runs/0/artifacts/public/logs/live_backing.log

[task 2020-09-24T23:48:28.150Z] 23:48:28    ERROR - No tests were found for flavor 'plain' and the following manifest filters:
[task 2020-09-24T23:48:28.152Z] 23:48:28    ERROR - skip_if, run_if, fail_if, subsuite(name=media), pathprefix(['dom/media/webrtc/tests/mochitests/mochitest.ini'])
[task 2020-09-24T23:48:28.152Z] 23:48:28    ERROR - 
[task 2020-09-24T23:48:28.153Z] 23:48:28    ERROR - Make sure the test paths (if any) are spelt correctly and the corresponding
[task 2020-09-24T23:48:28.154Z] 23:48:28    ERROR - --flavor and --subsuite are being used. See `mach mochitest --help` for a
[task 2020-09-24T23:48:28.155Z] 23:48:28    ERROR - list of valid flavors.
[task 2020-09-24T23:48:28.156Z] 23:48:28    ERROR - 
[task 2020-09-24T23:48:28.158Z] 23:48:28     INFO - SUITE-START | Running 0 tests
[task 2020-09-24T23:48:28.159Z] 23:48:28     INFO -  0 INFO TEST-START | Shutdown
[task 2020-09-24T23:48:28.159Z] 23:48:28     INFO -  1 INFO Passed:  0
[task 2020-09-24T23:48:28.160Z] 23:48:28     INFO -  2 INFO Failed:  0
[task 2020-09-24T23:48:28.161Z] 23:48:28     INFO -  3 INFO Todo:    0
[task 2020-09-24T23:48:28.162Z] 23:48:28     INFO -  4 INFO Mode:    e10s
[task 2020-09-24T23:48:28.163Z] 23:48:28     INFO -  5 INFO SimpleTest FINISHED
[task 2020-09-24T23:48:28.163Z] 23:48:28     INFO - Buffered messages finished
[task 2020-09-24T23:48:28.164Z] 23:48:28     INFO - SUITE-END | took 0s
[task 2020-09-24T23:48:28.199Z] 23:48:28    ERROR - Return code: 1
[task 2020-09-24T23:48:28.200Z] 23:48:28    ERROR - No checks run.
[task 2020-09-24T23:48:28.200Z] 23:48:28     INFO - TinderboxPrint: mochitest-mochitest-media<br/><em class="testfail">T-FAIL</em>
[task 2020-09-24T23:48:28.201Z] 23:48:28    ERROR - # TBPL FAILURE #
[task 2020-09-24T23:48:28.201Z] 23:48:28  WARNING - setting return code to 2
[task 2020-09-24T23:48:28.201Z] 23:48:28    ERROR - The mochitest suite: mochitest-media ran with return status: FAILURE
[task 2020-09-24T23:48:28.205Z] 23:48:28     INFO - Running post-action listener: _package_coverage_data
[task 2020-09-24T23:48:28.205Z] 23:48:28     INFO - Running post-action listener: _resource_record_post_action
[task 2020-09-24T23:48:28.205Z] 23:48:28     INFO - Running post-action listener: process_java_coverage_data
[task 2020-09-24T23:48:28.205Z] 23:48:28     INFO - [mozharness: 2020-09-24 23:48:28.203278Z] Finished run-tests step (success)
[task 2020-09-24T23:48:28.205Z] 23:48:28     INFO - Running post-run listener: _resource_record_post_run
[task 2020-09-24T23:48:28.251Z] 23:48:28     INFO - Validating Perfherder data against /builds/worker/workspace/mozharness/external_tools/performance-artifact-schema.json
[task 2020-09-24T23:48:28.255Z] 23:48:28     INFO - PERFHERDER_DATA: {"framework": {"name": "job_resource_usage"}, "suites": [{"subtests": [{"name": "cpu_percent", "value": 50.33}, {"name": "io_write_bytes", "value": 1761280}, {"name": "io.read_bytes", "value": 8192}, {"name": "io_write_time", "value": 284}, {"name": "io_read_time", "value": 0}], "extraOptions": ["e10s", "taskcluster-m5.large"], "name": "mochitest.mochitest-media.overall"}, {"subtests": [{"name": "time", "value": 0.03249788284301758}], "name": "mochitest.mochitest-media.start-pulseaudio"}, {"subtests": [{"name": "time", "value": 9.73744511604309}, {"name": "cpu_percent", "value": 50.0625}], "name": "mochitest.mochitest-media.install"}, {"subtests": [{"name": "time", "value": 0.0009038448333740234}], "name": "mochitest.mochitest-media.stage-files"}, {"subtests": [{"name": "time", "value": 0.9638168811798096}], "name": "mochitest.mochitest-media.run-tests"}]}
[task 2020-09-24T23:48:28.256Z] 23:48:28     INFO - Total resource usage - Wall time: 10s; CPU: Can't collect data; Read bytes: 8192; Write bytes: 1761280; Read time: 0; Write time: 284
[task 2020-09-24T23:48:28.256Z] 23:48:28     INFO - TinderboxPrint: I/O read bytes / time<br/>8,192 / 0
[task 2020-09-24T23:48:28.257Z] 23:48:28     INFO - TinderboxPrint: I/O write bytes / time<br/>1,761,280 / 284
[task 2020-09-24T23:48:28.257Z] 23:48:28     INFO - TinderboxPrint: CPU idle<br/>9.9 (49.6%)
[task 2020-09-24T23:48:28.257Z] 23:48:28     INFO - TinderboxPrint: CPU system<br/>0.4 (2.0%)
[task 2020-09-24T23:48:28.258Z] 23:48:28     INFO - TinderboxPrint: CPU user<br/>9.7 (48.5%)
[task 2020-09-24T23:48:28.258Z] 23:48:28     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[task 2020-09-24T23:48:28.259Z] 23:48:28     INFO - start-pulseaudio - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2020-09-24T23:48:28.259Z] 23:48:28     INFO - install - Wall time: 10s; CPU: 50.0%; Read bytes: 0; Write bytes: 1761280; Read time: 0; Write time: 284
[task 2020-09-24T23:48:28.259Z] 23:48:28     INFO - stage-files - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2020-09-24T23:48:28.260Z] 23:48:28     INFO - run-tests - Wall time: 1s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2020-09-24T23:48:28.261Z] 23:48:28  WARNING - returning nonzero exit status 2
[task 2020-09-24T23:48:28.276Z] cleanup
[task 2020-09-24T23:48:28.276Z] + cleanup
[task 2020-09-24T23:48:28.277Z] + local rv=2
[task 2020-09-24T23:48:28.277Z] + [[ -s /builds/worker/.xsession-errors ]]
[task 2020-09-24T23:48:28.277Z] + cp /builds/worker/.xsession-errors /builds/worker/artifacts/public/xsession-errors.log
[task 2020-09-24T23:48:28.279Z] + true
[task 2020-09-24T23:48:28.279Z] + cleanup_xvfb
[task 2020-09-24T23:48:28.280Z] ++ pidof Xvfb
[task 2020-09-24T23:48:28.283Z] + local xvfb_pid=48
[task 2020-09-24T23:48:28.283Z] + local vnc=false
[task 2020-09-24T23:48:28.283Z] + local interactive=false
[task 2020-09-24T23:48:28.294Z] + '[' -n 48 ']'
[task 2020-09-24T23:48:28.295Z] + [[ false == false ]]
[task 2020-09-24T23:48:28.295Z] + [[ false == false ]]
[task 2020-09-24T23:48:28.295Z] + kill 48
[task 2020-09-24T23:48:28.295Z] + screen -XS xvfb quit
[task 2020-09-24T23:48:28.328Z] No screen session found.
[task 2020-09-24T23:48:28.329Z] + true
[task 2020-09-24T23:48:28.329Z] + '[' ']'
[task 2020-09-24T23:48:28.329Z] + exit 2
[taskcluster 2020-09-24 23:48:28.616Z] === Task Finished ===
[taskcluster 2020-09-24 23:48:29.255Z] Unsuccessful task run with exit code: 2 completed in 96.376 seconds```

Andrew, could you take a look?

Flags: needinfo?(ahal)

Please fix this issue before relanding bug 1665713.

Flags: needinfo?(mfroman)
Keywords: regression
Regressed by: 1665713
Has Regression Range: --- → yes

Looks like this was a regression that ended up being backed out, so clearing NI. If my input is still required, please flag me again.

Flags: needinfo?(ahal)

Thank you for the followup. This is an interesting failure case since I did a ./mach try auto prior to landing and it looked ok. However, I failed to notice that it didn't run mda mochitests, which were the very tests moved in Bug 1665713. I'll take a look.

Flags: needinfo?(mfroman)

(In reply to Michael Froman [:mjf] from comment #4)

Thank you for the followup. This is an interesting failure case since I did a ./mach try auto prior to landing and it looked ok. However, I failed to notice that it didn't run mda mochitests, which were the very tests moved in Bug 1665713. I'll take a look.

Thanks for pointing that out, probably bugbug (the service computing what to run) either wasn't aware that the manifests existed, or lost all the data it had built up on the old manifest upon the rename and so didn't know it was relevant. I filed bug 1667401 to figure out a fix for this case.

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: INCOMPLETE → ---
Summary: Intermittent No tests were found for flavor 'plain' and the following manifest filters: → Intermittent No tests were found for flavor '*' and the following manifest filters:

There have been 69 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-tsan-qr
  • linux1804-64-qr
Whiteboard: [stockwell needswork:owner]
Whiteboard: [stockwell needswork:owner] → [stockwell needswork]

There have been 47 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-tsan-qr

This seems to have been affecting security/sandbox/test a lot recently. Specifically, it's trying to run tests from that directory on Linux ASan, which currently doesn't build sandboxing due to a conflict with LSan (more specifically: logic in old-configure doesn't set MOZ_SANDBOX, so toolkit.mozbuild doesn't add security/sandbox to DIRS; I don't know if the fact that it's using autoconf instead of pure Python is important, but I thought I should mention it). So those tests can't run on that build type, and that combination of build type and test manifest shouldn't be considered selectable.

(It seems that most of the sheriffs were aware of this bug and directed failures to it, and as a result I didn't know there was a problem even though patches I wrote or reviewed were being affected; but then bug 1744203 was backed out, and now I do know this bug exists.)

I've looked at a few of the failures linked in this bug, and they seem to follow the same general pattern: the test selection is asking for tests that don't exist for that platform or build type, but do exist for some other kind of build.

This is a constant annoyance, it's causing good patches to get backed out causing extra work for everyone, it's very confusing to developers and many fail to diagnose what's going on (https://bugzilla.mozilla.org/show_bug.cgi?id=1760977#c6). At this point it seems to me that this is quite a bit more than an S?/P5 bug.

Severity: normal → S2
Priority: P5 → P3

There have been 51 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-asan-qr
  • linux1804-64-qr
  • linux1804-64-tsan-qr
Flags: needinfo?(ahal)
Whiteboard: [stockwell unknown] → [stockwell unknown][stockwell needswork:owner]

Problem

So the problem here stems from moz.build and the fact that there are multiple ways to traverse them.

First is the default DIRS traversal. In this mode, a moz.build file can include other moz.build files via the DIRS variable. Sometimes, we only traverse down into a directory based on some condition of the build system (e.g, if we're building on Windows).

The second is a file-system traversal. In this mode, DIRS is ignored and we walk the entire source directory scanning for moz.build files. In this mode, every moz.build is processed even if it would have been ignored due to some build attribute in a DIRS traversal.

Both modes have their uses. The former is used by the build system since we don't want to process moz.build variables that are irrelevant to the build we're currently performaing. Taskgraph uses the latter because it needs to be able to find all tests, even if their manifests would not be included by any moz.build files on Linux (the platform Taskgraph runs on).

The problem is that these two traversal modes result in a mismatch.

For example, I looked at a recent log and saw that this happened for the manifest toolkit/components/aboutthirdparty/tests/browser/browser.ini. That manifest is included by toolkit/components/aboutthirdparty/moz.build. That moz.build file in turn is conditionally included by toolkit/components/moz.build, only if the toolkit is Windows. Sure enough, the failing task was on a Linux build.

So what ended up happening is:

  1. The Linux build did a DIRS traversal, never processed aboutthirdparty/moz.build and therefore those tests don't exist as far as it is concerned. So they never got copied to the test package artifact.
  2. In the Decision task, Taskgraph did a file system traversal, found toolkit/components/aboutthirdparty/tests/browser/browser.ini and scheduled a test task to run it.
  3. The test task starts, downloads the test package from the build and then says "Hey, the manifest I was supposed to run isn't here!"

Solutions

The easiest solution is to not define test manifests in moz.build files that get conditionally included in a DIRS traversal. In the example above, we would move this line into toolkit/components/moz.build and then add:

[DEFAULT]
skip-if = os != "win"

to the manifest. Though this is just a bandaid as we'll see this happen again and again anytime a new moz.build is conditionally included by DIRS. Longer term solutions might include:

  1. Have the build do a file-system traversal when packaging tests (I'm not sure if this is a separate step or not).
  2. Don't include test manifests from moz.build (maybe there could be some central location they get registered intsead).
  3. Ensure manifests are not included in conditional moz.build files via some CI check that fails with an explanatory error message.

Though none of those options seem trivial. There may be other alternatives I'm not thinking of.

Flags: needinfo?(ahal)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

There have been 68 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-asan-qr
  • linux1804-64-qr
  • linux1804-64-tsan-qr
[task 2022-11-25T15:55:01.006Z] 15:55:01     INFO - Checking for ssltunnel processes...
[task 2022-11-25T15:55:01.007Z] 15:55:01     INFO - Checking for xpcshell processes...
[task 2022-11-25T15:55:01.274Z] 15:55:01    ERROR - No tests were found for flavor 'plain' and the following manifest filters:
[task 2022-11-25T15:55:01.275Z] 15:55:01    ERROR - skip_if, run_if, fail_if, subsuite(name=None), pathprefix(['mobile/android/components/extensions/test/mochitest/mochitest.ini'])
[task 2022-11-25T15:55:01.275Z] 15:55:01    ERROR - 
[task 2022-11-25T15:55:01.276Z] 15:55:01    ERROR - Make sure the test paths (if any) are spelt correctly and the corresponding
[task 2022-11-25T15:55:01.277Z] 15:55:01    ERROR - --flavor and --subsuite are being used. See `mach mochitest --help` for a
[task 2022-11-25T15:55:01.277Z] 15:55:01    ERROR - list of valid flavors.
[task 2022-11-25T15:55:01.277Z] 15:55:01    ERROR - 
[task 2022-11-25T15:55:01.278Z] 15:55:01     INFO - SUITE-START | Running 0 tests
[task 2022-11-25T15:55:01.279Z] 15:55:01     INFO -  0 INFO TEST-START | Shutdown
[task 2022-11-25T15:55:01.280Z] 15:55:01     INFO -  1 INFO Passed:  0
[task 2022-11-25T15:55:01.281Z] 15:55:01     INFO -  2 INFO Failed:  0
[task 2022-11-25T15:55:01.281Z] 15:55:01     INFO -  3 INFO Todo:    0
[task 2022-11-25T15:55:01.282Z] 15:55:01     INFO -  4 INFO Mode:    e10s
[task 2022-11-25T15:55:01.282Z] 15:55:01     INFO -  5 INFO SimpleTest FINISHED
[task 2022-11-25T15:55:01.283Z] 15:55:01     INFO - Buffered messages finished
[task 2022-11-25T15:55:01.283Z] 15:55:01     INFO - SUITE-END | took 0s
Flags: needinfo?(ahal)
Whiteboard: [stockwell unknown] → [stockwell needswork:owner]

Hm, I'm not really sure what's going on here. The manifests that are failing with this seem to not have any skip-ifs in the DEFAULT section. So the failure condition for many of these seems to be something other than comment 89.

Flags: needinfo?(ahal)

There have been 52 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-asan-qr
  • linux1804-64-qr
[task 2023-02-24T03:17:21.682Z] 03:17:21     INFO - Checking for ssltunnel processes...
[task 2023-02-24T03:17:21.683Z] 03:17:21     INFO - Checking for xpcshell processes...
[task 2023-02-24T03:17:21.999Z] 03:17:21    ERROR - No tests were found for flavor 'plain' and the following manifest filters:
[task 2023-02-24T03:17:21.999Z] 03:17:21    ERROR - skip_if, run_if, fail_if, subsuite(name=None), pathprefix(['mobile/android/components/extensions/test/mochitest/mochitest.ini'])
[task 2023-02-24T03:17:21.999Z] 03:17:21    ERROR - 
[task 2023-02-24T03:17:21.999Z] 03:17:21    ERROR - Make sure the test paths (if any) are spelt correctly and the corresponding
[task 2023-02-24T03:17:21.999Z] 03:17:21    ERROR - --flavor and --subsuite are being used. See `mach mochitest --help` for a
[task 2023-02-24T03:17:21.999Z] 03:17:21    ERROR - list of valid flavors.
[task 2023-02-24T03:17:21.999Z] 03:17:21    ERROR - 
[task 2023-02-24T03:17:22.003Z] 03:17:22     INFO - SUITE-START | Running 0 tests
[task 2023-02-24T03:17:22.004Z] 03:17:22     INFO -  0 INFO TEST-START | Shutdown
[task 2023-02-24T03:17:22.004Z] 03:17:22     INFO -  1 INFO Passed:  0
[task 2023-02-24T03:17:22.004Z] 03:17:22     INFO -  2 INFO Failed:  0
[task 2023-02-24T03:17:22.004Z] 03:17:22     INFO -  3 INFO Todo:    0
[task 2023-02-24T03:17:22.005Z] 03:17:22     INFO -  4 INFO Mode:    e10s
[task 2023-02-24T03:17:22.006Z] 03:17:22     INFO -  5 INFO SimpleTest FINISHED
[task 2023-02-24T03:17:22.006Z] 03:17:22     INFO - Buffered messages finished
[task 2023-02-24T03:17:22.006Z] 03:17:22     INFO - SUITE-END | took 0s
[task 2023-02-24T03:17:22.027Z] 03:17:22     INFO - Return code: 1
[task 2023-02-24T03:17:22.028Z] 03:17:22    ERROR - No checks run.

There have been 60 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-asan-qr
  • linux1804-64-qr
  • windows10-64-2004-qr
Duplicate of this bug: 1848439
Whiteboard: [stockwell disable-recommended] → [stockwell infra]

Not sure if that helps for analysis, but I see this always failing on this try push. It seems very unlikely my few changes had any impact there.

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: REOPENED → RESOLVED
Closed: 5 years ago → 1 year ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=479395184&repo=autoland

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: REOPENED → RESOLVED
Closed: 1 year ago → 1 year ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=485334777&repo=autoland

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

Downgrading this bug to S3 as it doesn't fit the definition of S2:

Major functionality/product severely impaired or a high impact issue and a satisfactory workaround does not exist

Severity: S2 → S3

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: REOPENED → RESOLVED
Closed: 1 year ago → 1 year ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=515246357&repo=autoland

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Duplicate of this bug: 1560298

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: REOPENED → RESOLVED
Closed: 1 year ago → 2 months ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.