1913649 - Test marked `run-if = ["release_or_beta"]` runs when `release_or_beta: False`

Status: RESOLVED FIXED

(Testing :: Mochitest, defect, P1)

Description

[Jim Blandy :jimb]

The mochitest dom/webgpu/mochitest/test_disabled.html runs on ordinary try pushes from central, even though the only manifest that lists it is dom/webgpu/mochitest/mochitest-no-pref.toml, which says run-if = ["release_or_beta"].

For example, see this failure. Note that the log shows:

[task 2024-08-18T18:38:17.448Z] 18:38:17     INFO -     release_or_beta: False

Comment 1

[Jim Blandy :jimb]

I can reproduce this locally, if, instead of:

$ mach mochitest dom/webgpu

I run the command:

$ mach mochitest dom/webgpu/mochitest/mochitest-no-pref.toml 

I think the latter is closer to what TaskCluster is running.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jmaher, could you have a look please?

For more information, please visit BugBot documentation.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I am looking into this today, I was more on PTO the last 2 weeks. Thanks for filing this!

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

this appears to be a problem when we run a single manifest, now to find the code that avoids run-if/skip-if on a single manifest (or maybe a single test)

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

most like it is because of this clause:

            if len(tests) == 1 and "disabled" in test:
                del test["disabled"]

this is in "by design" so one could run a test locally and ignore the fact that it is disabled/skipped.

Some solutions might be:

1. find a way to ignore this rule while on try
2. determine why this entire task only has a single test, that doesn't seem effective
3. figure out how to filter the manifest out at task selection time and not schedule the task if there are no tests.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I have a fix for bug 1914635, this solves the problem- we ended up in the decision task having the wrong value for release_or_beta, so the decision task was picking webgpu manifests that matched release_or_beta and ignoring the rest.

so the solutions above, #3 is happening properly- I think solving for #1 would be a good path forward.

Comment 7

[Intermittent Failures Robot]

30 failures in 3668 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• try: 30

Platform and build breakdown:

• linux2204-64-wayland: 10
  • opt: 5
  • debug: 5
• macosx1015-64-qr: 10
  • debug: 5
  • opt: 5
• windows11-64-2009-qr: 10
  • debug: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1913649&startday=2024-09-02&endday=2024-09-08&tree=all

Comment 8

[Intermittent Failures Robot]

36 failures in 4448 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• try: 36

Platform and build breakdown:

• macosx1015-64-qr: 10
  • debug: 5
  • opt: 5
• windows11-64-2009-qr: 11
  • debug: 6
  • opt: 5
• linux2204-64-wayland: 15
  • opt: 5
  • debug: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1913649&startday=2024-09-09&endday=2024-09-15&tree=all

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

still waiting on review in bug 1914635