Closed Bug 1961446 Opened 29 days ago Closed 6 days ago

Frequent [tier 2] Android Pixel5 R-nofis-cf TEST-UNEXPECTED-PASS | svg/sizing/standalone--0-0--0-px.svg == svg/sizing/pass-empty.svg | image comparison, max difference: 0, number of differing pixels: 0

Categories

(Core :: SVG, defect)

Product:
Core
Component:
SVG
Type:
defect

Tracking

()

Status:
RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

Details

(Keywords: intermittent-failure, Whiteboard: [collect_confirm_failure])

This appeared on mozilla central and when it came first it looked like it's perma but after some time the retriggers turned green, as it can be seen here. Fwiw, it looks like these tests were not supposed to run on R-nofis-cf but it's interesting that, eventually, they turned green.

Hi Joel! Could you please help me figuring out what happened here? Also, you can close this bug if you consider so.

Thank you!

Severity: S4 → --
Flags: needinfo?(jmaher)
Priority: P5 → --
Summary: Perma [tier 2] Android Pixel5 R-nofis-cf TEST-UNEXPECTED-PASS | svg/sizing/standalone--0-0--0-px.svg == svg/sizing/pass-empty.svg | image comparison, max difference: 0, number of differing pixels: 0 → Frequent [tier 2] Android Pixel5 R-nofis-cf TEST-UNEXPECTED-PASS | svg/sizing/standalone--0-0--0-px.svg == svg/sizing/pass-empty.svg | image comparison, max difference: 0, number of differing pixels: 0

thanks for filing this. when a new intermittent bug is filed from CI, it will now launch a confirm failure run. This technically launches a confirm failure for each unique test and it did, in this case the jobs ran as "unexpected-pass"- this I need to look into. confirm failure is a lot like test-verify, it doesn't work in all situations. I wonder if:

  • reftests work in test-verify properly
  • test-verify works on android, specifically android-hw (pretty sure this is supported in general)
  • confirm-failure for reftest works on android-hardware
  • some specific tests don't work in test-verify/confirm-failure mode; we need to assess and annotate those again- it has been many years since we have done that exercise.

The retriggers of the confirm-failure job didn't run the single test instead it ran the entire job- when the retrigger happened it didn't include the environment variable MOZHARNESS_CONFIRM_PATHS. When we retrigger we call create_task which creates the task from the decision task. In the case of confirm failure, we start with the task, but add in extra env vars here.

So I believe the primary fix here is to ensure that retriggering a confirm-failure -cf task will have all the proper env vars.

Flags: needinfo?(jmaher)

I verified that:

  • confirm failure works on reftests (all platforms)
  • confirm failure works on android-hardware
  • confirm failure works on android-hw + reftest

This specific case where the confirm failure is generating "unexpected pass" is a case where running standalone results in different actions than running in a set of tests. typically we annotate with skip-if = ["verify"], but for reftest there isn't currently supported but could be.

In this case, there are 2 action items:

  1. support skip-if(verify) for reftest.list
  2. ensure when retriggering confirm-failure tasks that we include the env var MOZHARNESS_CONFIRM_PATHS

For the purpose of this bug, there are no action items I can see for test owners.

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: NEW → RESOLVED
Closed: 6 days ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.