1878425 - test-macosx1100-64-shippable-qr/opt-reftest-* tasks occasionally somehow run in a mode where pixel-rounding seems to be different, which results in tons of reftest failures

Status: NEW

(Core :: Layout, task)

Description

[Daniel Holbert [:dholbert]]

Some of the time, it appears our test-macosx1100-64-shippable-qr/opt-reftest-* tasks end up with a huge number of unexpected reftest failures (and some unexpected reftest passes), where the common theme among the failures/passes seems to be pixel-scaling and/or pixel rounding.

The massive number of failures also ends up causing the logs to exceed the treeherder log-parser size limit, so they don't show up on treeherder in a way that makes it easy to visualize the failures.

This happened on 3 distinct reftest runs today on mozilla-release:
https://firefoxci.taskcluster-artifacts.net/X9u7fC_5QfWNC5KvDSd51g/0/public/logs/live_backing.log
https://firefoxci.taskcluster-artifacts.net/cOhrtf5QSbCAq6GYurVwuw/0/public/logs/live_backing.log
https://firefoxci.taskcluster-artifacts.net/Li7SCFDKTby_a0i2yNfVrQ/0/public/logs/live_backing.log

I'm not seeing any other recent instances, but let's see if it continues to happen vs. if it was a one-off...

Comment 1

[Daniel Holbert [:dholbert]]

(Note, sheriffs filed bug 1878387 and bug 1878389 for the three failure-logs linked in comment 0 here.)

Comment 2

[Daniel Holbert [:dholbert]]

Attachment: gzipped log for X9u7fC url

Comment 3

[Daniel Holbert [:dholbert]]

Attachment: gzipped log for cOhrtf url

Comment 4

[Daniel Holbert [:dholbert]]

Attachment: gzipped log for Li7SCF url

Comment 5

[Daniel Holbert [:dholbert]]

I attached the gzipped logs that I linked in comment 0, for archival in case they disappear from taskcluster.

Comment 9

[Intermittent Failures Robot]

9 failures in 4009 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 4
• mozilla-release: 2
• mozilla-esr115: 1
• mozilla-central: 1
• try: 1

Platform and build breakdown:

• macosx1100-64-shippable-qr: 8
  • opt: 8
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1878425&startday=2024-02-05&endday=2024-02-11&tree=all

Comment 10

[Intermittent Failures Robot]

2 failures in 3125 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• try: 1

Platform and build breakdown:

• macosx1100-64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1878425&startday=2024-02-12&endday=2024-02-18&tree=all

Comment 11

[Intermittent Failures Robot]

4 failures in 3382 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-beta: 1
• mozilla-esr115: 1
• mozilla-central: 1

Platform and build breakdown:

• macosx1100-64-shippable-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1878425&startday=2024-02-19&endday=2024-02-25&tree=all

Comment 13

[Intermittent Failures Robot]

2 failures in 4052 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-central: 1

Platform and build breakdown:

• macosx1100-64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1878425&startday=2024-02-26&endday=2024-03-03&tree=all

Comment 15

[Daniel Holbert [:dholbert]]

FWIW, we accumulated a bunch of mistaken annotations in the past ~day that for logs like this one with a single reftest unexpected-pass on macosx1015, for a specific test that started passing due to a bug fix that landed recently (hooray). That expected-fail annotation was removed in Bug 1863421 comment 11 to reflect reality, so the unexpected passes should stop now.

Those unexpected passes were unrelated to the actual bug here -- so, that spike in reports here shouldn't be taken as any indication that this bug is suddenly getting more frequent than it previously was.

Comment 16

[Intermittent Failures Robot]

30 failures in 4198 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-release: 1
• mozilla-central: 4
• mozilla-beta: 4
• autoland: 16
• try: 5

Platform and build breakdown:

• macosx1100-64-shippable-qr: 8
  • opt: 8
• macosx1015-64-qr: 19
  • opt: 5
  • debug: 14
• macosx1015-64-shippable-qr: 2
  • opt: 2
• macosx1015-64-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1878425&startday=2024-03-04&endday=2024-03-10&tree=all

Comment 17

[Intermittent Failures Robot]

6 failures in 3791 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-central: 1
• mozilla-beta: 4

Platform and build breakdown:

• macosx1100-64-shippable-qr: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1878425&startday=2024-03-18&endday=2024-03-24&tree=all

Comment 18

[Intermittent Failures Robot]

1 failures in 3590 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• macosx1100-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1878425&startday=2024-03-25&endday=2024-03-31&tree=all

Comment 19

[Intermittent Failures Robot]

1 failures in 4577 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1878425&startday=2024-06-17&endday=2024-06-23&tree=all