<a class="header-button" href="https://bugzilla.mozilla.org/home" title="Go to home page"> Bugzilla

Comment 4

•

3 years ago

May be related to the fix landed for bug 1753019

Comment 5

•

3 years ago

•

Edited

(In reply to Dan Robertson (:dlrobertson) from comment #4)

May be related to the fix landed for bug 1753019

I don't think so (though I suppose it's hard to tell with this new approach of lumping failures from all subtests into a single bug).

I think it's just the issue we diagnosed in bug 1737725 comment 11, which was previously tracked in several bugs (basically, one per message which usually meant one per subtest) including bug 1737725, and is now tracked in this "single tracking bug" for the test group.

We do plan to work on a fix for this underlying issue in 2022 H2 (it's in our Stabilization Sprint 2a). I'll assign this bug to myself to reflect that.

I hope the sheriffs will afford us some leniency until that fix is in place (since this aggregation method now means the intermittent failure rate is above the disable-recommended threshold of 30/week, but nothing actually changed/regressed, we're just grouping more things into a single bug).

Assignee: nobody → botond

Flags: needinfo?(botond)

Comment hidden (Intermittent Failures Robot)

Marian-Vasile Laza

Comment 8

•

3 years ago

There have been 31 total failures in the last 7 days, recent failure log.
Affected platforms are:

linux1804-64-qr
linux1804-64-shippable-qr
macosx1015-64-qr
windows10-64-2004-qr
windows10-64-2004-shippable-qr

Comment hidden (Intermittent Failures Robot)

Marian-Vasile Laza

Comment 18

•

3 years ago

There have been 33 total failures in the last 7 days, recent failure log.
Affected platforms are:

linux1804-64-qr
linux1804-64-shippable-qr
macosx1015-64-qr
windows10-64-2004-ccov-qr
windows10-64-2004-qr
windows10-64-2004-shippable-qr

Comment hidden (Intermittent Failures Robot)

Andreea Pavel [:apavel]

Comment 20

•

3 years ago

Attached file Bug 1776972 - update disabling conditions for test_group_hittest-2.html r?#intermittent-reviewers — Details

Andreea Pavel [:apavel]

Updated

•

2 years ago

Keywords: leave-open

Whiteboard: [stockwell needswork:owner] → [stockwell needswork:owner][stockwell disabled]

Pulsebot

Comment 21

•

2 years ago

Pushed by apavel@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/ae892505fca1 update disabling conditions for test_group_hittest-2.html r=intermittent-reviewers,jmaher DONTBUILD

Comment hidden (Intermittent Failures Robot)

https://hg.mozilla.org/mozilla-central/rev/ae892505fca1

Comment 23

•

2 years ago

bugherder

Comment hidden (Intermittent Failures Robot)

Joel Maher ( :jmaher ) (UTC -8)

Comment 25

•

2 years ago

It looks like this is perma/High frequency, might be similar with Bug 1785026
Can you please take a look at it? log

Flags: needinfo?(drobertson)

Comment 26

•

2 years ago

Attached file summary of different failure modes — Details

```

Reporter

Comment 27

•

2 years ago

if you run ./mach test-info failure-report --bug 1776972, here are some of the big hits:

Querying data for bug [1776972, 1737725, 1709108, 1737815, 1738440, 1742356, 1743071, 1765516, 1771158, 1774395] annotated from 2022-07-17 to 2022-08-16 on trunk.

11 errors with:
13:13:19     INFO - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_hittest-2.html | Fixed ancestor should not be hit
  macosx1015-64-shippable-qr/opt-mochitest-plain: 5
  macosx1015-64-shippable-qr/opt-mochitest-plain-spi-nw-spi-nw: 2
  macosx1015-64-qr/opt-mochitest-plain-spi-nw-spi-nw: 2
  macosx1015-64-qr/opt-mochitest-plain: 2

22 errors with:
10:22:10     INFO - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_hittest-2.html | helper_hittest_bug1730606-2.html | root scroller (after second async scroll) scrollid - got 113, expected 111
  linux1804-64-qr/debug-mochitest-plain-spi-nw-spi-nw: 2
  windows10-64-2004-qr/debug-mochitest-plain-xorig-xorig: 9
  macosx1015-64-qr/debug-mochitest-plain-xorig-xorig: 10
  linux1804-64-shippable-qr/opt-mochitest-plain-nofis-nofis: 1

27 errors with:
11:06:09     INFO - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_hittest-2.html | helper_hittest_iframe_perspective-3.html | x coordinate is correct - got 150, expected 100
11:06:09     INFO - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_hittest-2.html | helper_hittest_iframe_perspective-3.html | y coordinate is correct - got 150, expected 100
  linux1804-64-shippable-qr/opt-mochitest-plain-headless-spi-nw-spi-nw: 7
  windows10-64-2004-shippable-qr/opt-mochitest-plain-xorig-xorig: 8
  linux1804-64-shippable-qr/opt-mochitest-plain-headless-headless: 2
  linux1804-64-qr/opt-mochitest-plain-xorig-xorig: 1
  linux1804-64-ccov-qr/opt-mochitest-plain-xorig-xorig: 1
  windows10-64-2004-ccov-qr/opt-mochitest-plain: 3
  windows10-64-2004-qr/opt-mochitest-plain-xorig-xorig: 2
  windows10-64-2004-qr/opt-mochitest-plain: 1
  linux1804-64-qr/opt-mochitest-plain-spi-nw-spi-nw: 1
  linux1804-64-qr/opt-mochitest-plain: 1

12 errors with:
03:06:45     INFO - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_hittest-2.html | helper_hittest_bug1730606-2.html | root scroller (after async scroll) scrollid - got 49, expected 47
03:06:45     INFO - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_hittest-2.html | helper_hittest_bug1730606-2.html | layer3 (after async scroll) scrollid - got 48, expected 49
  linux1804-64-shippable-qr/opt-mochitest-plain-headless-spi-nw-spi-nw: 1
  windows10-64-2004-qr/debug-mochitest-plain-xorig-xorig: 5
  linux1804-64-shippable-qr/opt-mochitest-plain-headless-headless: 1
  macosx1015-64-qr/debug-mochitest-plain-xorig-xorig: 5

20 errors with:
17:33:44     INFO - TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_hittest-2.html | Test timed out. -
  linux1804-64-shippable-qr/opt-mochitest-plain-headless-spi-nw-spi-nw: 7
  linux1804-64-shippable-qr/opt-mochitest-plain-headless-headless: 9
  linux1804-64-qr/opt-mochitest-plain-xorig-xorig: 2
  linux1804-64-ccov-qr/opt-mochitest-plain-xorig-xorig: 2

there are a few other failure messages, but those have <5 instances in the last month.

Comment 28

•

2 years ago

(In reply to Cristian Tuns from comment #25)

It looks like this is perma/High frequency, might be similar with Bug 1785026
Can you please take a look at it? log

Thanks for the logs! I'm new to how intermittent failures are tracked, but bug 1785026 was cased by bug 1782921, which landed just a few hours ago, so I think the bugs are a little different. I'm currently investigating bug 1785026.

Updated

•

2 years ago

Flags: needinfo?(drobertson)

Comment 29

•

2 years ago

The bugs are different but might have the same regression Bug (Bug 1782921) that's why I mentioned the tier 2 bug.

Comment 30

•

2 years ago

(In reply to Cristian Tuns from comment #29)

The bugs are different but might have the same regression Bug (Bug 1782921) that's why I mentioned the tier 2 bug.

I may be missing something, but the regression bug (bug 1782921) landed just a few hours ago, well after test_group_hittest-2.html became perma (bug 1776972 comment 23). I'd guess bug 1776972 comment 5 and bug 1737725 comment 11 still point to the cause of this intermittent.

Comment 31

•

2 years ago

•

Edited

This bug's original line is this: TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_hittest-2.html | helper_hittest_bug1730606-2.html | root scroller (after async scroll) scrollid - got 213, expected 211 but because of the new thing all the bugs are merged into one with the file name | single tracking bug but now the curent failure line is: TEST-UNEXPECTED-FAIL | gfx/layers/apz/test/mochitest/test_group_hittest-2.html | Fixed ancestor should not be hit .
"Fixed ancestor should not be hit " just like Bug 1785026.
I think you can Ignore the original fail, because this is different.
Aryx when you have the time can you help me with a more detailed explanation for Dan?

Updated

•

2 years ago

Flags: needinfo?(aryx.bugmail)

Comment 32

•

2 years ago

•

Edited

(In reply to Dan Robertson (:dlrobertson) from comment #30)

(In reply to Cristian Tuns from comment #29)

The bugs are different but might have the same regression Bug (Bug 1782921) that's why I mentioned the tier 2 bug.

I may be missing something, but the regression bug (bug 1782921) landed just a few hours ago, well after test_group_hittest-2.html became perma (bug 1776972 comment 23).

Comment 23 is when this test was disabled on some platforms, because its intermittent failure rate was too high (but not close to "perma").

The sharp increase in failure rate on remaining platforms (going "perma") is more recent and does coincide with bug 1782921 landing. Basically, it's the same issue as bug 1785026, it's just that for some reason, some failures contain "unknown test url" instead of "test_group_hittest-2.html" in the failure line and so get associated with bug 1785026 rather than this one.

Flags: needinfo?(aryx.bugmail)

Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)

Comment 33

•

2 years ago

(In reply to Botond Ballo [:botond] from comment #32)

for some reason, some failures contain "unknown test url" instead of "test_group_hittest-2.html" in the failure line

(May be related to the fact that helper_hittest_fixed-3 includes SimpleTest.js directly, which is known to cause problems with printing the test name.)

Comment hidden (Intermittent Failures Robot)

Updated

•

2 years ago

Summary: Perma gfx/layers/apz/test/mochitest/test_group_hittest-2.html | single tracking bug → Intermittent gfx/layers/apz/test/mochitest/test_group_hittest-2.html | single tracking bug

Comment hidden (Intermittent Failures Robot)

Narcis Beleuzu [:NarcisB]

Comment 64

•

2 years ago

Attached file Bug 1776972 - Re-enable test_group_hittest-2.html in some configurations after the fix for bug 1737725. r=hiro — Details

This patch reverts the test annotations to their state before D153415.

Phabricator Automation

Updated

•

2 years ago

Attachment #9325809 - Attachment description: Bug 1776972 - Re-enable test_group_hittest-2.html in some configurations after the fix for bug 1737725. r=dlrobertson → Bug 1776972 - Re-enable test_group_hittest-2.html in some configurations after the fix for bug 1737725. r=hiro

Pulsebot

Comment 65

•

2 years ago

Pushed by bballo@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/da24fef388ad Re-enable test_group_hittest-2.html in some configurations after the fix for bug 1737725. r=hiro

Comment 66

•

2 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/da24fef388ad

Comment hidden (Intermittent Failures Robot)

Cristina Horotan [:chorotan]

Comment 68

•

2 years ago

There have been 50 total failures in the last 7 days.
There are:

2 failures on linux1804-64-ccov-qr opt
2 failures on linux1804-64-qr opt
2 failures on linux1804-64-shippable-qr opt
1 failure on windows10-64-2004-qr debug
2 failures on windows11-64-2009-ccov-qr opt
17 failures on windows11-64-2009-qr opt
24 failures on windows11-64-2009-shippable-qr opt

Recent failure log.

Botond, as the owner of this component, can you help us assign the bug to someone? Thank you.

Flags: needinfo?(botond)

Whiteboard: [stockwell disabled][stockwell unknown] → [stockwell disabled][stockwell needswork:owner]

Comment 69

•

2 years ago

•

Edited

Looking at which subtests are failing, out of 52 failures over the past week, 23 are in helper_hittest_iframe_perspective-3.html, and 22 are in helper_hittest_iframe_perspective.html, with the remaining 7 being spread out across various other subtests at a low volume.

Given that, my plan is to do the following:

Quarantine the two affected subtests in a new test group, which is disabled on windows11
File a follow-up to track investigating the high failure rate of those two subtests on windows11 and track re-enabling them on that platform

Flags: needinfo?(botond)

Comment 70

•

2 years ago

(In reply to Botond Ballo [:botond] from comment #69)

Quarantine the two affected subtests in a new test group, which is disabled on windows11

Filed bug 1829010 for this. (I realized I probably shouldn't land patches in this bug, since this is the "single tracking bug" for the test group and as such it will stay open anyways to track any remaining failures in the test group. I will land any patches in separate bugs instead.)

Updated

•

2 years ago

Depends on: 1829010

Comment 71

•

2 years ago

(In reply to Botond Ballo [:botond] from comment #69)

File a follow-up to track investigating the high failure rate of those two subtests on windows11 and track re-enabling them on that platform

Filed bug 1829021 for this.

Depends on: 1829021

Comment hidden (Intermittent Failures Robot)