1858041 - [meta] Multiple tier 2 failures happening on a11y-checks

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect, P2)

Description

[Norisz Fay [:noriszfay]]

Multiple accessibility tier 2 failures caused by the following push on M-a11y-checks & M-swr-a11y-checks with some of them being perma.

Failures seen here.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Disability Access APIs' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Intermittent Failures Robot]

39 failures in 618 pushes (0.063 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 39

Platform and build breakdown:

• linux1804-64-qr: 31
  • opt: 31
• linux1804-64-shippable-qr: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-09&endday=2023-10-09&tree=all

Comment 3

[Anna Yeddi [:ayeddi]]

Attachment: Bug 1858041 - Temporarily fail-if some Tier 2 a11y_checks failing on Autoland. r=Jamie

We are working on enabling Tier 2 a11y-checks to ensure our products have basic accessibility built-in (bug 1692110) and before that, we need to prepare the existing code base. While we triage and investigate existent tests that would fail once the a11y-checks are enabled in the CI, we are going to expect to be failing these tests altogether and then handle each one of them individually.

This task is to expect failing of these tests with fail-if condition of the failing tests while we continue investigation into the reasons these tests failed and to be able to backtrack these tests once they're resolved.

Related bugs:

• Fail-if tests: meta bug 1848402
• Explore capturing click events for a11y_checks: bug 1692110

Comment 4

[Intermittent Failures Robot]

195 failures in 760 pushes (0.257 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 165
• mozilla-central: 30

Platform and build breakdown:

• linux1804-64-qr: 77
  • opt: 73
  • debug: 4
• linux1804-64-shippable-qr: 107
  • opt: 107
• windows11-64-2009-qr: 1
  • opt: 1
• linux1804-64-tsan-qr: 4
  • opt: 4
• linux1804-64-asan-qr: 2
  • opt: 2
• linux1804-64-ccov-qr: 3
  • opt: 3
• macosx1015-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-10&endday=2023-10-10&tree=all

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9dc8d4452ed5
Temporarily fail-if some Tier 2 a11y_checks failing on Autoland. r=Jamie,devtools-reviewers,mixedpuppy,nchevobbe

Comment 6

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/9dc8d4452ed5

Comment 13

[Norisz Fay [:noriszfay]]

hello Anna, could you take a look at these remaining tier 2 bugs that still linger on autoland? Thank you!

Comment 14

[Intermittent Failures Robot]

86 failures in 768 pushes (0.112 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 82
• mozilla-central: 4

Platform and build breakdown:

• linux1804-64-qr: 47
  • opt: 46
  • debug: 1
• linux1804-64-shippable-qr: 38
  • opt: 38
• macosx1015-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-11&endday=2023-10-11&tree=all

Comment 18

[Anna Yeddi [:ayeddi]]

(In reply to Norisz Fay [:noriszfay] from comment #13)

hello Anna, could you take a look at these remaining tier 2 bugs that still linger on autoland? Thank you!

Will write a patch today, thank you for the link!

Comment 20

[Anna Yeddi [:ayeddi]]

Attachment: Bug 1858041 - Temporarily skip of fail-if some Tier 2 a11y_checks failing or crashing on Autoland. r=Jamie

There are some a11y_checks that are intermittently failing on Autoland and pass on Try and vice versa. Also, a couple of tests are crashing while performing a11y_checks, thus they are to be skipped while we continue the investigation into each of them.

Related bugs:

• Fail-if tests: meta bug 1848402
• Explore capturing click events for a11y_checks: bug 1692110

Comment 21

[Intermittent Failures Robot]

85 failures in 766 pushes (0.111 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 81
• mozilla-central: 4

Platform and build breakdown:

• linux1804-64-qr: 34
  • opt: 34
• linux1804-64-shippable-qr: 51
  • opt: 51

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-12&endday=2023-10-12&tree=all

Comment 22

[Intermittent Failures Robot]

101 failures in 638 pushes (0.158 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 22
• autoland: 79

Platform and build breakdown:

• linux1804-64-shippable-qr: 61
  • opt: 61
• linux1804-64-qr: 40
  • opt: 38
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-13&endday=2023-10-13&tree=all

Comment 23

[Intermittent Failures Robot]

43 failures in 116 pushes (0.371 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 36
• mozilla-central: 7

Platform and build breakdown:

• linux1804-64-qr: 18
  • opt: 18
• linux1804-64-shippable-qr: 25
  • opt: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-14&endday=2023-10-14&tree=all

Comment 24

[Intermittent Failures Robot]

618 failures in 3764 pushes (0.164 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 535
• mozilla-central: 83

Platform and build breakdown:

• linux1804-64-qr: 272
  • opt: 265
  • debug: 7
• linux1804-64-shippable-qr: 333
  • opt: 333
• windows11-64-2009-qr: 1
  • opt: 1
• linux1804-64-tsan-qr: 4
  • opt: 4
• linux1804-64-asan-qr: 2
  • opt: 2
• linux1804-64-ccov-qr: 3
  • opt: 3
• macosx1015-64-qr: 2
  • opt: 2
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-09&endday=2023-10-15&tree=all

Comment 25

[Intermittent Failures Robot]

50 failures in 631 pushes (0.079 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 42
• mozilla-central: 8

Platform and build breakdown:

• linux1804-64-qr: 19
  • opt: 19
• linux1804-64-shippable-qr: 31
  • opt: 31

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-16&endday=2023-10-16&tree=all

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/746ba32c4a02
Temporarily skip of fail-if some Tier 2 a11y_checks failing or crashing on Autoland. r=Jamie,fxview-reviewers,ochameau,devtools-reviewers,places-reviewers,mixedpuppy

Comment 27

[Cristian Tuns]

Hi, if the patch is supposed to fix the issues, I think it is not working: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=G_PNoAaqQUGar4gJoCafLQ.0&revision=4091bed4a0aae7772537e08e6cb49bf0bc7434c9&searchStr=Linux%2C18.04%2Cx64%2CWebRender%2Copt%2CMochitests%2Cwith%2Caccessibility%2Cchecks%2Cenabled%2Ctest-linux1804-64-qr%2Fopt-mochitest can you please take another look?

Comment 28

[Anna Yeddi [:ayeddi]]

Attachment: Bug 1858041 - Pt.3. Temporarily skip some Tier 2 a11y_checks for unexpecredly failing/passing or crashing on Autoland. r=Jamie

There are some a11y_checks that are intermittently failing on Autoland and pass on Try and vice versa. Also, a couple of tests are crashing while performing a11y_checks, thus they are to be skipped while we continue the investigation into each of them.

Related bugs:

• Fail-if tests: meta bug 1848402
• Explore capturing click events for a11y_checks: bug 1692110

Comment 29

[James Teh [:Jamie]]

Hi Joel. Do you have any idea why a11y-checks jobs would be running on Mac (example)? As I understand it, they should only be running on Linux as tier 2 at this stage. The eventual goal is to run them on all platforms, but we're not there yet, so this was surprising to us.

Comment 30

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/746ba32c4a02

Comment 31

[Intermittent Failures Robot]

94 failures in 750 pushes (0.125 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 74
• mozilla-central: 20

Platform and build breakdown:

• linux1804-64-qr: 42
  • opt: 42
• linux1804-64-shippable-qr: 52
  • opt: 52

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-17&endday=2023-10-17&tree=all

Comment 32

[Joel Maher ( :jmaher ) (UTC -8)]

that was added with the add-new action, which allows you to add all kinds of things, even if they are not supported. In this case you could add it for linux, windows, mac, but not android because android doesn't run any browser-chrome tests.

There is no default scheduling or other ways to make this work, so I think you are ok.

Comment 33

[Intermittent Failures Robot]

26 failures in 848 pushes (0.031 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 25
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-qr: 12
  • opt: 12
• linux1804-64-shippable-qr: 14
  • opt: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-18&endday=2023-10-18&tree=all

Comment 34

[Intermittent Failures Robot]

63 failures in 800 pushes (0.079 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 13
• autoland: 50

Platform and build breakdown:

• linux1804-64-shippable-qr: 30
  • opt: 30
• linux1804-64-qr: 33
  • opt: 32
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-19&endday=2023-10-19&tree=all

Comment 35

[Intermittent Failures Robot]

48 failures in 306 pushes (0.157 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 40
• mozilla-central: 8

Platform and build breakdown:

• linux1804-64-qr: 19
  • opt: 18
  • debug: 1
• linux1804-64-shippable-qr: 29
  • opt: 29

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-20&endday=2023-10-20&tree=all

Comment 36

[Intermittent Failures Robot]

295 failures in 3484 pushes (0.085 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 242
• mozilla-central: 53

Platform and build breakdown:

• linux1804-64-qr: 132
  • opt: 130
  • debug: 2
• linux1804-64-shippable-qr: 163
  • opt: 163

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-16&endday=2023-10-22&tree=all

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f222bdcd6517
Pt.3. Temporarily skip some Tier 2 a11y_checks for unexpecredly failing/passing or crashing on Autoland. r=Jamie,extension-reviewers,devtools-reviewers,rpl,places-reviewers

Comment 38

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/f222bdcd6517

Comment 39

[Intermittent Failures Robot]

30 failures in 789 pushes (0.038 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 30

Platform and build breakdown:

• linux1804-64-qr: 22
  • opt: 22
• linux1804-64-shippable-qr: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-23&endday=2023-10-23&tree=all

Comment 41

[Noemi Erli[:noemi_erli]]

This is still happening a lot, here are some examples:

• https://treeherder.mozilla.org/logviewer?job_id=433684835&repo=autoland&lineNumber=23680
• https://treeherder.mozilla.org/logviewer?job_id=433693142&repo=autoland&lineNumber=36219
• https://treeherder.mozilla.org/logviewer?job_id=433690190&repo=autoland&lineNumber=9586
• https://treeherder.mozilla.org/logviewer?job_id=433690206&repo=autoland&lineNumber=21944
• https://treeherder.mozilla.org/logviewer?job_id=433666294&repo=autoland&lineNumber=51381
• https://treeherder.mozilla.org/logviewer?job_id=433666392&repo=autoland&lineNumber=13742
  Ana, could you please take a look?
  Thanks,

Comment 42

[Intermittent Failures Robot]

15 failures in 706 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 14
• try: 1

Platform and build breakdown:

• linux1804-64-shippable-qr: 6
  • opt: 6
• linux1804-64-qr: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-24&endday=2023-10-24&tree=all

Comment 43

[Intermittent Failures Robot]

56 failures in 631 pushes (0.089 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 48
• mozilla-central: 8

Platform and build breakdown:

• linux1804-64-qr: 29
  • opt: 26
  • debug: 3
• linux1804-64-shippable-qr: 27
  • opt: 27

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-25&endday=2023-10-25&tree=all

Comment 45

[Anna Yeddi [:ayeddi]]

(In reply to Noemi Erli[:noemi_erli] from comment #41)

This is still happening a lot, here are some examples:

• https://treeherder.mozilla.org/logviewer?job_id=433684835&repo=autoland&lineNumber=23680 => filed crash bug 1861277
• https://treeherder.mozilla.org/logviewer?job_id=433693142&repo=autoland&lineNumber=36219 => in the Pt.4 patch
• https://treeherder.mozilla.org/logviewer?job_id=433690190&repo=autoland&lineNumber=9586 => in the Pt.4 patch
• https://treeherder.mozilla.org/logviewer?job_id=433690206&repo=autoland&lineNumber=21944 => updated crash bug 1857056
• https://treeherder.mozilla.org/logviewer?job_id=433666294&repo=autoland&lineNumber=51381 => crash bug 1849230
• https://treeherder.mozilla.org/logviewer?job_id=433666392&repo=autoland&lineNumber=13742 => updated crash bug 1857744
  Ana, could you please take a look?
  Thanks,

Thank you for the ping, Noemi! I added comments next to each line above and submitted a patch for a review.

Comment 46

[Anna Yeddi [:ayeddi]]

Attachment: Bug 1858041 - Pt.4. Temporarily skip some Tier 2 a11y_checks for unexpecredly failing/passing or crashing on Autoland. r=Jamie

There are some a11y_checks that are intermittently failing on Autoland and pass on Try and vice versa. Also, a couple of tests are crashing while performing a11y_checks, thus they are to be skipped while we continue the investigation into each of them.

Related bugs:

• Fail-if tests: meta bug 1848402
• Explore capturing click events for a11y_checks: bug 1692110

Comment 47

[Intermittent Failures Robot]

131 failures in 3657 pushes (0.036 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 113
• try: 2
• mozilla-central: 16

Platform and build breakdown:

• linux1804-64-qr: 72
  • opt: 69
  • debug: 3
• linux1804-64-shippable-qr: 58
  • opt: 58
• macosx1015-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-23&endday=2023-10-29&tree=all

Comment 48

[Intermittent Failures Robot]

17 failures in 574 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-qr: 6
  • opt: 6
• linux1804-64-shippable-qr: 11
  • opt: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-30&endday=2023-10-30&tree=all

Comment 49

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/53b8e85e4486
Pt.4. Temporarily skip some Tier 2 a11y_checks for unexpecredly failing/passing or crashing on Autoland. r=devtools-reviewers,credential-management-reviewers,issammani

Comment 50

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/53b8e85e4486

Comment 51

[Intermittent Failures Robot]

19 failures in 573 pushes (0.033 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 3

Platform and build breakdown:

• linux1804-64-qr: 10
  • opt: 10
• linux1804-64-shippable-qr: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-31&endday=2023-10-31&tree=all

Comment 52

[Norisz Fay [:noriszfay]]

Hey Anna, could you take a look at these tests too? It seems a few are still remaining here and there such as: browser_panelUINotifications_multiWindow.js, browser_doorhanger_installs.js.

Comment 54

[Intermittent Failures Robot]

16 failures in 216 pushes (0.074 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-central: 3
• try: 1

Platform and build breakdown:

• linux1804-64-qr: 6
  • opt: 6
• linux1804-64-shippable-qr: 10
  • opt: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-04&endday=2023-11-04&tree=all

Comment 55

[Intermittent Failures Robot]

122 failures in 3356 pushes (0.036 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 101
• mozilla-central: 16
• try: 5

Platform and build breakdown:

• linux1804-64-qr: 54
  • opt: 54
• linux1804-64-shippable-qr: 68
  • opt: 68

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-10-30&endday=2023-11-05&tree=all

Comment 56

[Intermittent Failures Robot]

21 failures in 669 pushes (0.031 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• mozilla-central: 1
• try: 3

Platform and build breakdown:

• linux1804-64-qr: 13
  • opt: 13
• linux1804-64-shippable-qr: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-06&endday=2023-11-06&tree=all

Comment 57

[Intermittent Failures Robot]

17 failures in 552 pushes (0.031 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 3
• try: 1

Platform and build breakdown:

• linux1804-64-qr: 9
  • opt: 9
• linux1804-64-shippable-qr: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-07&endday=2023-11-07&tree=all

Comment 58

[Intermittent Failures Robot]

16 failures in 738 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• try: 2
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-qr: 10
  • opt: 10
• linux1804-64-shippable-qr: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-08&endday=2023-11-08&tree=all

Comment 59

[Anna Yeddi [:ayeddi]]

(In reply to Norisz Fay [:noriszfay] from comment #52)

Hey Anna, could you take a look at these tests too? It seems a few are still remaining here and there such as: browser_panelUINotifications_multiWindow.js, browser_doorhanger_installs.js.

Thank you, Norisz!

I filed a separate bug 1863815 to fix the browser_panelUINotifications_multiWindow.js failure and the patch for it is under the review.

I could not find any new instanced of the browser_doorhanger_installs.js failing, but I keep tracking the failures across the board. If there won't be any new failure, I'll close this bug when the bug 1863815 is closed too. Unless there would new failures appear...

Comment 60

[Intermittent Failures Robot]

23 failures in 658 pushes (0.035 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 20
• mozilla-central: 3

Platform and build breakdown:

• linux1804-64-qr: 11
  • opt: 11
• linux1804-64-shippable-qr: 12
  • opt: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-09&endday=2023-11-09&tree=all

Comment 61

[Intermittent Failures Robot]

15 failures in 453 pushes (0.033 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 9
• mozilla-central: 4
• try: 2

Platform and build breakdown:

• linux1804-64-qr: 8
  • opt: 8
• linux1804-64-shippable-qr: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-10&endday=2023-11-10&tree=all

Comment 62

[Intermittent Failures Robot]

99 failures in 3288 pushes (0.03 failures/push) were associated with this bug in the last 7 days.

This is the #7 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 77
• try: 9
• mozilla-central: 13

Platform and build breakdown:

• linux1804-64-qr: 55
  • opt: 55
• linux1804-64-shippable-qr: 44
  • opt: 44

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-06&endday=2023-11-12&tree=all

Comment 63

[Norisz Fay [:noriszfay]]

Hey Anna, this failure seems to be lingering around recently, could you take a look please? Thank you! Failure log here

Comment 64

[Anna Yeddi [:ayeddi]]

(In reply to Norisz Fay [:noriszfay] from comment #63)

Hey Anna, this failure seems to be lingering around recently, could you take a look please? Thank you! Failure log here

Hey Norisz, I just pushed the WIP patch for these few tests to Try - I'll be adding this as a part 2 patch for the bug 1863584 since these tests were reenabled in there but not fail-ifed. They used to just crash, so it was not clear if they could be failing or not.

Thank you for bringing them up!

Comment 65

[Intermittent Failures Robot]

15 failures in 558 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 3
• autoland: 12

Platform and build breakdown:

• linux1804-64-qr: 6
  • opt: 6
• linux1804-64-shippable-qr: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-13&endday=2023-11-13&tree=all

Comment 66

[Intermittent Failures Robot]

19 failures in 670 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-qr: 12
  • opt: 12
• linux1804-64-shippable-qr: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-14&endday=2023-11-14&tree=all

Comment 67

[Norisz Fay [:noriszfay]]

Some "new" failure seems to have appeared as can be seen here, could you take a look please? Thank you! Log here.

Comment 68

[Intermittent Failures Robot]

63 failures in 3495 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• try: 3
• autoland: 53
• mozilla-central: 7

Platform and build breakdown:

• linux1804-64-qr: 31
  • opt: 31
• linux1804-64-shippable-qr: 32
  • opt: 32

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-13&endday=2023-11-19&tree=all

Comment 69

[Norisz Fay [:noriszfay]]

Hey Anna, any chance you could take a look at this one too? Thank you! Log here

Comment 70

[Intermittent Failures Robot]

16 failures in 615 pushes (0.026 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 11
• mozilla-central: 5

Platform and build breakdown:

• linux1804-64-shippable-qr: 10
  • opt: 10
• linux1804-64-qr: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-23&endday=2023-11-23&tree=all

Comment 72

[Intermittent Failures Robot]

43 failures in 457 pushes (0.094 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 35
• mozilla-central: 7
• try: 1

Platform and build breakdown:

• linux1804-64-qr: 28
  • opt: 28
• linux1804-64-shippable-qr: 14
  • opt: 14
• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-24&endday=2023-11-24&tree=all

Comment 73

[Intermittent Failures Robot]

93 failures in 3623 pushes (0.026 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 78
• try: 2
• mozilla-central: 13

Platform and build breakdown:

• linux1804-64-qr: 55
  • opt: 55
• linux1804-64-shippable-qr: 37
  • opt: 37
• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-20&endday=2023-11-26&tree=all

Comment 74

[Anna Yeddi [:ayeddi]]

(In reply to Norisz Fay [:noriszfay] from comment #69)

Hey Anna, any chance you could take a look at this one too? Thank you! Log here

Thank you, Norisz! Opening that one now - it looks like the failures were fixed, which is great news :)

Comment 76

[Intermittent Failures Robot]

31 failures in 4108 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 23
• mozilla-central: 8

Platform and build breakdown:

• linux1804-64-qr: 15
  • opt: 15
• linux1804-64-shippable-qr: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-11-27&endday=2023-12-03&tree=all

Comment 77

[Anna Yeddi [:ayeddi]]

Attachment: Bug 1858041 - [devtools] Replace `fail-if` with `skip-if` for 2 tests that inconsistently failing/passing a11y-checks. r=nchevobbe

Since the major accessibility remediation work is on the way, one more test is now passing the Tier 2 a11y-checks, thus we are removing the fail-if condition from its manifest to allow for the test harness to expect appropriate results (passing).

Related bugs: bug 1849028, bug 1867062

Comment 78

[Intermittent Failures Robot]

19 failures in 3671 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 16
• try: 2
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-qr: 9
  • opt: 9
• linux1804-64-shippable-qr: 10
  • opt: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-12-04&endday=2023-12-10&tree=all

Comment 79

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/16386070090d
[devtools] Replace `fail-if` with `skip-if` for 2 tests that inconsistently failing/passing a11y-checks. r=devtools-reviewers,nchevobbe

Comment 80

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/16386070090d

Comment 81

[Intermittent Failures Robot]

10 failures in 4224 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-shippable-qr: 6
  • opt: 6
• linux1804-64-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-12-11&endday=2023-12-17&tree=all

Comment 82

[Intermittent Failures Robot]

3 failures in 3349 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• linux1804-64-shippable-qr: 2
  • opt: 2
• linux1804-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-12-18&endday=2023-12-24&tree=all

Comment 83

[Intermittent Failures Robot]

1 failures in 1162 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2023-12-25&endday=2023-12-31&tree=all

Comment 84

[Intermittent Failures Robot]

1 failures in 2274 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2024-01-01&endday=2024-01-07&tree=all

Comment 85

[Intermittent Failures Robot]

1 failures in 3599 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2024-01-15&endday=2024-01-21&tree=all

Comment 86

[Intermittent Failures Robot]

1 failures in 3952 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2024-01-29&endday=2024-02-04&tree=all

Comment 87

[Intermittent Failures Robot]

1 failures in 4009 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-tsan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2024-02-05&endday=2024-02-11&tree=all

Comment 88

[Intermittent Failures Robot]

1 failures in 4052 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2024-02-26&endday=2024-03-03&tree=all

Comment 89

[Anna Yeddi [:ayeddi]]

It's been a while since any unexpected a11y-checks were failing for the pre-existent (before Oct 2023) tests, so I'm closing this bug for now.

Feel free to reopen it, if there are any new concerns.

Comment 90

[Intermittent Failures Robot]

1 failures in 4227 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1858041&startday=2024-11-11&endday=2024-11-17&tree=all