1838695 - Frequent mozilla/tests/webgpu/chunked/<number>/cts.https.html | single tracking bug

Status: RESOLVED FIXED

(Core :: Graphics: WebGPU, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=419445188&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DXq8di7DR9yjfjxQDhmuPw/runs/0/artifacts/public/logs/live_backing.log

---

TEST-START | /_mozilla/webgpu/chunked/14/cts.https.html?q=webgpu:api,validation,encoding,cmds,clearBuffer:buffer,device_mismatch:*
[task 2023-06-15T16:38:41.639Z] 16:38:41     INFO - Closing window add187e0-e897-483c-a6b6-55407ccb2da5
[task 2023-06-15T16:39:06.634Z] 16:39:06     INFO - Got timeout in harness
[task 2023-06-15T16:39:06.645Z] 16:39:06     INFO - TEST-UNEXPECTED-TIMEOUT | /_mozilla/webgpu/chunked/14/cts.https.html?q=webgpu:api,validation,encoding,cmds,clearBuffer:buffer,device_mismatch:* | TestRunner hit external timeout (this may indicate a hang)
[task 2023-06-15T16:39:06.645Z] 16:39:06     INFO - TEST-INFO took 25024ms
[task 2023-06-15T16:40:17.988Z] 16:40:17     INFO - Browser exited with return code 572
[task 2023-06-15T16:40:28.002Z] 16:40:28  WARNING - Forcibly terminating runner process
[task 2023-06-15T16:40:28.050Z] 16:40:28     INFO - Application command: Z:\task_168684417078243\build\application\firefox\firefox.exe -marionette about:blank --wait-for-browser -profile C:\Users\task_168684417078243\AppData\Local\Temp\tmp03432srs
[task 2023-06-15T16:40:28.069Z] 16:40:28     INFO - PID 2140 | 1686847105357	Marionette	INFO	Marionette enabled
[task 2023-06-15T16:40:28.070Z] 16:40:28     INFO - PID 2140 | 1686847105441	Marionette	INFO	Listening on port 56440
[task 2023-06-15T16:40:28.071Z] 16:40:28     INFO - PID 2140 | JavaScript error: resource://gre/modules/XULStore.sys.mjs, line 60: Error: Can't find profile directory.
[task 2023-06-15T16:40:28.072Z] 16:40:28     INFO - PID 2140 | console.error: (new Error("Polling for changes failed: Unexpected content-type \"text/plain;charset=US-ASCII\".", "resource://services-settings/remote-settings.sys.mjs", 324))
[task 2023-06-15T16:40:28.073Z] 16:40:28     INFO - PID 2140 | console.error: (new Error("Unexpected content-type \"text/plain;charset=US-ASCII\"", "resource://services-settings/Utils.sys.mjs", 399))
[task 2023-06-15T16:40:28.073Z] 16:40:28     INFO - PID 2140 | console.error: (new Error("Unexpected content-type \"text/plain;charset=US-ASCII\"", "resource://services-settings/Utils.sys.mjs", 399))
[task 2023-06-15T16:40:28.074Z] 16:40:28     INFO - PID 2140 | console.error: (new Error("Unexpected content-type \"text/plain;charset=US-ASCII\"", "resource://services-settings/Utils.sys.mjs", 399))
[task 2023-06-15T16:40:28.075Z] 16:40:28     INFO - PID 2140 | console.error: (new Error("Unexpected content-type \"text/plain;charset=US-ASCII\"", "resource://services-settings/Utils.sys.mjs", 399))
[task 2023-06-15T16:40:28.076Z] 16:40:28     INFO - Starting runner
[task 2023-06-15T16:40:29.116Z] 16:40:29     INFO - Installing extension from Z:\task_168684417078243\build\tests\extensions\[e-mail]
[task 2023-06-15T16:40:29.352Z] 16:40:29     INFO - TEST-START | /_mozilla/webgpu/chunked/14/cts.https.html?q=webgpu:api,validation,debugMarker:push_pop_call_count_unbalance,command_encoder:*

Comment 1

[Intermittent Failures Robot]

1 failures in 4079 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-06-12&endday=2023-06-18&tree=all

Comment 4

[Intermittent Failures Robot]

22 failures in 793 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• mozilla-central: 5

Platform and build breakdown:

• windows11-32-2009-qr: 8
  • debug: 5
  • opt: 3
• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows11-32-2009-shippable-qr: 10
  • opt: 10
• windows11-64-2009-qr: 3
  • debug: 1
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-06-22&endday=2023-06-22&tree=all

Comment 5

[Cosmin Sabou [:CosminS]]

This has spiked in failures on webgpu runs after bug 1831263, https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2023-06-16&endday=2023-06-23&tree=trunk&bug=1838695
Could you have a look over these?

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1839768

Comment 7

[Intermittent Failures Robot]

20 failures in 606 pushes (0.033 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 19
• mozilla-central: 1

Platform and build breakdown:

• windows11-32-2009-qr: 13
  • opt: 6
  • debug: 7
• windows11-64-2009-qr: 2
  • debug: 1
  • opt: 1
• windows11-32-2009-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-06-23&endday=2023-06-23&tree=all

Comment 8

[Intermittent Failures Robot]

50 failures in 3880 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #23 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 11
• autoland: 39

Platform and build breakdown:

• windows11-32-2009-qr: 27
  • debug: 17
  • opt: 10
• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows11-32-2009-shippable-qr: 17
  • opt: 17
• windows11-64-2009-qr: 5
  • debug: 2
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-06-19&endday=2023-06-25&tree=all

Comment 9

[Erich Gubler [:ErichDonGubler]]

:CosmimS: I intend to look at this as part of my immediate work pipeline in bug 1836479. 👍🏻

Comment 10

[Intermittent Failures Robot]

15 failures in 763 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 4
• autoland: 11

Platform and build breakdown:

• windows11-32-2009-qr: 9
  • debug: 4
  • opt: 5
• windows11-32-2009-shippable-qr: 5
  • opt: 5
• windows11-64-2009-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-06-27&endday=2023-06-27&tree=all

Comment 11

[Intermittent Failures Robot]

17 failures in 668 pushes (0.025 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 11
• mozilla-central: 6

Platform and build breakdown:

• windows11-32-2009-qr: 11
  • debug: 7
  • opt: 4
• windows11-32-2009-shippable-qr: 5
  • opt: 5
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-06-29&endday=2023-06-29&tree=all

Comment 12

[Intermittent Failures Robot]

66 failures in 3594 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #9 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 44
• mozilla-central: 21
• try: 1

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 26
  • opt: 26
• windows11-32-2009-qr: 33
  • opt: 12
  • debug: 21
• windows11-64-2009-qr: 6
  • debug: 3
  • opt: 3
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-06-26&endday=2023-07-02&tree=all

Comment 13

[Intermittent Failures Robot]

16 failures in 523 pushes (0.031 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 3

Platform and build breakdown:

• windows11-32-2009-qr: 12
  • debug: 7
  • opt: 5
• windows11-32-2009-shippable-qr: 3
  • opt: 3
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-04&endday=2023-07-04&tree=all

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1831263

Comment 15

[Intermittent Failures Robot]

18 failures in 704 pushes (0.026 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 14
• mozilla-central: 4

Platform and build breakdown:

• windows11-32-2009-qr: 13
  • opt: 6
  • debug: 7
• windows11-32-2009-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-06&endday=2023-07-06&tree=all

Comment 16

[Noemi Erli[:noemi_erli]]

In the past week we had 115 occurrences of this failures on:

• windows11-32-2009-qr opt and debug
• windows11-32-2009-shippable-qr opt
• windows11-64-2009-qr opt and debug
• windows11-64-2009-shippable-qr opt
  Recent failure log

[task 2023-07-07T10:45:45.586Z] 10:45:45     INFO - Closing window 80e89224-c3b2-418a-9709-ea2310c5af01
[task 2023-07-07T10:45:46.525Z] 10:45:46     INFO - PID 8728 | [GFX1-[]G: CompositorBridgFXeChil1-]: Cod mpositorBrreceidives geCh[GF[IPCiGldX[F GcFl1o[-XG]Xs11F-:X-e  1 wi]tr:]Ch  reaecC-]s::oe oomp oCCmopnossimipiotoves Ismo=PAiCtporBr crobntoBrilrismaidgodtgleSorBCohsuet drhwithi BrreowreaCslndi odngeh
[task 2023-07-07T10:45:46.526Z] 10:45:46     INFO - PID 8728 | ildire drCchgi=eldi erCevchAieeld irbvneosre ceecs eIImPiiCvPvC  ees cIslac PloIlPCCS houtsd oesccw n
[task 2023-07-07T10:45:46.527Z] 10:45:46     INFO - PID 8728 | lloeswit h ree woitwsaithsohn=e wit Area sonrbenasoh rno==reAabmaslShonn=AuotAbrnombalnrodmarlShSomahuutwlSnh
[task 2023-07-07T10:45:46.527Z] 10:45:46     INFO - PID 8728 | uttddownow
[task 2023-07-07T10:45:46.528Z] 10:45:46     INFO - PID 8728 | n
[task 2023-07-07T10:45:46.528Z] 10:45:46     INFO - PID 8728 | down
[task 2023-07-07T10:45:46.551Z] 10:45:46     INFO - PID 8728 | [GFX1-]: CompositorBridgeChild receives IPC close with reason=AbnormalShutdown
[task 2023-07-07T10:45:46.555Z] 10:45:46     INFO - 
[task 2023-07-07T10:45:46.555Z] 10:45:46     INFO - TEST-UNEXPECTED-FAIL | /_mozilla/webgpu/chunked/12/cts.https.html?q=webgpu:api,validation,createBindGroup:buffer,usage:* | :type="uniform" - assert_unreached: 
[task 2023-07-07T10:45:46.555Z] 10:45:46     INFO -   - EXCEPTION: WebGPU device failed to initialize with AbortError "Internal communication error!"; not retrying
[task 2023-07-07T10:45:46.555Z] 10:45:46     INFO -     assert@https://web-platform.test:8443/_mozilla/webgpu/common/util/util.js:37:11
[task 2023-07-07T10:45:46.555Z] 10:45:46     INFO -     acquire@https://web-platform.test:8443/_mozilla/webgpu/webgpu/util/device_pool.js:36:11
[task 2023-07-07T10:45:46.555Z] 10:45:46     INFO -     
[task 2023-07-07T10:45:46.555Z] 10:45:46     INFO -  Reached unreachable code
[task 2023-07-07T10:45:46.555Z] 10:45:46     INFO - wpt_fn@https://web-platform.test:8443/_mozilla/webgpu/common/runtime/wpt.js:65:25```

Comment 17

[Intermittent Failures Robot]

61 failures in 625 pushes (0.098 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 51
• mozilla-central: 10

Platform and build breakdown:

• windows11-32-2009-qr: 24
  • debug: 13
  • opt: 11
• windows11-64-2009-qr: 12
  • opt: 7
  • debug: 5
• windows11-32-2009-shippable-qr: 13
  • opt: 13
• windows11-64-2009-shippable-qr: 12
  • opt: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-07&endday=2023-07-07&tree=all

Comment 18

[Intermittent Failures Robot]

52 failures in 129 pushes (0.403 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 42
• mozilla-central: 10

Platform and build breakdown:

• windows11-32-2009-qr: 22
  • opt: 10
  • debug: 12
• windows11-32-2009-shippable-qr: 14
  • opt: 14
• windows11-64-2009-shippable-qr: 9
  • opt: 9
• windows11-64-2009-qr: 7
  • debug: 3
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-08&endday=2023-07-08&tree=all

Comment 19

[Intermittent Failures Robot]

206 failures in 2956 pushes (0.07 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 165
• mozilla-central: 41

Platform and build breakdown:

• windows11-32-2009-qr: 99
  • debug: 54
  • opt: 45
• windows11-64-2009-shippable-qr: 32
  • opt: 32
• windows11-32-2009-shippable-qr: 50
  • opt: 50
• windows11-64-2009-qr: 25
  • opt: 12
  • debug: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-03&endday=2023-07-09&tree=all

Comment 20

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Erich, is there an ETA to resolve these frequent test failures?

Comment 21

[Jim Blandy :jimb]

We're looking into this. These intermittents significantly reduce the value of the CTS to us, so it's a priority. But we do not have a diagnosis, so we don't have an ETA.

Comment 22

[Jim Blandy :jimb]

Since the intermittents aren't isolated to any particular WebGPU CTS tests, as far as we know, it's difficult to identify some subset of tests we could disable to quiet things down. If we still don't have a diagnosis after a few days, we can talk about broader measures to keep this from wasting people's time until we can get it fixed.

Comment 23

[Intermittent Failures Robot]

45 failures in 589 pushes (0.076 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 34
• mozilla-central: 10
• try: 1

Platform and build breakdown:

• windows11-32-2009-qr: 20
  • debug: 11
  • opt: 9
• windows11-64-2009-qr: 12
  • opt: 7
  • debug: 5
• windows11-32-2009-shippable-qr: 8
  • opt: 8
• windows11-64-2009-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-10&endday=2023-07-10&tree=all

Comment 24

[Intermittent Failures Robot]

66 failures in 647 pushes (0.102 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 58
• mozilla-central: 8

Platform and build breakdown:

• windows11-64-2009-qr: 18
  • debug: 12
  • opt: 6
• windows11-32-2009-shippable-qr: 16
  • opt: 16
• windows11-64-2009-shippable-qr: 8
  • opt: 8
• windows11-32-2009-qr: 24
  • debug: 14
  • opt: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-11&endday=2023-07-11&tree=all

Comment 25

[Intermittent Failures Robot]

65 failures in 655 pushes (0.099 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 48
• mozilla-central: 17

Platform and build breakdown:

• windows11-32-2009-qr: 22
  • debug: 13
  • opt: 9
• windows11-64-2009-qr: 16
  • debug: 9
  • opt: 7
• windows11-32-2009-shippable-qr: 15
  • opt: 15
• windows11-64-2009-shippable-qr: 12
  • opt: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-12&endday=2023-07-12&tree=all

Comment 26

[Intermittent Failures Robot]

47 failures in 706 pushes (0.067 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 42
• mozilla-central: 5

Platform and build breakdown:

• windows11-64-2009-qr: 17
  • opt: 8
  • debug: 9
• windows11-32-2009-qr: 19
  • opt: 11
  • debug: 8
• windows11-64-2009-shippable-qr: 3
  • opt: 3
• windows11-32-2009-shippable-qr: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-13&endday=2023-07-13&tree=all

Comment 27

[Intermittent Failures Robot]

37 failures in 521 pushes (0.071 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 37

Platform and build breakdown:

• windows11-32-2009-qr: 15
  • debug: 7
  • opt: 8
• windows11-64-2009-qr: 8
  • debug: 5
  • opt: 3
• windows11-32-2009-shippable-qr: 11
  • opt: 11
• windows11-64-2009-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-14&endday=2023-07-14&tree=all

Comment 28

[Intermittent Failures Robot]

29 failures in 90 pushes (0.322 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 23
• mozilla-central: 6

Platform and build breakdown:

• windows11-32-2009-qr: 12
  • debug: 5
  • opt: 7
• windows11-32-2009-shippable-qr: 8
  • opt: 8
• windows11-64-2009-shippable-qr: 4
  • opt: 4
• windows11-64-2009-qr: 5
  • opt: 4
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-15&endday=2023-07-15&tree=all

Comment 29

[Intermittent Failures Robot]

310 failures in 3309 pushes (0.094 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 253
• mozilla-central: 56
• try: 1

Platform and build breakdown:

• windows11-32-2009-qr: 118
  • debug: 61
  • opt: 57
• windows11-64-2009-qr: 78
  • opt: 35
  • debug: 43
• windows11-32-2009-shippable-qr: 75
  • opt: 75
• windows11-64-2009-shippable-qr: 38
  • opt: 38
• macosx1015-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-10&endday=2023-07-16&tree=all

Comment 30

[Intermittent Failures Robot]

40 failures in 558 pushes (0.072 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 33
• mozilla-central: 7

Platform and build breakdown:

• windows11-32-2009-qr: 20
  • debug: 9
  • opt: 11
• windows11-64-2009-qr: 5
  • debug: 5
• windows11-32-2009-shippable-qr: 10
  • opt: 10
• windows11-64-2009-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-17&endday=2023-07-17&tree=all

Comment 31

[Intermittent Failures Robot]

83 failures in 665 pushes (0.125 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 67
• mozilla-central: 16

Platform and build breakdown:

• windows11-32-2009-qr: 34
  • debug: 18
  • opt: 16
• windows11-64-2009-qr: 17
  • debug: 10
  • opt: 7
• windows11-32-2009-shippable-qr: 19
  • opt: 19
• windows11-64-2009-shippable-qr: 13
  • opt: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-18&endday=2023-07-18&tree=all

Comment 32

[Erich Gubler [:ErichDonGubler]]

There were a significant number of failures associated with this bug that are because of this. I'm hoping that the number of failures here goes down significantly after fixing it. 🤞🏻

Comment 33

[Intermittent Failures Robot]

68 failures in 756 pushes (0.09 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 34
• autoland: 34

Platform and build breakdown:

• windows11-32-2009-qr: 25
  • debug: 16
  • opt: 9
• windows11-64-2009-qr: 13
  • debug: 9
  • opt: 4
• windows11-32-2009-shippable-qr: 20
  • opt: 20
• windows11-64-2009-shippable-qr: 10
  • opt: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-19&endday=2023-07-19&tree=all

Comment 34

[Intermittent Failures Robot]

24 failures in 645 pushes (0.037 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• mozilla-central: 7

Platform and build breakdown:

• windows11-64-2009-qr: 5
  • debug: 4
  • opt: 1
• windows11-32-2009-shippable-qr: 8
  • opt: 8
• windows11-32-2009-qr: 11
  • debug: 5
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-20&endday=2023-07-20&tree=all

Comment 35

[Intermittent Failures Robot]

34 failures in 691 pushes (0.049 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 24
• mozilla-central: 10

Platform and build breakdown:

• windows11-32-2009-qr: 16
  • debug: 9
  • opt: 7
• windows11-32-2009-shippable-qr: 12
  • opt: 12
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• windows11-64-2009-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-21&endday=2023-07-21&tree=all

Comment 36

[Intermittent Failures Robot]

25 failures in 112 pushes (0.223 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 12

Platform and build breakdown:

• windows11-32-2009-qr: 12
  • debug: 7
  • opt: 5
• windows11-32-2009-shippable-qr: 8
  • opt: 8
• windows11-64-2009-qr: 3
  • debug: 2
  • opt: 1
• windows11-64-2009-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-22&endday=2023-07-22&tree=all

Comment 37

[Intermittent Failures Robot]

298 failures in 3520 pushes (0.085 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 206
• mozilla-central: 92

Platform and build breakdown:

• windows11-32-2009-qr: 128
  • debug: 69
  • opt: 59
• windows11-64-2009-qr: 53
  • debug: 34
  • opt: 19
• windows11-32-2009-shippable-qr: 84
  • opt: 84
• windows11-64-2009-shippable-qr: 33
  • opt: 33

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-17&endday=2023-07-23&tree=all

Comment 38

[Intermittent Failures Robot]

37 failures in 635 pushes (0.058 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 29
• mozilla-central: 7
• try: 1

Platform and build breakdown:

• windows11-32-2009-qr: 17
  • debug: 10
  • opt: 7
• windows11-64-2009-qr: 5
  • debug: 1
  • opt: 4
• windows11-32-2009-shippable-qr: 12
  • opt: 12
• windows11-64-2009-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-24&endday=2023-07-24&tree=all

Comment 39

[Intermittent Failures Robot]

43 failures in 635 pushes (0.068 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 34
• mozilla-central: 9

Platform and build breakdown:

• windows11-32-2009-qr: 21
  • debug: 10
  • opt: 11
• windows11-32-2009-shippable-qr: 11
  • opt: 11
• windows11-64-2009-qr: 8
  • debug: 3
  • opt: 5
• windows11-64-2009-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-25&endday=2023-07-25&tree=all

Comment 40

[Intermittent Failures Robot]

30 failures in 686 pushes (0.044 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 26
• mozilla-central: 4

Platform and build breakdown:

• windows11-32-2009-qr: 13
  • debug: 7
  • opt: 6
• windows11-64-2009-qr: 9
  • opt: 9
• windows11-32-2009-shippable-qr: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-26&endday=2023-07-26&tree=all

Comment 41

[Natalia Csoregi [:nataliaCs]]

Update

There have been 207 total failures within the last 7 days:

• 96 failures on Windows 11 x86 22H2 WebRender opt
• 63 failures on Windows 11 x86 22H2 WebRender Shippable opt
• 36 failures on Windows 11 x64 22H2 WebRender opt
• 12 failures on Windows 11 x64 22H2 WebRender Shippable opt

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=424073238&repo=autoland&lineNumber=1758

Comment 42

[Intermittent Failures Robot]

19 failures in 704 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-central: 4

Platform and build breakdown:

• windows11-32-2009-qr: 9
  • debug: 5
  • opt: 4
• windows11-64-2009-qr: 3
  • debug: 1
  • opt: 2
• windows11-32-2009-shippable-qr: 6
  • opt: 6
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-27&endday=2023-07-27&tree=all

Comment 43

[Intermittent Failures Robot]

40 failures in 573 pushes (0.07 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 25
• mozilla-central: 15

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 3
  • opt: 1
• windows11-32-2009-qr: 16
  • opt: 6
  • debug: 10
• windows11-32-2009-shippable-qr: 15
  • opt: 15
• windows11-64-2009-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-28&endday=2023-07-28&tree=all

Comment 44

[Intermittent Failures Robot]

15 failures in 100 pushes (0.15 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 8
• mozilla-central: 7

Platform and build breakdown:

• windows11-32-2009-qr: 7
  • debug: 4
  • opt: 3
• windows11-32-2009-shippable-qr: 6
  • opt: 6
• windows11-64-2009-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-29&endday=2023-07-29&tree=all

Comment 45

[Intermittent Failures Robot]

198 failures in 3403 pushes (0.058 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 144
• mozilla-central: 52
• try: 2

Platform and build breakdown:

• windows11-32-2009-qr: 91
  • debug: 52
  • opt: 39
• windows11-64-2009-qr: 30
  • debug: 8
  • opt: 22
• windows11-32-2009-shippable-qr: 63
  • opt: 63
• windows11-64-2009-shippable-qr: 14
  • opt: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-24&endday=2023-07-30&tree=all

Comment 46

[Erich Gubler [:ErichDonGubler]]

It seems that the highest number of intermittent failures are on 32-bit builds, by a wide margin. I think the next priority for now is going to be tackling those.

Comment 47

[Intermittent Failures Robot]

48 failures in 536 pushes (0.09 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 38
• mozilla-central: 10

Platform and build breakdown:

• windows11-64-2009-qr: 5
  • debug: 1
  • opt: 4
• windows11-32-2009-qr: 24
  • debug: 12
  • opt: 12
• windows11-32-2009-shippable-qr: 18
  • opt: 18
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-31&endday=2023-07-31&tree=all

Comment 48

[Intermittent Failures Robot]

41 failures in 630 pushes (0.065 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 4
• autoland: 37

Platform and build breakdown:

• windows11-32-2009-qr: 23
  • debug: 10
  • opt: 13
• windows11-32-2009-shippable-qr: 13
  • opt: 13
• windows11-64-2009-qr: 5
  • debug: 2
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-01&endday=2023-08-01&tree=all

Comment 49

[Intermittent Failures Robot]

38 failures in 679 pushes (0.056 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 22
• mozilla-central: 16

Platform and build breakdown:

• windows11-32-2009-qr: 18
  • debug: 12
  • opt: 6
• windows11-32-2009-shippable-qr: 14
  • opt: 14
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• windows11-64-2009-qr: 4
  • debug: 3
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-02&endday=2023-08-02&tree=all

Comment 50

[Intermittent Failures Robot]

45 failures in 652 pushes (0.069 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 33
• mozilla-central: 12

Platform and build breakdown:

• windows11-32-2009-qr: 24
  • debug: 15
  • opt: 9
• windows11-32-2009-shippable-qr: 16
  • opt: 16
• windows11-64-2009-qr: 2
  • opt: 1
  • debug: 1
• windows11-64-2009-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-03&endday=2023-08-03&tree=all

Comment 51

[Intermittent Failures Robot]

43 failures in 616 pushes (0.07 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 36
• mozilla-central: 7

Platform and build breakdown:

• windows11-32-2009-qr: 21
  • debug: 11
  • opt: 10
• windows11-32-2009-shippable-qr: 13
  • opt: 13
• windows11-64-2009-shippable-qr: 5
  • opt: 5
• windows11-64-2009-qr: 4
  • opt: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-04&endday=2023-08-04&tree=all

Comment 52

[Intermittent Failures Robot]

16 failures in 103 pushes (0.155 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 1
• mozilla-central: 7
• autoland: 8

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1
• windows11-32-2009-qr: 6
  • debug: 4
  • opt: 2
• windows11-32-2009-shippable-qr: 6
  • opt: 6
• windows11-64-2009-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-05&endday=2023-08-05&tree=all

Comment 53

[Intermittent Failures Robot]

239 failures in 3276 pushes (0.073 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 182
• mozilla-central: 56
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 25
  • debug: 12
  • opt: 13
• windows11-32-2009-qr: 118
  • debug: 65
  • opt: 53
• windows11-32-2009-shippable-qr: 82
  • opt: 82
• windows11-64-2009-shippable-qr: 14
  • opt: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-07-31&endday=2023-08-06&tree=all

Comment 54

[Narcis Beleuzu [:NarcisB]]

This failure turned into a perma failure and there are 269 failures in the last 7 days.
Nicolas, can you please take a look over ?

Comment 55

[Erich Gubler [:ErichDonGubler]]

:NarcisB: Could you elaborate on what but signatures turned perma? This bug aggregates numerous signatures, and part of our work has been to isolate them into separate well-understood bugs.

Comment 56

[Narcis Beleuzu [:NarcisB]]

:ErichDonGubler, I did a few backfills & retriggers and all of them failed

Comment 57

[Intermittent Failures Robot]

28 failures in 556 pushes (0.05 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 7

Platform and build breakdown:

• windows11-32-2009-qr: 16
  • debug: 10
  • opt: 6
• windows11-64-2009-qr: 3
  • debug: 1
  • opt: 2
• windows11-32-2009-shippable-qr: 8
  • opt: 8
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-07&endday=2023-08-07&tree=all

Comment 58

[Iulian Moraru]

This seems to be permafailing only with this failure line TEST-UNEXPECTED-FAIL | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="maxBufferSize" - assert_unreached:
Here are some examples that show that this is not permafailing on all webgpu jobs or other signatures:

• example 1
• example 2
• example 3

Snippet from permafailing job:

[task 2023-08-07T21:20:44.729Z] 21:20:44     INFO - TEST-START | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:*
[task 2023-08-07T21:20:44.734Z] 21:20:44     INFO - Closing window 512136e6-ed72-4bd1-8c7d-4fe09779f824
[task 2023-08-07T21:20:51.636Z] 21:20:51     INFO - .......
[task 2023-08-07T21:20:51.636Z] 21:20:51     INFO - TEST-PASS | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="maxSampledTexturesPerShaderStage" 
[task 2023-08-07T21:20:51.636Z] 21:20:51     INFO - TEST-PASS | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="maxSamplersPerShaderStage" 
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO - TEST-PASS | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="maxStorageBuffersPerShaderStage" 
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO - TEST-PASS | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="maxStorageTexturesPerShaderStage" 
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO - TEST-PASS | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="maxUniformBuffersPerShaderStage" 
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO - TEST-PASS | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="maxUniformBufferBindingSize" 
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO - TEST-PASS | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="maxStorageBufferBindingSize" 
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO - TEST-PASS | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="minUniformBufferOffsetAlignment" 
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO - TEST-PASS | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="minStorageBufferOffsetAlignment" 
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO - TEST-PASS | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="maxVertexBuffers" 
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO - TEST-UNEXPECTED-FAIL | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | :limit="maxBufferSize" - assert_unreached: 
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO -   - EXCEPTION: requestDevice: Request for limit 'maxBufferSize' must be <= supported 0, was 268435455.
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO -     subcase: mul=1;add=-1
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO -     @https://web-platform.test:8443/_mozilla/webgpu/webgpu/api/operation/adapter/requestDevice.spec.js:263:36
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO -   - INFO: subcase: mul=1;add=-1
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO -     OK
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO -   - EXCEPTION: requestDevice: Request for limit 'maxBufferSize' must be <= supported 0, was 268435356.
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO -     subcase: mul=1;add=-100
[task 2023-08-07T21:20:51.637Z] 21:20:51     INFO -     @https://web-platform.test:8443/_mozilla/webgpu/webgpu/api/operation/adapter/requestDevice.spec.js:263:36
[task 2023-08-07T21:20:51.638Z] 21:20:51     INFO -   - INFO: subcase: mul=1;add=-100
[task 2023-08-07T21:20:51.638Z] 21:20:51     INFO -     OK
[task 2023-08-07T21:20:51.638Z] 21:20:51     INFO -  Reached unreachable code
[task 2023-08-07T21:20:51.638Z] 21:20:51     INFO - wpt_fn@https://web-platform.test:8443/_mozilla/webgpu/common/runtime/wpt.js:65:25
[task 2023-08-07T21:20:51.643Z] 21:20:51     INFO - ...........
[task 2023-08-07T21:20:51.643Z] 21:20:51     INFO - TEST-OK | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,adapter,requestDevice:limit,worse_than_default:* | took 6909ms
[task 2023-08-07T21:20:51.645Z] 21:20:51     INFO - PID 2172 | 1691443251640	Marionette	INFO	Stopped listening on port 53094
[task 2023-08-07T21:20:52.327Z] 21:20:52     INFO - PID 2172 | console.error: ({})
[task 2023-08-07T21:20:52.636Z] 21:20:52     INFO - Browser exited with return code 0
[task 2023-08-07T21:20:52.637Z] 21:20:52     INFO - Closing logging queue
[task 2023-08-07T21:20:52.637Z] 21:20:52     INFO - queue closed
[task 2023-08-07T21:20:52.841Z] 21:20:52     INFO - Application command: Z:\task_169144156238687\build\application\firefox\firefox.exe -marionette about:blank --wait-for-browser -profile C:\Users\task_169144156238687\AppData\Local\Temp\tmpxza982he
[task 2023-08-07T21:20:52.852Z] 21:20:52     INFO - PID 3652 | 1691443213094	Marionette	INFO	Marionette enabled
[task 2023-08-07T21:20:52.853Z] 21:20:52     INFO - PID 3652 | 1691443213190	Marionette	INFO	Listening on port 53095
[task 2023-08-07T21:20:52.853Z] 21:20:52     INFO - PID 3652 | JavaScript error: resource://gre/modules/XULStore.sys.mjs, line 60: Error: Can't find profile directory.
[task 2023-08-07T21:20:52.854Z] 21:20:52     INFO - PID 3652 | console.error: (new Error("Unexpected content-type \"text/plain;charset=US-ASCII\"", "resource://services-settings/Utils.sys.mjs", 399))
[task 2023-08-07T21:20:52.855Z] 21:20:52     INFO - PID 3652 | console.error: (new Error("Unexpected content-type \"text/plain;charset=US-ASCII\"", "resource://services-settings/Utils.sys.mjs", 399))
[task 2023-08-07T21:20:52.855Z] 21:20:52     INFO - PID 3652 | console.error: (new Error("Unexpected content-type \"text/plain;charset=US-ASCII\"", "resource://services-settings/Utils.sys.mjs", 399))
[task 2023-08-07T21:20:52.856Z] 21:20:52     INFO - PID 3652 | console.error: (new Error("Unexpected content-type \"text/plain;charset=US-ASCII\"", "resource://services-settings/Utils.sys.mjs", 399))
[task 2023-08-07T21:20:52.857Z] 21:20:52     INFO - PID 3652 | console.error: (new Error("Unexpected content-type \"text/plain;charset=US-ASCII\"", "resource://services-settings/Utils.sys.mjs", 399))
[task 2023-08-07T21:20:52.857Z] 21:20:52     INFO - PID 3652 | console.error: (new Error("Polling for changes failed: Unexpected content-type \"text/plain;charset=US-ASCII\".", "resource://services-settings/remote-settings.sys.mjs", 321))
[task 2023-08-07T21:20:52.857Z] 21:20:52     INFO - Starting runner
[task 2023-08-07T21:20:54.217Z] 21:20:54     INFO - Installing extension from Z:\task_169144156238687\build\tests\extensions\[e-mail]
[task 2023-08-07T21:20:54.473Z] 21:20:54     INFO - TEST-START | /_mozilla/webgpu/chunked/1/cts.https.html?q=webgpu:api,operation,buffers,map:mapAsync,write:*

Comment 59

[Iulian Moraru]

Hi Erich! If you want to take a look at the details I've mentioned above, to get a more clearer picture.
And, should we file a separate bug for the permafailure?

Comment 60

[Iulian Moraru]

Filed Bug 1847771 to be easier to track the permafailure and not mess up the orange factor on this bug. I've managed to find what caused this.

Comment 61

[Intermittent Failures Robot]

15 failures in 688 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 11
• mozilla-central: 4

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 5
  • opt: 5
• windows11-64-2009-shippable-qr: 4
  • opt: 4
• windows11-32-2009-qr: 5
  • opt: 4
  • debug: 1
• windows11-64-2009-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-08&endday=2023-08-08&tree=all

Comment 62

[Intermittent Failures Robot]

25 failures in 740 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 18
• mozilla-central: 7

Platform and build breakdown:

• windows11-64-2009-qr: 5
  • opt: 4
  • debug: 1
• windows11-32-2009-qr: 11
  • opt: 6
  • debug: 5
• windows11-32-2009-shippable-qr: 7
  • opt: 7
• windows11-64-2009-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-09&endday=2023-08-09&tree=all

Comment 63

[Intermittent Failures Robot]

53 failures in 696 pushes (0.076 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 42
• try: 2
• mozilla-central: 9

Platform and build breakdown:

• windows11-32-2009-qr: 23
  • debug: 13
  • opt: 10
• windows11-32-2009-shippable-qr: 15
  • opt: 15
• windows11-64-2009-shippable-qr: 6
  • opt: 6
• windows11-64-2009-qr: 9
  • opt: 6
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-10&endday=2023-08-10&tree=all

Comment 64

[Intermittent Failures Robot]

15 failures in 493 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 11
• mozilla-central: 4

Platform and build breakdown:

• windows11-32-2009-qr: 8
  • opt: 5
  • debug: 3
• windows11-64-2009-qr: 2
  • opt: 2
• windows11-32-2009-shippable-qr: 4
  • opt: 4
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-11&endday=2023-08-11&tree=all

Comment 65

[Intermittent Failures Robot]

16 failures in 105 pushes (0.152 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 9
• mozilla-central: 7

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • opt: 2
  • debug: 1
• windows11-32-2009-shippable-qr: 4
  • opt: 4
• windows11-64-2009-shippable-qr: 5
  • opt: 5
• windows11-32-2009-qr: 4
  • debug: 3
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-12&endday=2023-08-12&tree=all

Comment 66

[Intermittent Failures Robot]

161 failures in 3341 pushes (0.048 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 116
• mozilla-central: 43
• try: 2

Platform and build breakdown:

• windows11-32-2009-qr: 69
  • debug: 37
  • opt: 32
• windows11-64-2009-qr: 24
  • debug: 7
  • opt: 17
• windows11-32-2009-shippable-qr: 48
  • opt: 48
• windows11-64-2009-shippable-qr: 20
  • opt: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-07&endday=2023-08-13&tree=all

Comment 67

[Intermittent Failures Robot]

22 failures in 560 pushes (0.039 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 6

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • opt: 3
• windows11-32-2009-shippable-qr: 6
  • opt: 6
• windows11-64-2009-shippable-qr: 6
  • opt: 6
• windows11-32-2009-qr: 7
  • opt: 6
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-14&endday=2023-08-14&tree=all

Comment 68

[Intermittent Failures Robot]

20 failures in 608 pushes (0.033 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• mozilla-central: 3

Platform and build breakdown:

• windows11-32-2009-qr: 5
  • opt: 3
  • debug: 2
• windows11-64-2009-qr: 3
  • opt: 3
• windows11-32-2009-shippable-qr: 8
  • opt: 8
• windows11-64-2009-shippable-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-15&endday=2023-08-15&tree=all

Comment 69

[Intermittent Failures Robot]

17 failures in 732 pushes (0.023 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-central: 5

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 7
  • opt: 7
• windows11-64-2009-qr: 5
  • opt: 3
  • debug: 2
• windows11-64-2009-shippable-qr: 3
  • opt: 3
• windows11-32-2009-qr: 2
  • opt: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-16&endday=2023-08-16&tree=all

Comment 70

[Intermittent Failures Robot]

18 failures in 599 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 7
• autoland: 11

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 3
  • opt: 1
• windows11-32-2009-qr: 4
  • opt: 3
  • debug: 1
• windows11-32-2009-shippable-qr: 8
  • opt: 8
• windows11-64-2009-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-17&endday=2023-08-17&tree=all

Comment 71

[Intermittent Failures Robot]

27 failures in 424 pushes (0.064 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• mozilla-central: 10

Platform and build breakdown:

• windows11-32-2009-qr: 9
  • opt: 5
  • debug: 4
• windows11-32-2009-shippable-qr: 5
  • opt: 5
• windows11-64-2009-shippable-qr: 6
  • opt: 6
• windows11-64-2009-qr: 7
  • debug: 6
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-18&endday=2023-08-18&tree=all

Comment 72

[Intermittent Failures Robot]

118 failures in 3076 pushes (0.038 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 82
• mozilla-central: 36

Platform and build breakdown:

• windows11-64-2009-qr: 27
  • opt: 12
  • debug: 15
• windows11-32-2009-shippable-qr: 36
  • opt: 36
• windows11-32-2009-qr: 30
  • opt: 18
  • debug: 12
• windows11-64-2009-shippable-qr: 25
  • opt: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-14&endday=2023-08-20&tree=all

Comment 73

[Intermittent Failures Robot]

23 failures in 254 pushes (0.091 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 7

Platform and build breakdown:

• windows11-64-2009-qr: 6
  • debug: 3
  • opt: 3
• windows11-32-2009-qr: 7
  • opt: 3
  • debug: 4
• windows11-64-2009-shippable-qr: 5
  • opt: 5
• windows11-32-2009-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-21&endday=2023-08-21&tree=all

Comment 74

[Intermittent Failures Robot]

24 failures in 339 pushes (0.071 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17
• mozilla-central: 7

Platform and build breakdown:

• windows11-32-2009-qr: 8
  • debug: 6
  • opt: 2
• windows11-64-2009-qr: 8
  • opt: 6
  • debug: 2
• windows11-32-2009-shippable-qr: 4
  • opt: 4
• windows11-64-2009-shippable-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-22&endday=2023-08-22&tree=all

Comment 75

[Intermittent Failures Robot]

49 failures in 433 pushes (0.113 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 37
• mozilla-central: 12

Platform and build breakdown:

• windows11-64-2009-qr: 13
  • opt: 7
  • debug: 6
• windows11-32-2009-qr: 12
  • opt: 7
  • debug: 5
• windows11-32-2009-shippable-qr: 12
  • opt: 12
• windows11-64-2009-shippable-qr: 12
  • opt: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-23&endday=2023-08-23&tree=all

Comment 76

[Intermittent Failures Robot]

15 failures in 446 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-central: 3

Platform and build breakdown:

• windows11-64-2009-qr: 6
  • debug: 4
  • opt: 2
• windows11-32-2009-qr: 4
  • debug: 3
  • opt: 1
• windows11-32-2009-shippable-qr: 2
  • opt: 2
• windows11-64-2009-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-24&endday=2023-08-24&tree=all

Comment 77

[Intermittent Failures Robot]

26 failures in 323 pushes (0.08 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 5

Platform and build breakdown:

• windows11-32-2009-qr: 9
  • opt: 6
  • debug: 3
• windows11-64-2009-qr: 8
  • opt: 6
  • debug: 2
• windows11-32-2009-shippable-qr: 6
  • opt: 6
• windows11-64-2009-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-25&endday=2023-08-25&tree=all

Comment 78

[Natalia Csoregi [:nataliaCs]]

Update

There have been 164 failures within the last 7 days:

• 46 failures on Windows 11 x86 22H2 WebRender opt/debug
• 33 failures on Windows 11 x86 22H2 WebRender Shippable opt
• 50 failures on Windows 11 x64 22H2 WebRender debug
• 35 failures on Windows 11 x64 22H2 WebRender Shippable opt

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=427155783&repo=mozilla-central&lineNumber=7277

Comment 79

[Intermittent Failures Robot]

17 failures in 362 pushes (0.047 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 9
• mozilla-central: 8

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • debug: 2
  • opt: 2
• windows11-64-2009-qr: 4
  • opt: 1
  • debug: 3
• windows11-32-2009-shippable-qr: 4
  • opt: 4
• windows11-64-2009-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-26&endday=2023-08-26&tree=all

Comment 80

[Intermittent Failures Robot]

162 failures in 2218 pushes (0.073 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 117
• mozilla-central: 45

Platform and build breakdown:

• windows11-64-2009-qr: 49
  • debug: 22
  • opt: 27
• windows11-32-2009-qr: 44
  • opt: 21
  • debug: 23
• windows11-64-2009-shippable-qr: 33
  • opt: 33
• windows11-32-2009-shippable-qr: 36
  • opt: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-21&endday=2023-08-27&tree=all

Comment 81

[Intermittent Failures Robot]

26 failures in 485 pushes (0.054 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 9
• autoland: 17

Platform and build breakdown:

• windows11-64-2009-qr: 9
  • debug: 4
  • opt: 5
• windows11-32-2009-shippable-qr: 6
  • opt: 6
• windows11-32-2009-qr: 4
  • debug: 2
  • opt: 2
• windows11-64-2009-shippable-qr: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-28&endday=2023-08-28&tree=all

Comment 82

[Intermittent Failures Robot]

53 failures in 533 pushes (0.099 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 46
• mozilla-central: 7

Platform and build breakdown:

• windows11-64-2009-qr: 14
  • debug: 7
  • opt: 7
• windows11-32-2009-qr: 17
  • opt: 10
  • debug: 7
• windows11-32-2009-shippable-qr: 17
  • opt: 17
• windows11-64-2009-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-29&endday=2023-08-29&tree=all

Comment 83

[Intermittent Failures Robot]

18 failures in 641 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-central: 3

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 2
  • opt: 1
• windows11-32-2009-shippable-qr: 7
  • opt: 7
• windows11-64-2009-shippable-qr: 5
  • opt: 5
• windows11-32-2009-qr: 3
  • debug: 1
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-31&endday=2023-08-31&tree=all

Comment 84

[Intermittent Failures Robot]

23 failures in 577 pushes (0.04 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 19
• mozilla-central: 4

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 12
  • opt: 12
• windows11-32-2009-qr: 3
  • debug: 2
  • opt: 1
• windows11-64-2009-shippable-qr: 4
  • opt: 4
• windows11-64-2009-qr: 4
  • opt: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-01&endday=2023-09-01&tree=all

Comment 85

[Intermittent Failures Robot]

19 failures in 136 pushes (0.14 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-central: 7

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • opt: 2
  • debug: 2
• windows11-32-2009-shippable-qr: 6
  • opt: 6
• windows11-64-2009-shippable-qr: 3
  • opt: 3
• windows11-64-2009-qr: 6
  • opt: 4
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-02&endday=2023-09-02&tree=all

Comment 86

[Intermittent Failures Robot]

160 failures in 3019 pushes (0.053 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 40
• autoland: 120

Platform and build breakdown:

• windows11-64-2009-qr: 43
  • debug: 21
  • opt: 22
• windows11-32-2009-qr: 33
  • debug: 15
  • opt: 18
• windows11-32-2009-shippable-qr: 55
  • opt: 55
• windows11-64-2009-shippable-qr: 29
  • opt: 29

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-08-28&endday=2023-09-03&tree=all

Comment 87

[Intermittent Failures Robot]

17 failures in 348 pushes (0.049 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-central: 5

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • opt: 2
  • debug: 2
• windows11-64-2009-qr: 3
  • debug: 2
  • opt: 1
• windows11-32-2009-shippable-qr: 7
  • opt: 7
• windows11-64-2009-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-04&endday=2023-09-04&tree=all

Comment 88

[Intermittent Failures Robot]

23 failures in 634 pushes (0.036 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 2

Platform and build breakdown:

• windows11-32-2009-qr: 8
  • opt: 5
  • debug: 3
• windows11-32-2009-shippable-qr: 8
  • opt: 8
• windows11-64-2009-qr: 6
  • opt: 5
  • debug: 1
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-05&endday=2023-09-05&tree=all

Comment 89

[Intermittent Failures Robot]

56 failures in 724 pushes (0.077 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 49
• mozilla-central: 7

Platform and build breakdown:

• windows11-32-2009-qr: 6
  • debug: 2
  • opt: 4
• windows11-32-2009-shippable-qr: 7
  • opt: 7
• windows11-64-2009-shippable-qr: 3
  • opt: 3
• windows11-64-2009-qr: 2
  • opt: 2
• linux2204-64-wayland: 22
  • opt: 11
  • debug: 11
• linux2204-64-wayland-shippable: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-06&endday=2023-09-06&tree=all

Comment 90

[Intermittent Failures Robot]

61 failures in 653 pushes (0.093 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 55
• mozilla-central: 6

Platform and build breakdown:

• linux2204-64-wayland: 27
  • opt: 12
  • debug: 15
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• linux2204-64-wayland-shippable: 19
  • opt: 19
• windows11-64-2009-qr: 6
  • opt: 6
• windows11-32-2009-qr: 6
  • opt: 5
  • debug: 1
• windows11-32-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-07&endday=2023-09-07&tree=all

Comment 91

[Intermittent Failures Robot]

45 failures in 682 pushes (0.066 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 4
• autoland: 41

Platform and build breakdown:

• linux2204-64-wayland-shippable: 16
  • opt: 16
• windows11-64-2009-qr: 6
  • debug: 2
  • opt: 4
• linux2204-64-wayland: 16
  • opt: 9
  • debug: 7
• windows11-32-2009-qr: 3
  • opt: 2
  • debug: 1
• windows11-64-2009-shippable-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-08&endday=2023-09-08&tree=all

Comment 92

[Intermittent Failures Robot]

36 failures in 125 pushes (0.288 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 29
• mozilla-central: 7

Platform and build breakdown:

• linux2204-64-wayland: 10
  • debug: 6
  • opt: 4
• windows11-32-2009-qr: 4
  • opt: 3
  • debug: 1
• windows11-32-2009-shippable-qr: 5
  • opt: 5
• linux2204-64-wayland-shippable: 13
  • opt: 13
• windows11-64-2009-qr: 2
  • opt: 2
• windows11-64-2009-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-09&endday=2023-09-09&tree=all

Comment 93

[Intermittent Failures Robot]

276 failures in 3267 pushes (0.084 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 239
• mozilla-central: 37

Platform and build breakdown:

• windows11-32-2009-qr: 38
  • opt: 27
  • debug: 11
• windows11-64-2009-qr: 26
  • debug: 5
  • opt: 21
• windows11-32-2009-shippable-qr: 30
  • opt: 30
• windows11-64-2009-shippable-qr: 16
  • opt: 16
• linux2204-64-wayland: 91
  • opt: 43
  • debug: 48
• linux2204-64-wayland-shippable: 75
  • opt: 75

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-04&endday=2023-09-10&tree=all

Comment 94

[Intermittent Failures Robot]

89 failures in 545 pushes (0.163 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 81
• mozilla-central: 5
• try: 3

Platform and build breakdown:

• windows11-64-2009-qr: 19
  • debug: 12
  • opt: 7
• linux2204-64-wayland: 16
  • debug: 11
  • opt: 5
• windows11-32-2009-shippable-qr: 11
  • opt: 11
• linux2204-64-wayland-shippable: 16
  • opt: 16
• windows11-64-2009-shippable-qr: 11
  • opt: 11
• windows11-32-2009-qr: 16
  • debug: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-11&endday=2023-09-11&tree=all

Comment 95

[Intermittent Failures Robot]

184 failures in 559 pushes (0.329 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 45
• autoland: 139

Platform and build breakdown:

• windows11-32-2009-qr: 40
  • debug: 22
  • opt: 18
• windows11-32-2009-shippable-qr: 36
  • opt: 36
• linux2204-64-wayland-shippable: 11
  • opt: 11
• windows11-64-2009-qr: 36
  • debug: 12
  • opt: 24
• linux2204-64-wayland: 15
  • opt: 6
  • debug: 9
• windows11-64-2009-shippable-qr: 38
  • opt: 38
• linux1804-64-shippable-qr: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-12&endday=2023-09-12&tree=all

Comment 96

[Intermittent Failures Robot]

129 failures in 662 pushes (0.195 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 110
• mozilla-central: 19

Platform and build breakdown:

• windows11-32-2009-qr: 31
  • debug: 12
  • opt: 19
• windows11-64-2009-qr: 31
  • opt: 20
  • debug: 11
• linux2204-64-wayland: 6
  • opt: 6
• windows11-32-2009-shippable-qr: 25
  • opt: 25
• windows11-64-2009-shippable-qr: 28
  • opt: 28
• linux2204-64-wayland-shippable: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-13&endday=2023-09-13&tree=all

Comment 97

[Narcis Beleuzu [:NarcisB]]

Update:
There have been 626 failures within the last 7 days:

• 207 failures on linux2204-64-wayland
• 206 failures on windows11-64-2009
• 205 failures on windows11-32-2009
• 8 failures on linux1804-64-shippable-qr

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=429101013&repo=mozilla-central&lineNumber=2267

Eric, can you please take a look?

Comment 98

[Intermittent Failures Robot]

148 failures in 697 pushes (0.212 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 102
• mozilla-central: 46

Platform and build breakdown:

• windows11-64-2009-qr: 35
  • debug: 17
  • opt: 18
• windows11-32-2009-qr: 26
  • debug: 13
  • opt: 13
• windows11-32-2009-shippable-qr: 30
  • opt: 30
• linux2204-64-wayland-shippable: 16
  • opt: 16
• windows11-64-2009-shippable-qr: 28
  • opt: 28
• linux2204-64-wayland: 10
  • opt: 10
• linux1804-64-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-14&endday=2023-09-14&tree=all

Comment 99

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Attachment: Bug 1838695 - set web-platform webgpu tests as tier 3. r=jimb

These tests fail intermittently or permanently. The plan is to fix them in a
burndown. After they all pass, the tier can be changed to 2 or 1, and their
results will be monitored (sheriffed).

Comment 100

[Intermittent Failures Robot]

137 failures in 697 pushes (0.197 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 106
• mozilla-central: 31

Platform and build breakdown:

• windows11-64-2009-qr: 33
  • debug: 15
  • opt: 18
• windows11-32-2009-qr: 33
  • opt: 16
  • debug: 17
• linux2204-64-wayland-shippable: 20
  • opt: 20
• windows11-32-2009-shippable-qr: 21
  • opt: 21
• windows11-64-2009-shippable-qr: 19
  • opt: 19
• linux2204-64-wayland: 10
  • opt: 10
• macosx1015-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-15&endday=2023-09-15&tree=all

Comment 101

[Intermittent Failures Robot]

112 failures in 100 pushes (1.12 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 85
• mozilla-central: 27

Platform and build breakdown:

• windows11-64-2009-qr: 28
  • debug: 14
  • opt: 14
• linux2204-64-wayland: 5
  • opt: 5
• windows11-32-2009-qr: 21
  • opt: 9
  • debug: 12
• windows11-32-2009-shippable-qr: 20
  • opt: 20
• linux2204-64-wayland-shippable: 15
  • opt: 15
• windows11-64-2009-shippable-qr: 21
  • opt: 21
• linux1804-64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-16&endday=2023-09-16&tree=all

Comment 102

[Intermittent Failures Robot]

890 failures in 3337 pushes (0.267 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 701
• mozilla-central: 186
• try: 3

Platform and build breakdown:

• windows11-64-2009-qr: 212
  • debug: 98
  • opt: 114
• linux2204-64-wayland: 67
  • debug: 20
  • opt: 47
• windows11-32-2009-shippable-qr: 169
  • opt: 169
• linux2204-64-wayland-shippable: 98
  • opt: 98
• windows11-64-2009-shippable-qr: 155
  • opt: 155
• windows11-32-2009-qr: 184
  • debug: 93
  • opt: 91
• macosx1015-64-qr: 1
  • debug: 1
• linux1804-64-shippable-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-11&endday=2023-09-17&tree=all

Comment 103

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2e0962bc258b
set web-platform webgpu tests as tier 3.

Comment 104

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/2e0962bc258b

Comment 105

[BugBot [:suhaib / :marco/ :calixte]]

Since nightly and release are affected, beta will likely be affected too.
For more information, please visit BugBot documentation.

Comment 106

[Intermittent Failures Robot]

59 failures in 653 pushes (0.09 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 32
• mozilla-central: 27

Platform and build breakdown:

• windows11-64-2009-qr: 16
  • debug: 6
  • opt: 10
• linux2204-64-wayland: 2
  • opt: 2
• windows11-32-2009-qr: 9
  • opt: 6
  • debug: 3
• linux2204-64-wayland-shippable: 9
  • opt: 9
• windows11-64-2009-shippable-qr: 11
  • opt: 11
• windows11-32-2009-shippable-qr: 11
  • opt: 11
• linux1804-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-18&endday=2023-09-18&tree=all

Comment 107

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Joel, some of the tests still run as part of the backlog task. How can the tests run as part of the tier 3 task and no webgpu tests run as part of the backlog task?

Comment 108

[Erich Gubler [:ErichDonGubler]]

:NarcisB, :aryx, :jmaher: I have been attending to this for the last day or so after being on PTO last week, sorry for not updating the NI here.

The main problem we have with the backlog task is the issue I mention in comment 5 on bug 1852952 and comment 4 on bug 1852858; the webgpu-platform-tests-backlog task still runs canvas and webgpu tasks. These tasks can be sensitive to driver and hardware environment, which is why web-platform-test-webgpu was broken out in the first place (CC :jmaher) with their own runner configuration and set of supported platforms. In those comments I also mention patch D188387, which I believe should fix this issue.

I'm sad to downgrade to tier 2, but I have no objection (CC :jimb). I apologize that managing these failures on tier 2 has been enough work to warrant the downgrade.

Comment 109

[Joel Maher ( :jmaher ) (UTC -8)]

I have a patch up to fix the backlog tests ( https://phabricator.services.mozilla.com/D188413 ), but no real traction yet, let me nudge on that.

Comment 110

[Joel Maher ( :jmaher ) (UTC -8)]

you are welcome to steal some of the backlog changes in web-platform.yml and I can remove them from my patch.

Comment 111

[Intermittent Failures Robot]

82 failures in 3850 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #6 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 32
• mozilla-central: 49
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 17
  • debug: 7
  • opt: 10
• linux2204-64-wayland: 2
  • opt: 2
• windows11-32-2009-qr: 9
  • opt: 6
  • debug: 3
• linux2204-64-wayland-shippable: 9
  • opt: 9
• windows11-64-2009-shippable-qr: 23
  • opt: 23
• windows11-32-2009-shippable-qr: 11
  • opt: 11
• linux1804-64-shippable-qr: 11
  • opt: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-18&endday=2023-09-24&tree=all

Comment 112

[Intermittent Failures Robot]

6 failures in 3726 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 6

Platform and build breakdown:

• windows11-64-2009-shippable-qr: 3
  • opt: 3
• linux1804-64-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-09-25&endday=2023-10-01&tree=all

Comment 113

[Intermittent Failures Robot]

15 failures in 4018 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 15

Platform and build breakdown:

• linux1804-64-shippable-qr: 9
  • opt: 9
• windows11-64-2009-shippable-qr: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-10-02&endday=2023-10-08&tree=all

Comment 114

[Intermittent Failures Robot]

3 failures in 3764 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• linux1804-64-ccov-qr: 2
  • opt: 2
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-10-09&endday=2023-10-15&tree=all

Comment 115

[Cosmin Sabou [:CosminS]]

Nicolas, could you please write a patch that would stop running mozilla/tests/webgpu/chunked/** tests as part of the backlog? There was one is https://phabricator.services.mozilla.com/D188387 but got abandoned. From what I gather from Erich Gubler these should not be running.
These are the jobs on mozilla-central that all have permafailures on tests from /_mozilla/webgpu/chunked/. Bugs filed because of these failures are: Bug 1852952, 1859202, 1859198, 1859178, 1847771, 1852858.

Comment 120

[Erich Gubler [:ErichDonGubler]]

:CosminS: This bug is for general issues discovered in WebGPU CTS tests while executing in the expected job(s) and environment(s). AFAIK, the running of WebGPU CTS in WPT backlog tests is a recent regression in tests that the WebGPU team does not own. Therefore, the issues you've pinged :nical on are not relevant to this bug. Could we please combine those duplicate issues into a separate bug?

Comment 121

[Cosmin Sabou [:CosminS]]

Left only Bug 1859178 to track those failures. Being a permafail on several jobs it would be nice to have that patch what would stop running those tests on backlog as stated in https://bugzilla.mozilla.org/show_bug.cgi?id=1852952#c10.

Comment 122

[Intermittent Failures Robot]

5 failures in 3484 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5

Platform and build breakdown:

• android-em-7-0-x86_64-shippable-lite-qr: 4
  • opt: 4
• linux1804-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-10-16&endday=2023-10-22&tree=all

Comment 123

[Intermittent Failures Robot]

6 failures in 3623 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6

Platform and build breakdown:

• linux1804-64-ccov-qr: 3
  • opt: 3
• windows11-64-2009-ccov-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1838695&startday=2023-11-20&endday=2023-11-26&tree=all

Comment 124

[Erich Gubler [:ErichDonGubler]]

RE: comment 123: The test-coverage CI tasks that yielded the above should be running in the same environment as current (tier 3) web-platform-tests-webgpu tests, but they do not. This should be fixed. NI'ing :nataliaCs to get clarity on who I need to contact WRT ownership of those tests.

I don't believe it's useful to keep this bug open, as we (the WebGPU team) changed WebGPU CTS testing to tier 3 specifically to eliminate this bug as an ineffectual hotspot of sheriff and team attention. I'm going to close it, and accept whatever dialogue is necessary to converge stakeholder understanding on my decision post-hoc.

Comment 125

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=437233392&repo=mozilla-central

Comment 126

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=437233401&repo=mozilla-central

Comment 127

[Cosmin Sabou [:CosminS]]

Erich, I think is best to close this as worksforme or fixed so that the failures bot won't reopen it.

NI'ing :nataliaCs to get clarity on who I need to contact WRT ownership of those tests.

Joel, James, can we single out these TCw jobs that run the webgpu tests to be tier 3?

Comment 128

[Joel Maher ( :jmaher ) (UTC -8)]

we could add --exclude-tag=webgpu here:
https://searchfox.org/mozilla-central/source/taskcluster/ci/test/web-platform.yml#589

we might want to consider as well:
--exclude-tag=canvas
--exclude-tag=privatebrowsing

:jgraham, do you have other thoughts?

Comment 129

[James Graham [:jgraham]]

I don't really know what that coverage job is used for (or what the outputs are used for), so I don't really know the impact of disabling specific tests. Assuming the job is useful at all it makes some sense that it would run all tests, so I don't see why we'd want to exclude canvas or privatebrowsing. Maybe webgpu is different if those tests really don't run at all without hardware support.