1800035 - Intermittent Windows Marionette Return code: 10

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=394617514&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/KBV8B0OfTMmWo3ZbCco2Sg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2022-10-27T23:09:43.867Z] 23:09:43     INFO -  DEBUG: Starting phase profile-before-change
[task 2022-10-27T23:09:43.873Z] 23:09:43     INFO -  DEBUG: Spinning the event loop
[task 2022-10-27T23:09:43.874Z] 23:09:43     INFO -  DEBUG: Completed blocker EnvironmentAddonBuilder for phase AddonManager: Waiting to start provider shutdown.
[task 2022-10-27T23:09:43.874Z] 23:09:43     INFO -  DEBUG: Completed blocker PageActions: purging unregistered actions from cache for phase profile-before-change
[task 2022-10-27T23:09:43.874Z] 23:09:43     INFO -  DEBUG: Completed blocker Places Expiration: shutdown for phase Places Connection shutdown
[task 2022-10-27T23:09:43.875Z] 23:09:43     INFO -  DEBUG: Completed blocker JSON store: writing data for phase IOUtils: waiting for profileBeforeChange IO to complete
[task 2022-10-27T23:09:43.876Z] 23:09:43     INFO -  DEBUG: Completed blocker JSON store: writing data for phase IOUtils: waiting for profileBeforeChange IO to complete
[task 2022-10-27T23:09:43.876Z] 23:09:43     INFO -  DEBUG: Completed blocker JSON store: writing data for phase IOUtils: waiting for profileBeforeChange IO to complete
[task 2022-10-27T23:09:43.876Z] 23:09:43     INFO -  DEBUG: Completed blocker JSON store: writing data for phase IOUtils: waiting for profileBeforeChange IO to complete
[task 2022-10-27T23:09:43.877Z] 23:09:43     INFO -  DEBUG: Completed blocker RemoteSettingsClient - finish IDB access. for phase profile-before-change
[task 2022-10-27T23:09:43.877Z] 23:09:43     INFO -  DEBUG: Completed blocker Remote Settings profile-before-change for phase profile-before-change
[task 2022-10-27T23:09:43.877Z] 23:09:43     INFO -  DEBUG: Completed blocker DownloadAutoSaveView: writing data for phase profile-before-change
[task 2022-10-27T23:09:43.878Z] 23:09:43     INFO -  DEBUG: Completed blocker Search service: shutting down for phase IOUtils: waiting for profileBeforeChange IO to complete
[task 2022-10-27T23:09:43.878Z] 23:09:43     INFO -  DEBUG: Completed blocker places.sqlite#1: waiting for shutdown for phase Sqlite.sys.mjs: wait until all connections are closed
[task 2022-10-27T23:09:43.878Z] 23:09:43     INFO -  DEBUG: Completed blocker OS.File: flush I/O queued before profileBeforeChange for phase profile-before-change
[task 2022-10-27T23:09:43.906Z] 23:09:43     INFO -  DEBUG: Completed blocker PlacesUtils wrapped connection closing as part of Places shutdown for phase Places Connection shutdown
[task 2022-10-27T23:09:43.914Z] 23:09:43     INFO -  DEBUG: Completed blocker PlacesUtils wrapped connection must be closed before Sqlite.sys.mjs for phase Sqlite.sys.mjs: wait until all clients have completed their task
[task 2022-10-27T23:09:43.915Z] 23:09:43     INFO -  DEBUG: Completed blocker SitePermsAddonProvider for phase AddonManager: Waiting for providers to shut down.
[task 2022-10-27T23:09:43.915Z] 23:09:43     INFO -  DEBUG: Completed blocker GMPProvider for phase AddonManager: Waiting for providers to shut down.
[task 2022-10-27T23:09:43.915Z] 23:09:43     INFO -  DEBUG: Completed blocker XPIProvider for phase AddonManager: Waiting for providers to shut down.
[task 2022-10-27T23:09:43.915Z] 23:09:43     INFO -  DEBUG: Completed blocker AddonManager: shutting down. for phase profile-before-change
[task 2022-10-27T23:09:43.916Z] 23:09:43     INFO -  DEBUG: Completed blocker CrashMonitor: Writing notifications to file after receiving profile-before-change and awaiting all checkpoints written for phase IOUtils: waiting for profileBeforeChange IO to complete
[task 2022-10-27T23:09:43.922Z] 23:09:43     INFO -  DEBUG: Completed blocker content-prefs.sqlite#0: waiting for shutdown for phase Sqlite.sys.mjs: wait until all connections are closed
[task 2022-10-27T23:09:43.924Z] 23:09:43     INFO -  DEBUG: Completed blocker Closing ContentPrefService2 connection. for phase Sqlite.sys.mjs: wait until all clients have completed their task
[task 2022-10-27T23:09:43.924Z] 23:09:43     INFO -  DEBUG: Completed blocker places.sqlite#0: waiting for shutdown for phase Sqlite.sys.mjs: wait until all connections are closed
[task 2022-10-27T23:09:43.924Z] 23:09:43     INFO -  DEBUG: Completed blocker PlacesUtils read-only connection closing as part of Places shutdown for phase Places Connection shutdown
[task 2022-10-27T23:09:43.924Z] 23:09:43     INFO -  DEBUG: Completed blocker PlacesUtils read-only connection must be closed before Sqlite.sys.mjs for phase Sqlite.sys.mjs: wait until all clients have completed their task
[task 2022-10-27T23:09:43.927Z] 23:09:43     INFO -  DEBUG: Completed blocker Sqlite.sys.mjs shutdown blocker for phase profile-before-change
[task 2022-10-27T23:09:43.937Z] 23:09:43     INFO -  DEBUG: Finished phase profile-before-change
[task 2022-10-27T23:09:43.958Z] 23:09:43     INFO -  DEBUG: Starting phase profile-before-change-telemetry
[task 2022-10-27T23:09:43.964Z] 23:09:43     INFO -  DEBUG: Spinning the event loop
[task 2022-10-27T23:09:44.247Z] 23:09:44     INFO -  DEBUG: Adding blocker Waiting for ping task for phase TelemetryController: Waiting for pending ping activity
[task 2022-10-27T23:09:44.300Z] 23:09:44     INFO -  DEBUG: Adding blocker Waiting for ping task for phase TelemetryController: Waiting for pending ping activity
[task 2022-10-27T23:09:44.322Z] 23:09:44     INFO -  DEBUG: Completed blocker Waiting for ping task for phase TelemetryController: Waiting for pending ping activity
[task 2022-10-27T23:09:44.328Z] 23:09:44     INFO -  DEBUG: Completed blocker Waiting for ping task for phase TelemetryController: Waiting for pending ping activity
[task 2022-10-27T23:09:44.340Z] 23:09:44     INFO -  DEBUG: Completed blocker TelemetryController: shutting down for phase profile-before-change-telemetry
[task 2022-10-27T23:09:44.341Z] 23:09:44     INFO -  DEBUG: Finished phase profile-before-change-telemetry
[task 2022-10-27T23:09:44.342Z] 23:09:44     INFO -  DEBUG: Starting phase xpcom-will-shutdown
[task 2022-10-27T23:09:44.342Z] 23:09:44     INFO -  DEBUG: Spinning the event loop
[task 2022-10-27T23:09:44.346Z] 23:09:44     INFO -  DEBUG: Completed blocker OS.File: flush pending requests, warn about unclosed files, shut down service. for phase xpcom-will-shutdown
[task 2022-10-27T23:09:44.346Z] 23:09:44     INFO -  DEBUG: Finished phase xpcom-will-shutdown
[task 2022-10-27T23:09:44.349Z] 23:09:44     INFO -  DEBUG: Starting phase web-workers-shutdown
[task 2022-10-27T23:09:44.349Z] 23:09:44     INFO -  DEBUG: Spinning the event loop
[task 2022-10-27T23:09:44.350Z] 23:09:44     INFO -  DEBUG: Finished phase web-workers-shutdown
[task 2022-10-27T23:09:46.561Z] 23:09:46    ERROR - Return code: 10
[task 2022-10-27T23:09:46.561Z] 23:09:46     INFO - TinderboxPrint: marionette<br/>1013/<em class="testfail">1</em>/36
[task 2022-10-27T23:09:46.561Z] 23:09:46     INFO - gecko.log not found
[task 2022-10-27T23:09:46.561Z] 23:09:46     INFO - TinderboxPrint: marionette<br/>1013/<em class="testfail">1</em>/36
[task 2022-10-27T23:09:46.561Z] 23:09:46     INFO - Marionette exited with return code 10: WARNING
[task 2022-10-27T23:09:46.561Z] 23:09:46  WARNING - # TBPL WARNING #
[task 2022-10-27T23:09:46.561Z] 23:09:46  WARNING - setting return code to 1
[task 2022-10-27T23:09:46.561Z] 23:09:46     INFO - Running post-action listener: _package_coverage_data
[task 2022-10-27T23:09:46.561Z] 23:09:46     INFO - Running post-action listener: _resource_record_post_action
[task 2022-10-27T23:09:46.562Z] 23:09:46     INFO - Running post-action listener: process_java_coverage_data
[task 2022-10-27T23:09:46.562Z] 23:09:46     INFO - [mozharness: 2022-10-27 23:09:46.562432Z] Finished run-tests step (success)
[task 2022-10-27T23:09:46.562Z] 23:09:46     INFO - Running post-run listener: _resource_record_post_run
[task 2022-10-27T23:09:46.762Z] 23:09:46     INFO - Total resource usage - Wall time: 2775s; CPU: 42%; Read bytes: 306483200; Write bytes: 6099653632; Read time: 0; Write time: 149
[task 2022-10-27T23:09:46.762Z] 23:09:46     INFO - TinderboxPrint: CPU usage<br/>41.7%
[task 2022-10-27T23:09:46.762Z] 23:09:46     INFO - TinderboxPrint: I/O read bytes / time<br/>306,483,200 / 0
[task 2022-10-27T23:09:46.762Z] 23:09:46     INFO - TinderboxPrint: I/O write bytes / time<br/>6,099,653,632 / 149
[task 2022-10-27T23:09:46.762Z] 23:09:46     INFO - TinderboxPrint: CPU idle<br/>12,960.9 (58.3%)
[task 2022-10-27T23:09:46.762Z] 23:09:46     INFO - TinderboxPrint: CPU system<br/>2,466.9 (11.1%)
[task 2022-10-27T23:09:46.762Z] 23:09:46     INFO - TinderboxPrint: CPU user<br/>6,772.3 (30.5%)
[task 2022-10-27T23:09:46.775Z] 23:09:46     INFO - install - Wall time: 10s; CPU: 25%; Read bytes: 73728; Write bytes: 394252288; Read time: 0; Write time: 25
[task 2022-10-27T23:09:46.787Z] 23:09:46     INFO - run-tests - Wall time: 2766s; CPU: 42%; Read bytes: 305488896; Write bytes: 5612545024; Read time: 0; Write time: 122
[task 2022-10-27T23:09:47.431Z] 23:09:47  WARNING - returning nonzero exit status 1
[taskcluster 2022-10-27T23:09:47.514Z]    Exit Code: 1
[taskcluster 2022-10-27T23:09:47.514Z]    User Time: 31.25ms
[taskcluster 2022-10-27T23:09:47.514Z]  Kernel Time: 15.625ms
[taskcluster 2022-10-27T23:09:47.514Z]    Wall Time: 47m43.1632606s
[taskcluster 2022-10-27T23:09:47.514Z]       Result: FAILED
[taskcluster 2022-10-27T23:09:47.514Z] === Task Finished ===
[taskcluster 2022-10-27T23:09:47.514Z] Task Duration: 47m43.1666893s
[taskcluster 2022-10-27T23:09:47.611Z] Uploading artifact public/logs/localconfig.json from file logs\localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2023-01-25T22:02:42.793Z
[taskcluster 2022-10-27T23:09:47.772Z] Uploading artifact public/test_info/marionette_errorsummary.log from file build\blobber_upload_dir\marionette_errorsummary.log with content encoding "gzip", mime type "text/plain" and expiry 2023-01-25T22:02:42.793Z
[taskcluster 2022-10-27T23:09:48.602Z] Uploading artifact public/test_info/marionette_raw.log from file build\blobber_upload_dir\marionette_raw.log with content encoding "gzip", mime type "text/plain" and expiry 2023-01-25T22:02:42.793Z
[taskcluster 2022-10-27T23:09:48.888Z] Uploading artifact public/test_info/report.html from file build\blobber_upload_dir\report.html with content encoding "gzip", mime type "text/html; charset=utf-8" and expiry 2023-01-25T22:02:42.793Z
[taskcluster 2022-10-27T23:09:48.995Z] Uploading artifact public/test_info/resource-usage.json from file build\blobber_upload_dir\resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2023-01-25T22:02:42.793Z
[taskcluster 2022-10-27T23:09:49.481Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2023-01-25T22:02:42.793Z
[taskcluster:error] exit status 1

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The Marionette harness will exit with an code of 10 when tests were failing. Interestingly I see:

[task 2022-10-27T23:09:42.639Z] 23:09:42     INFO -  passed: 1013
[task 2022-10-27T23:09:42.639Z] 23:09:42     INFO -  failed: 1
[task 2022-10-27T23:09:42.639Z] 23:09:42     INFO -  todo: 36 (skipped: 30)
[task 2022-10-27T23:09:42.640Z] 23:09:42     INFO -  FAILED TESTS

But there is no failure listed in the parsed log. No idea what's broken here given that we haven't changed anything in that code for quite some time. Maybe there was a Python upgrade recently for test workers which might have caused that behavior?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

the base win10 image hasn't changed. As this is intermittent and on ASAN only, this makes me wonder if we have some timing issue. For successful runs I see test summary of 1012/0/36 (pass/fail/skip-todo), but for the failures I see (1013/1/36). I don't know if that means anything.

Quite likely the browser could have changed some timing in shutdown.

This job (win10/asan-marionette) fails 12.5% of the time in the last month, it is broken down in these bugs:

bug id	frequency
1782045	10
1791299	9
1800035	6
1797552	2
1792855	2
1691876	2
1790942	2
1795485	1
1798158	1
1413849	1
1780602	1
1669403	1
1727152	1

while this bug isn't the most frequent, it might be the most confusing.

Comment 3

[Intermittent Failures Robot]

2 failures in 3445 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• windows10-64-2004-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2022-11-07&endday=2022-11-13&tree=all

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #2)

successful runs I see test summary of 1012/0/36 (pass/fail/skip-todo), but for the failures I see (1013/1/36). I don't know if that means anything.

That's interesting. Why do we have one time 1012 tests and another time 1014 tests that have been running? It might be good to check and compare different logs from passing and failing jobs to see which tests are run on top in failing cases. But I would defer that until the failure rate is high enough.

Comment 5

[Natalia Csoregi [:nataliaCs]]

First occurrence so far: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=Windows%2C10%2Cx64%2C2004%2Casan%2CWebRender%2Copt%2Ctest-windows10-64-2004-asan-qr%2Fopt-marionette%2CMn&tochange=2dddf127c6ab66595268672db7088712f1322aff&fromchange=50a2d35fa2324ea846a385390b18e0ab5d211fcc&selectedTaskRun=SdTmi2p2SO2ijNEYvYFbsw.0

Comment 6

[Intermittent Failures Robot]

4 failures in 3589 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 2

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • debug: 1
• windows10-64-2004-asan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2022-11-14&endday=2022-11-20&tree=all

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I cannot see backfill jobs but only Mn ones. Given that on autoland the test order might change and have an affect I triggered new backfills now:

https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=Windows%2C10%2Cx64%2C2004%2Casan%2CWebRender%2Copt%2Ctest-windows10-64-2004-asan-qr%2Fopt-marionette%2CMn&tochange=d1982cee06ca5c4df2bd03dc4f47ee489f8b5a17&fromchange=50a2d35fa2324ea846a385390b18e0ab5d211fcc

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

No new conclusion from running additional jobs. It's still the above changelog which isn't helpful. I doubt the change to devtools might have caused that.

Whenever checking the full log there is a single failure present for:

[task 2022-11-15T00:12:54.928Z] 00:12:54 INFO - TEST-FAIL | testing/marionette/harness/marionette_harness/tests/unit/test_expectedfail.py TestFail.test_fails | took 135ms

But as you can see this one is expected. So not sure what's causing us to fail with an exit code of 10 at all.

Comment 9

[Intermittent Failures Robot]

9 failures in 3888 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 5
• try: 2

Platform and build breakdown:

• windows10-64-2004-asan-qr: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2022-11-21&endday=2022-11-27&tree=all

Comment 10

[Intermittent Failures Robot]

10 failures in 3938 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• try: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows10-64-2004-asan-qr: 9
  • opt: 9
• windows10-64-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2022-11-28&endday=2022-12-04&tree=all

Comment 11

[Intermittent Failures Robot]

5 failures in 3510 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 2

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • debug: 1
• windows10-64-2004-qr: 1
  • debug: 1
• windows10-64-2004-asan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2022-12-05&endday=2022-12-11&tree=all

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Maybe the following errors for Telemetry during shutdown might not be related to this particular error in Marionette but might warrant investigation and separate bugs to be filed.

https://treeherder.mozilla.org/logviewer?job_id=399246634&repo=mozilla-central&lineNumber=155239

Chris, could someone from the Telemetry team please have a look? Thanks.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I actually filed bug 1805153 and moved the needinfo over there.

Comment 14

[Intermittent Failures Robot]

3 failures in 3558 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• windows10-64-2004-asan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2022-12-12&endday=2022-12-18&tree=all

Comment 15

[Intermittent Failures Robot]

1 failures in 2999 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64-2004-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2022-12-19&endday=2022-12-25&tree=all

Comment 16

[Intermittent Failures Robot]

3 failures in 1328 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• windows10-64-2004-asan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2022-12-26&endday=2023-01-01&tree=all

Comment 17

[Intermittent Failures Robot]

3 failures in 2607 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-beta: 2

Platform and build breakdown:

• windows10-64-2004-asan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-01-02&endday=2023-01-08&tree=all

Comment 18

[Intermittent Failures Robot]

1 failures in 3530 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64-2004-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-01-09&endday=2023-01-15&tree=all

Comment 19

[Intermittent Failures Robot]

3 failures in 3717 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• windows10-64-2004-asan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-01-16&endday=2023-01-22&tree=all

Comment 20

[Intermittent Failures Robot]

3 failures in 3914 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-asan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-01-23&endday=2023-01-29&tree=all

Comment 21

[Intermittent Failures Robot]

2 failures in 3606 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-asan-qr: 1
  • opt: 1
• windows10-64-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-01-30&endday=2023-02-05&tree=all

Comment 22

[BMO Automation]

Moving bug to Testing::Marionette Client and Harness component per bug 1815831.

Comment 23

[Intermittent Failures Robot]

1 failures in 3643 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-02-20&endday=2023-02-26&tree=all

Comment 24

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 25

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=407365560&repo=autoland

Comment 26

[Intermittent Failures Robot]

2 failures in 3863 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-03-27&endday=2023-04-02&tree=all

Comment 27

[Intermittent Failures Robot]

3 failures in 3386 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 2

Platform and build breakdown:

• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-04-03&endday=2023-04-09&tree=all

Comment 28

[Intermittent Failures Robot]

2 failures in 3206 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1
• windows11-32-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-04-10&endday=2023-04-16&tree=all

Comment 30

[Intermittent Failures Robot]

2 failures in 3177 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-central: 1

Platform and build breakdown:

• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-04-24&endday=2023-04-30&tree=all

Comment 31

[Intermittent Failures Robot]

1 failures in 3411 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-05-01&endday=2023-05-07&tree=all

Comment 32

[Intermittent Failures Robot]

4 failures in 3578 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• try: 1
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-05-15&endday=2023-05-21&tree=all

Comment 33

[Intermittent Failures Robot]

2 failures in 3946 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-05-22&endday=2023-05-28&tree=all

Comment 34

[Intermittent Failures Robot]

2 failures in 3471 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-05-29&endday=2023-06-04&tree=all

Comment 35

[Intermittent Failures Robot]

5 failures in 3885 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5

Platform and build breakdown:

• windows11-32-2009-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-06-05&endday=2023-06-11&tree=all

Comment 37

[Intermittent Failures Robot]

9 failures in 4079 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• try: 1
• mozilla-central: 2

Platform and build breakdown:

• windows11-32-2009-qr: 5
  • debug: 5
• linux1804-64-qr: 1
  • debug: 1
• windows11-64-2009-qr: 2
  • debug: 2
• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-06-12&endday=2023-06-18&tree=all

Comment 38

[Intermittent Failures Robot]

1 failures in 3880 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-06-19&endday=2023-06-25&tree=all

Comment 40

[Intermittent Failures Robot]

9 failures in 3594 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• try: 2

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • debug: 4
• windows11-64-2009-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-06-26&endday=2023-07-02&tree=all

Comment 41

[Intermittent Failures Robot]

9 failures in 2956 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• try: 1
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 4
• windows11-32-2009-qr: 4
  • debug: 4
• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-07-03&endday=2023-07-09&tree=all

Comment 42

[Intermittent Failures Robot]

8 failures in 3309 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-esr115: 2
• autoland: 4

Platform and build breakdown:

• windows11-64-2009-asan-qr: 1
  • opt: 1
• windows11-32-2009-qr: 6
  • debug: 6
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-07-10&endday=2023-07-16&tree=all

Comment 43

[Intermittent Failures Robot]

4 failures in 3520 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 2

Platform and build breakdown:

• windows11-32-2009-qr: 3
  • debug: 3
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-07-17&endday=2023-07-23&tree=all

Comment 44

[Intermittent Failures Robot]

6 failures in 3403 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• autoland: 2
• mozilla-central: 2

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• windows11-64-2009-asan-qr: 1
  • opt: 1
• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-07-24&endday=2023-07-30&tree=all

Comment 45

[Intermittent Failures Robot]

14 failures in 3276 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• try: 1
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-asan-qr: 4
  • opt: 4
• windows11-64-2009-qr: 5
  • debug: 5
• windows11-32-2009-qr: 4
  • debug: 4
• windows10-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-07-31&endday=2023-08-06&tree=all

Comment 46

[Intermittent Failures Robot]

8 failures in 3341 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• mozilla-central: 2
• mozilla-beta: 2
• autoland: 2

Platform and build breakdown:

• windows11-32-2009-qr: 6
  • debug: 6
• windows11-64-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-08-07&endday=2023-08-13&tree=all

Comment 47

[Intermittent Failures Robot]

12 failures in 3076 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• try: 1
• mozilla-beta: 2
• mozilla-central: 1
• mozilla-esr115: 1

Platform and build breakdown:

• windows11-64-2009-qr: 5
  • debug: 5
• windows11-64-2009-asan-qr: 3
  • opt: 3
• windows11-32-2009-qr: 4
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-08-14&endday=2023-08-20&tree=all

Comment 48

[Intermittent Failures Robot]

7 failures in 2218 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• try: 3

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-32-2009-qr: 3
  • debug: 3
• windows11-64-2009-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-08-21&endday=2023-08-27&tree=all

Comment 49

[Intermittent Failures Robot]

11 failures in 3019 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 5
• mozilla-esr115: 1
• mozilla-beta: 1
• try: 1

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • debug: 4
• windows11-64-2009-asan-qr: 3
  • opt: 3
• windows11-64-2009-qr: 4
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-08-28&endday=2023-09-03&tree=all

Comment 50

[Intermittent Failures Robot]

7 failures in 3267 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 3
• mozilla-central: 2
• autoland: 1
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 4
• windows11-64-2009-asan-qr: 1
  • opt: 1
• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-09-04&endday=2023-09-10&tree=all

Comment 51

[Intermittent Failures Robot]

10 failures in 3850 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-esr115: 2
• mozilla-central: 3

Platform and build breakdown:

• windows11-32-2009-qr: 6
  • debug: 6
• linux1804-64-qr: 1
  • debug: 1
• windows11-64-2009-qr: 1
  • debug: 1
• windows11-64-2009-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-09-18&endday=2023-09-24&tree=all

Comment 52

[Intermittent Failures Robot]

9 failures in 3726 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 1
• mozilla-esr115: 1

Platform and build breakdown:

• windows11-64-2009-asan-qr: 4
  • opt: 4
• windows11-32-2009-qr: 3
  • debug: 3
• windows11-64-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-09-25&endday=2023-10-01&tree=all

Comment 53

[Intermittent Failures Robot]

12 failures in 4018 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-beta: 1
• mozilla-central: 2
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• windows11-64-2009-asan-qr: 3
  • opt: 3
• windows11-64-2009-devedition-qr: 1
  • opt: 1
• windows11-32-2009-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-10-02&endday=2023-10-08&tree=all

Comment 54

[Intermittent Failures Robot]

4 failures in 3764 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • opt: 1
  • debug: 1
• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-10-09&endday=2023-10-15&tree=all

Comment 55

[Intermittent Failures Robot]

11 failures in 3484 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• try: 1

Platform and build breakdown:

• windows11-32-2009-qr: 11
  • debug: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-10-16&endday=2023-10-22&tree=all

Comment 56

[Intermittent Failures Robot]

12 failures in 3657 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 10
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-qr: 7
  • debug: 7
• windows11-32-2009-qr: 4
  • debug: 4
• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-10-23&endday=2023-10-29&tree=all

Comment 57

[Intermittent Failures Robot]

10 failures in 3356 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-beta: 2

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• windows11-32-2009-qr: 7
  • debug: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-10-30&endday=2023-11-05&tree=all

Comment 58

[Intermittent Failures Robot]

12 failures in 3288 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-beta: 2

Platform and build breakdown:

• windows11-32-2009-qr: 8
  • debug: 8
• windows11-64-2009-asan-qr: 1
  • opt: 1
• windows11-64-2009-qr: 2
  • debug: 1
  • opt: 1
• windows11-64-2009-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-11-06&endday=2023-11-12&tree=all

Comment 59

[Intermittent Failures Robot]

7 failures in 3495 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 5
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 2
  • opt: 1
• windows11-64-2009-asan-qr: 1
  • opt: 1
• linux1804-64-qr: 1
  • debug: 1
• linux1804-64-asan-qr: 1
  • opt: 1
• windows10-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-11-13&endday=2023-11-19&tree=all

Comment 60

[Intermittent Failures Robot]

6 failures in 3623 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 2
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-asan-qr: 2
  • opt: 2
• windows11-64-2009-qr: 3
  • debug: 3
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-11-20&endday=2023-11-26&tree=all

Comment 61

[Intermittent Failures Robot]

3 failures in 4108 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• try: 1

Platform and build breakdown:

• windows11-64-2009-asan-qr: 2
  • opt: 2
• android-hw-a51-11-0-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-11-27&endday=2023-12-03&tree=all

Comment 62

[Intermittent Failures Robot]

11 failures in 3671 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 6
• mozilla-beta: 1
• try: 2

Platform and build breakdown:

• android-hw-a51-11-0-aarch64-shippable-qr: 1
  • opt: 1
• windows11-64-2009-qr: 5
  • debug: 5
• windows11-32-2009-qr: 3
  • debug: 2
  • opt: 1
• windows11-64-2009-devedition-qr: 1
  • opt: 1
• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-12-04&endday=2023-12-10&tree=all

Comment 63

[Intermittent Failures Robot]

15 failures in 4224 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-central: 4

Platform and build breakdown:

• windows11-32-2009-qr: 3
  • debug: 2
  • opt: 1
• windows2012-64-shippable: 1
  • opt: 1
• windows11-64-2009-qr: 3
  • debug: 3
• android-hw-a51-11-0-aarch64-shippable-qr: 5
  • opt: 5
• windows11-64-2009-asan-qr: 2
  • opt: 2
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-12-11&endday=2023-12-17&tree=all

Comment 64

[Intermittent Failures Robot]

8 failures in 3349 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 2
• mozilla-beta: 2

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • opt: 3
  • debug: 1
• windows11-64-2009-qr: 1
  • debug: 1
• windows11-32-2009-shippable-qr: 1
  • opt: 1
• windows2012-64-shippable: 1
  • opt: 1
• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-12-18&endday=2023-12-24&tree=all

Comment 65

[Intermittent Failures Robot]

1 failures in 1162 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2023-12-25&endday=2023-12-31&tree=all

Comment 66

[Intermittent Failures Robot]

10 failures in 2274 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-central: 2
• autoland: 5
• mozilla-esr115: 1
• try: 1

Platform and build breakdown:

• windows11-32-2009-qr: 3
  • debug: 3
• android-hw-a51-11-0-aarch64-shippable-qr: 1
  • opt: 1
• windows11-64-2009-qr: 6
  • debug: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-01-01&endday=2024-01-07&tree=all

Comment 67

[Intermittent Failures Robot]

7 failures in 3217 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 1
• mozilla-beta: 2

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 4
• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-01-08&endday=2024-01-14&tree=all

Comment 68

[Intermittent Failures Robot]

3 failures in 3599 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1
• windows11-64-2009-asan-qr: 1
  • opt: 1
• android-hw-p5-13-0-android-aarch64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-01-15&endday=2024-01-21&tree=all

Comment 69

[Intermittent Failures Robot]

5 failures in 4034 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-01-22&endday=2024-01-28&tree=all

Comment 70

[Intermittent Failures Robot]

6 failures in 3952 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-beta: 2

Platform and build breakdown:

• windows11-64-2009-asan-qr: 2
  • opt: 2
• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-01-29&endday=2024-02-04&tree=all

Comment 71

[Intermittent Failures Robot]

7 failures in 4009 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• try: 1
• mozilla-esr115: 3

Platform and build breakdown:

• windows11-64-2009-asan-qr: 2
  • opt: 2
• windows11-32-2009-qr: 3
  • debug: 3
• windows11-64-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-02-05&endday=2024-02-11&tree=all

Comment 72

[Intermittent Failures Robot]

3 failures in 3125 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• windows11-64-2009-asan-qr: 1
  • opt: 1
• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-02-12&endday=2024-02-18&tree=all

Comment 73

[Intermittent Failures Robot]

11 failures in 3382 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-beta: 2
• mozilla-central: 2
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-64-2009-asan-qr: 5
  • opt: 5
• windows11-32-2009-qr: 3
  • debug: 3
• windows2012-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-02-19&endday=2024-02-25&tree=all

Comment 74

[Intermittent Failures Robot]

6 failures in 4052 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-beta: 2
• try: 1

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • debug: 4
• windows11-64-2009-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-02-26&endday=2024-03-03&tree=all

Comment 76

[Intermittent Failures Robot]

13 failures in 4198 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 4
• mozilla-esr115: 1
• try: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-asan-qr: 1
  • opt: 1
• windows11-64: 2
  • opt: 2
• windows11-32-2009-qr: 5
  • opt: 1
  • debug: 4
• windows11-64-2009-qr: 2
  • debug: 2
• windows10-64-shippable-qr: 1
  • opt: 1
• macosx1015-64-qr: 1
  • debug: 1
• windows11-32-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-03-04&endday=2024-03-10&tree=all

Comment 77

[Intermittent Failures Robot]

7 failures in 3849 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• try: 1
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 2
  • opt: 1
• windows11-32-2009-qr: 1
  • debug: 1
• windows11-64-2009-asan-qr: 2
  • opt: 2
• windows2012-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-03-11&endday=2024-03-17&tree=all

Comment 78

[Intermittent Failures Robot]

8 failures in 3791 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-beta: 1
• mozilla-central: 3

Platform and build breakdown:

• windows11-64-2009-asan-qr: 2
  • opt: 2
• windows11-32-2009-qr: 1
  • debug: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows2012-64-shippable: 2
  • opt: 2
• windows11-64: 1
  • opt: 1
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-03-18&endday=2024-03-24&tree=all

Comment 79

[Intermittent Failures Robot]

8 failures in 3590 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-beta: 1
• mozilla-central: 1

Platform and build breakdown:

• windows11-64: 1
  • opt: 1
• windows11-32-2009-qr: 5
  • debug: 5
• windows11-64-2009-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-03-25&endday=2024-03-31&tree=all

Comment 80

[Intermittent Failures Robot]

7 failures in 3805 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1
• windows11-32-2009-qr: 4
  • opt: 1
  • debug: 3
• windows11-64-2009-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-04-01&endday=2024-04-07&tree=all

Comment 81

[Intermittent Failures Robot]

5 failures in 3235 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 3
• try: 1

Platform and build breakdown:

• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-qr: 2
  • debug: 1
  • opt: 1
• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-04-08&endday=2024-04-14&tree=all

Comment 82

[Intermittent Failures Robot]

8 failures in 4252 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1
• mozilla-beta: 1
• autoland: 4
• mozilla-release: 1
• mozilla-central: 1

Platform and build breakdown:

• windows11-32-2009-qr: 6
  • debug: 6
• windows11-64-2009-asan-qr: 1
  • opt: 1
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-04-15&endday=2024-04-21&tree=all

Comment 83

[Intermittent Failures Robot]

26 failures in 4336 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 13
• mozilla-beta: 5
• mozilla-esr115: 1
• mozilla-central: 7

Platform and build breakdown:

• windows11-64-2009-qr: 7
  • debug: 6
  • opt: 1
• windows11-32-2009-qr: 8
  • debug: 8
• windows11-64-2009-asan-qr: 6
  • opt: 6
• windows2012-64-shippable: 3
  • opt: 3
• windows11-64-2009-devedition-qr: 1
  • opt: 1
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-04-22&endday=2024-04-28&tree=all

Comment 84

[Intermittent Failures Robot]

28 failures in 4059 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 3
• autoland: 14
• mozilla-esr115: 5
• mozilla-central: 6

Platform and build breakdown:

• windows11-64-2009-asan-qr: 5
  • opt: 5
• windows11-64-2009-qr: 9
  • debug: 9
• windows11-32-2009-qr: 9
  • debug: 9
• android-hw-a51-11-0-aarch64-shippable-qr: 3
  • opt: 3
• windows10-64-shippable-qr: 1
  • opt: 1
• windows2012-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-04-29&endday=2024-05-05&tree=all

Comment 85

[Intermittent Failures Robot]

13 failures in 3649 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 8
• try: 1

Platform and build breakdown:

• windows2012-64-shippable: 3
  • opt: 3
• windows11-64-2009-qr: 5
  • debug: 5
• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-asan-qr: 2
  • opt: 2
• android-hw-a51-11-0-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-05-06&endday=2024-05-12&tree=all

Comment 86

[Intermittent Failures Robot]

17 failures in 4547 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 4
• autoland: 9
• mozilla-central: 2
• try: 2

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • debug: 4
• windows11-32-2009-qr: 9
  • debug: 9
• windows11-64-2009-asan-qr: 3
  • opt: 3
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-05-13&endday=2024-05-19&tree=all

Comment 87

[Intermittent Failures Robot]

18 failures in 3995 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 13
• mozilla-beta: 1
• mozilla-central: 3

Platform and build breakdown:

• windows11-32-2009-qr: 7
  • debug: 7
• windows11-64-2009-asan-qr: 4
  • opt: 4
• windows11-64-2009-qr: 5
  • debug: 5
• android-hw-a51-11-0-aarch64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-05-20&endday=2024-05-26&tree=all

Comment 88

[Intermittent Failures Robot]

16 failures in 4202 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• mozilla-beta: 3
• mozilla-central: 1

Platform and build breakdown:

• windows11-32-2009-qr: 11
  • debug: 11
• windows11-64-2009-asan-qr: 2
  • opt: 2
• windows11-64-2009-qr: 2
  • debug: 2
• android-hw-p5-13-0-android-aarch64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-05-27&endday=2024-06-02&tree=all

Comment 89

[Intermittent Failures Robot]

10 failures in 3649 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• try: 2
• mozilla-central: 2

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• android-hw-p5-13-0-android-aarch64-qr: 2
  • opt: 2
• windows11-32-2009-qr: 4
  • debug: 4
• android-hw-a51-11-0-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-06-03&endday=2024-06-09&tree=all

Comment 90

[Intermittent Failures Robot]

13 failures in 4625 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-esr115: 1
• mozilla-beta: 3
• try: 2
• mozilla-central: 2

Platform and build breakdown:

• windows11-32-2009-qr: 5
  • debug: 5
• windows11-64-2009-asan-qr: 3
  • opt: 3
• windows11-64-2009-qr: 2
  • debug: 2
• android-hw-p5-13-0-android-aarch64-qr: 2
  • opt: 2
• windows10-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-06-10&endday=2024-06-16&tree=all

Comment 91

[Intermittent Failures Robot]

21 failures in 4577 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 5
• mozilla-beta: 3
• try: 2
• autoland: 9
• mozilla-central: 2

Platform and build breakdown:

• windows11-64-2009-qr: 6
  • debug: 6
• windows11-32-2009-qr: 12
  • debug: 12
• windows11-64: 2
  • opt: 2
• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-06-17&endday=2024-06-23&tree=all

Comment 92

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This seems to be a mozharness related issue given that it surfaces around log parsing for tinderbox. We have two lines of code that could cause the return code of 10:

https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/base/script.py#1664 (run_command)
https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/base/script.py#1870 (get_output_from_command)

I assume that the failure might come from parsing the log files and checking for failures.

Comment 93

[Intermittent Failures Robot]

26 failures in 4116 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 9
• autoland: 9
• mozilla-central: 2
• mozilla-release: 2
• try: 4

Platform and build breakdown:

• android-hw-p5-13-0-android-aarch64-qr: 10
  • opt: 10
• windows11-32-2009-qr: 12
  • debug: 12
• windows11-64-2009-qr: 2
  • debug: 2
• windows11-64-2009-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-06-24&endday=2024-06-30&tree=all

Comment 94

[Intermittent Failures Robot]

7 failures in 3199 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 1
• try: 2

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • debug: 4
• windows11-64-2009-qr: 1
  • debug: 1
• windows11-64-2009-asan-qr: 1
  • opt: 1
• linux2204-64-wayland: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-07-01&endday=2024-07-07&tree=all

Comment 95

[Intermittent Failures Robot]

12 failures in 3780 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-beta: 2
• autoland: 6
• mozilla-central: 3

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• android-hw-p5-13-0-android-aarch64-qr: 2
  • opt: 2
• windows11-32-2009-qr: 5
  • debug: 5
• windows11-64-2009-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-07-08&endday=2024-07-14&tree=all

Comment 96

[Intermittent Failures Robot]

6 failures in 3799 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-beta: 1
• try: 1

Platform and build breakdown:

• windows11-64-2009-asan-qr: 3
  • opt: 3
• windows11-32-2009-qr: 2
  • debug: 2
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-07-15&endday=2024-07-21&tree=all

Comment 97

[Intermittent Failures Robot]

12 failures in 4124 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-esr128: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows11-32-2009-qr: 6
  • debug: 6
• windows11-64-2009-asan-qr: 2
  • opt: 2
• android-hw-p5-13-0-android-aarch64-qr: 2
  • debug: 1
  • opt: 1
• windows11-64-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-07-22&endday=2024-07-28&tree=all

Comment 98

[Intermittent Failures Robot]

13 failures in 3839 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 1
• mozilla-esr128: 1
• try: 5

Platform and build breakdown:

• windows11-32-2009-qr: 6
  • debug: 6
• linux1804-64-ccov-qr: 1
  • opt: 1
• windows11-64-2009-asan-qr: 3
  • opt: 3
• macosx1015-64-qr: 1
  • debug: 1
• windows10-64-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-07-29&endday=2024-08-04&tree=all

Comment 99

[Intermittent Failures Robot]

14 failures in 3459 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• try: 2

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• windows11-32-2009-qr: 4
  • debug: 4
• windows11-64-2009-asan-qr: 6
  • opt: 6
• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-08-05&endday=2024-08-11&tree=all

Comment 100

[Intermittent Failures Robot]

18 failures in 2154 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-esr115: 1
• mozilla-beta: 3
• mozilla-central: 5

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1
• macosx1015-64-qr: 2
  • debug: 2
• windows11-64-2009-asan-qr: 5
  • opt: 5
• windows11-32-2009-qr: 5
  • debug: 5
• windows11-64-2009-qr: 2
  • debug: 2
• macosx1015-64-shippable-qr: 1
  • opt: 1
• linux64-snap-esr128: 1
  • debug: 1
• android-hw-p5-13-0-android-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-08-12&endday=2024-08-18&tree=all

Comment 101

[Intermittent Failures Robot]

25 failures in 3218 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• mozilla-central: 2
• mozilla-esr115: 3
• mozilla-esr128: 5
• mozilla-beta: 3

Platform and build breakdown:

• windows11-32-2009-qr: 10
  • debug: 10
• windows11-64-2009-asan-qr: 4
  • opt: 4
• windows11-64-2009-qr: 9
  • debug: 9
• linux1804-64-qr: 1
  • debug: 1
• windows11-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-08-19&endday=2024-08-25&tree=all

Comment 102

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This actually happens when the Marionette harness sends a TEST-START but misses to send a TEST-PASS or TEST-FAIL as it can be seen here for the test layout/base/tests/marionette/test_accessiblecaret_selection_mode.py AccessibleCaretSelectionModeTestCase.test_handle_tilt_when_carets_overlap_each_other_textarea_rtl:

https://treeherder.mozilla.org/logviewer?job_id=471272624&repo=autoland&lineNumber=132339

[task 2024-08-22T00:05:46.635Z] 00:05:46     INFO -  TEST-START | layout/base/tests/marionette/test_accessiblecaret_selection_mode.py AccessibleCaretSelectionModeTestCase.test_handle_tilt_when_carets_overlap_each_other_textarea_rtl
[..]
[task 2024-08-22T00:05:48.292Z] 00:05:48     INFO -  1724285148291	Marionette	TRACE	[20] MarionetteCommands actor destroyed for window id 12884901939
[task 2024-08-22T00:05:48.295Z] 00:05:48     INFO -  1724285148295	Marionette	DEBUG	53 <- [1,97,null,{"value":null}]
[task 2024-08-22T00:05:48.296Z] 00:05:48     INFO -  TEST-START | layout/base/tests/marionette/test_accessiblecaret_selection_mode.py AccessibleCaretSelectionModeTestCase.test_long_press_changes_focus_from_content_to_contenteditable

When this job fails it's actually always about this test. When I check the test file this specific test is generated via parameterization. As it looks like the test works fine and should have passed but whether we fail a check or miss to actually print the status.

Comment 103

[Intermittent Failures Robot]

7 failures in 3879 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-release: 1
• autoland: 4
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• windows11-64: 1
  • opt: 1
• windows11-32-2009-qr: 2
  • debug: 2
• linux1804-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-08-26&endday=2024-09-01&tree=all

Comment 104

[Intermittent Failures Robot]

14 failures in 3668 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 7
• mozilla-esr115: 2
• try: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-asan-qr: 6
  • opt: 6
• windows11-64: 2
  • opt: 2
• windows11-64-2009-qr: 2
  • debug: 2
• android-hw-a55-14-0-android-aarch64-shippable-qr: 1
  • opt: 1
• windows11-32-2009-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-09-02&endday=2024-09-08&tree=all

Comment 105

[Julian Descottes [:jdescottes]]

Interestingly, when forcing a random error in the test, the failure is correctly picked up by the harness.

However when forcing an error with self.assertEqual(target_content, ...), then we have the same issue: the job fails but the error is not reported. The error message that should be printed in this case is:

TEST_END: FAIL, expected PASS - AssertionError: ' فيرفكس موزيلا فيرفكس' != 'AAA'
-  فيرفكس موزيلا فيرفكس
+ AAA

(AAA is just a random string I am expecting here to make the test fail). I wonder if on windows the test harness fails to process the arabic characters in the error message.

Comment 106

[Julian Descottes [:jdescottes]]

Note that other failures for this test are tracked under Bug 1855083

Comment 107

[Julian Descottes [:jdescottes]]

Just to confirm: this is a windows only issue triggered by the fact that we are unable to print the failure message containing arabic characters. I am trying at the moment to force PYTHONIOENCODING=utf-8 in the environment to see if it helps. https://treeherder.mozilla.org/jobs?repo=try&revision=7331676ff70d01f5a3c2d27cc7c4801d04c0612b

Comment 108

[Julian Descottes [:jdescottes]]

Hi Alex,

Henrik suggested reaching out to you about such issues.

Here we have an issue where Python on windows CI jobs is not able to print a failure message containing unicode characters, for instance

TEST_END: FAIL, expected PASS - AssertionError: ' فيرفكس موزيلا فيرفكس' != 'AAA'

With this log missing, the log parser is unable to detect the proper failure.

What I tried above was to force env["PYTHONIOENCODING"] = "utf-8" in testing/mozharness/scripts/marionette.py (changeset) and this seems to fix the issue, although the printed error message is a bit harder to read:

AssertionError: ' \u0641\u064a\u0631\u0641\u0643\u0633 \u0645\u0648\u0632\u064a\u0644\u0627 \u0641\u064a\u0631\u0641\u0643\u0633' != 'AAA'

(which reminds me of https://bugzilla.mozilla.org/show_bug.cgi?id=1902981 which I filed a few months ago)

So it seems that there is a problem when trying to print unicode characters on Windows CI. Do you have a suggestion on how to fix it properly (potentially different from what I did in the patch above?)

Talking with :jgraham, there is also a suggestion to use sys.stdout.reconfigure to properly set the encoding

Comment 109

[Alex Hochheiden [:ahochheiden] (PTO until Feb 3)]

I think env["PYTHONIOENCODING"] = "utf-8" seems like the best solution.

The sys.stdout.reconfigure(encoding='utf-8') should also work, though.

There's also this you could play around with:

import locale
locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')

(But you might need set ar_SA.UTF-8 specifically for that test with that approach? Not sure).

Disclaimer: I've dealt with problems like this before, but I'm by no means an expert.

Comment 110

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hi Alex. I’m not an expert with UTF-8 either, but I’m curious about what the default encoding is on Windows. Do we have any custom settings for mozbuild? If not, could it be a solution to set the encoding via mozbuild, ensuring that it works not only on CI systems but also for users running tests locally on their machines?

Comment 111

[Alex Hochheiden [:ahochheiden] (PTO until Feb 3)]

On Windows I'd expect the default encoding to either be cp-1252 or utf-8 (depending on the OS version and Python version, but maybe also locale?).

Setting it to utf-8 in the MozillaBuild profile would have side effects on tools that do use the system encoding (I assumed this, but checked with :glandium to confirm).

I don't think there's a better way than explicitly setting the encoding to handle a particular case.

Comment 112

[Intermittent Failures Robot]

14 failures in 4448 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• mozilla-esr128: 2
• autoland: 6
• mozilla-release: 2
• mozilla-beta: 2
• try: 1

Platform and build breakdown:

• android-hw-a51-11-0-aarch64-shippable: 1
  • opt: 1
• windows11-32-2009-qr: 4
  • debug: 4
• windows11-64-2009-qr: 4
  • debug: 4
• macosx1100-aarch64-shippable-qr: 1
  • opt: 1
• windows11-64-2009-asan-qr: 2
  • opt: 2
• android-hw-p5-13-0-android-aarch64-qr: 1
  • opt: 1
• windows11-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-09-09&endday=2024-09-15&tree=all

Comment 113

[Julian Descottes [:jdescottes]]

Thanks for all the input, I think for now we should go with the initial approach at least to get the failures correctly classified from now on.

Comment 114

[Julian Descottes [:jdescottes]]

Attachment: Bug 1800035 - [mozharness] Force PYTHONIOENCODING to utf-8 in testing/mozharness/scripts/marionette.py

Error messages with unicode characters failed to be correctly printed on windows.
Forcing PYTHONIOENCODING to utf-8 avoids the issue.

Comment 115

[Julian Descottes [:jdescottes]]

Aryx: quick question related to this bug. Here the reason the issue was hard to investigate was that we had a TEST-START without a corresponding TEST-PASS - or any other line indicating the final result for the test.

In general do you think this is something we should try to detect in logs, and trigger failures in those cases? Or are there tests for which it is expected to have TEST-START but not necessarily any result?

Comment 116

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

If there is a TEST-START marker, there should also be a marker to identify the end of the logging block for the test if the test suite continues to be executed - if Firefox gets terminated/crashed and testing does not resume, there should be no such marker. End markers are e.g. TEST-OK, TEST-SKIP, TEST-UNEXPECTED-FAIL, TEST-FAIL.
In such a case of a crash of the application, a fatal error is supposed to get logged by the test suite and provide information.
In case of Marionette, it's a test step (TEST-PASS) which does not get logged but the same concept applies: if test execution continues after failure, an end marker for the failed test should be logged.

Comment 117

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/573b8b270102
[mozharness] Force PYTHONIOENCODING to utf-8 in testing/mozharness/scripts/marionette.py r=webdriver-reviewers,jgraham

Comment 118

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/573b8b270102

Comment 119

[Intermittent Failures Robot]

16 failures in 4860 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• mozilla-beta: 5
• autoland: 6
• mozilla-central: 1
• mozilla-esr128: 1

Platform and build breakdown:

• windows11-64-2009-shippable-qr: 1
  • opt: 1
• android-hw-p5-13-0-android-aarch64-shippable-qr: 1
  • opt: 1
• android-hw-p5-13-0-android-aarch64-qr: 2
  • opt: 1
  • debug: 1
• windows11-32-2009-qr: 2
  • debug: 2
• macosx1015-64-qr: 1
  • debug: 1
• windows11-64-2009-asan-qr: 5
  • opt: 5
• windows11-64-2009-qr: 3
  • debug: 3
• windows11-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-09-16&endday=2024-09-22&tree=all

Comment 120

[Intermittent Failures Robot]

1 failures in 4889 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1

Platform and build breakdown:

• windows11-64-2009-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-09-23&endday=2024-09-29&tree=all

Comment 121

[Intermittent Failures Robot]

3 failures in 3574 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-central: 2

Platform and build breakdown:

• windows11-64-2009-shippable-qr: 1
  • opt: 1
• android-hw-a51-11-0-aarch64-shippable-qr: 1
  • opt: 1
• android-hw-p5-13-0-android-aarch64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-09-30&endday=2024-10-06&tree=all

Comment 122

[Intermittent Failures Robot]

14 failures in 4679 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• comm-central: 9
• mozilla-beta: 1
• mozilla-esr115: 1
• mozilla-release: 1

Platform and build breakdown:

• android-hw-p5-13-0-android-aarch64-shippable-qr: 2
  • opt: 2
• macosx1015-64-qr: 2
  • debug: 2
• windows11-32-2009-qr: 3
  • opt: 1
  • debug: 2
• windows11-64-2009-qr: 3
  • opt: 2
  • debug: 1
• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-64: 1
  • opt: 1
• android-hw-p5-13-0-android-aarch64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-10-07&endday=2024-10-13&tree=all

Comment 123

[Henrik Skupin [:whimboo][⌚️UTC+2]]

All the recent failure classifications were incorrect and got re-classified. There is only one issue on esr115 which we are going to ignore.

Comment 124

[Intermittent Failures Robot]

4 failures in 4515 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1
• autoland: 3

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1
• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows11-64: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-11-04&endday=2024-11-10&tree=all

Comment 125

[Intermittent Failures Robot]

3 failures in 4227 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr128: 1
• autoland: 1
• mozilla-central: 1

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • debug: 1
• android-hw-a55-14-0-aarch64-shippable: 1
  • opt: 1
• macosx1015-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-11-11&endday=2024-11-17&tree=all

Comment 126

[Intermittent Failures Robot]

Summary

1 failures in 4615 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • debug: 1

Table

	1proc	1proc-s	1proc-swr	a11y-checks	a11y-checks-s	a11y-checks-swr	a11y-checks-swr-s	condprof	condprof-s	fis	fis-hv	fis-hv-s	fis-s	headless	headless-s	http2	http3	mda-gpu	mda-gpu-s	msix	msix-s	no_variant	nofis	nofis-aab	nofis-s	nofis-ship	nofis-ship-s	nofis-spi	nofis-swr	nofis-swr-s	nogpu	nogpu-s	s	spi	spi-nw	spi-nw-1proc	spi-nw-nofis	spi-nw-nofis-s	spi-nw-s	spi-s	swr	swr-s	wmfme	wr-dc0-s	wr-dc1-p	wr-dc2-o	xorig	xorig-s
windows11-32-2009-qr/debug																						1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2024-12-09&endday=2024-12-15&tree=all

Comment 127

[Intermittent Failures Robot]

Summary

1 failures in 4035 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr128: 1

Platform and build breakdown:

• windows11-64-2009-asan-qr: 1
  • opt: 1

Table

	no_variant
windows11-64-2009-asan-qr--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1800035&startday=2025-01-20&endday=2025-01-26&tree=all