1630162 - Intermittent Wd TEST-UNEXPECTED-TIMEOUT | expected OK [shutdown hang]

Status: REOPENED

(Remote Protocol :: Marionette, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ccoroiu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=297631286&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OIV_AGuRTDeuoSBzs4niRQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-04-15T00:29:36.827Z] 00:29:36 INFO - PID 1496 | 1586910576818 Marionette DEBUG Resetting recommended pref dom.file.createInChild
[task 2020-04-15T00:29:36.827Z] 00:29:36 INFO - PID 1496 | 1586910576818 Marionette DEBUG Resetting recommended pref dom.max_chrome_script_run_time
[task 2020-04-15T00:29:36.828Z] 00:29:36 INFO - PID 1496 | 1586910576819 Marionette DEBUG Resetting recommended pref dom.max_script_run_time
[task 2020-04-15T00:29:36.828Z] 00:29:36 INFO - PID 1496 | 1586910576820 Marionette DEBUG Resetting recommended pref dom.push.connection.enabled
[task 2020-04-15T00:29:36.829Z] 00:29:36 INFO - PID 1496 | 1586910576820 Marionette DEBUG Resetting recommended pref extensions.getAddons.cache.enabled
[task 2020-04-15T00:29:36.830Z] 00:29:36 INFO - PID 1496 | 1586910576820 Marionette DEBUG Resetting recommended pref extensions.webservice.discoverURL
[task 2020-04-15T00:29:36.830Z] 00:29:36 INFO - PID 1496 | 1586910576821 Marionette DEBUG Resetting recommended pref network.http.prompt-temp-redirect
[task 2020-04-15T00:29:36.831Z] 00:29:36 INFO - PID 1496 | 1586910576821 Marionette DEBUG Resetting recommended pref network.http.speculative-parallel-limit
[task 2020-04-15T00:29:36.831Z] 00:29:36 INFO - PID 1496 | 1586910576821 Marionette DEBUG Resetting recommended pref privacy.trackingprotection.enabled
[task 2020-04-15T00:29:36.832Z] 00:29:36 INFO - PID 1496 | 1586910576821 Marionette DEBUG Resetting recommended pref security.fileuri.strict_origin_policy
[task 2020-04-15T00:29:36.832Z] 00:29:36 INFO - PID 1496 | 1586910576821 Marionette DEBUG Resetting recommended pref security.notification_enable_delay
[task 2020-04-15T00:29:36.832Z] 00:29:36 INFO - PID 1496 | 1586910576822 Marionette DEBUG Resetting recommended pref signon.autofillForms
[task 2020-04-15T00:29:36.832Z] 00:29:36 INFO - PID 1496 | 1586910576823 Marionette DEBUG Resetting recommended pref signon.rememberSignons
[task 2020-04-15T00:29:36.832Z] 00:29:36 INFO - PID 1496 | 1586910576824 Marionette DEBUG Resetting recommended pref toolkit.cosmeticAnimations.enabled
[task 2020-04-15T00:29:36.832Z] 00:29:36 INFO - PID 1496 | 1586910576826 Marionette DEBUG Marionette stopped listening
[task 2020-04-15T00:29:36.832Z] 00:29:36 INFO - PID 1496 | DEBUG: Starting phase xpcom-will-shutdown
[task 2020-04-15T00:29:36.832Z] 00:29:36 INFO - PID 1496 | DEBUG: Spinning the event loop
[task 2020-04-15T00:29:36.922Z] 00:29:36 INFO - PID 1496 | DEBUG: Finished phase xpcom-will-shutdown
[task 2020-04-15T00:29:36.923Z] 00:29:36 INFO - PID 1496 |
[task 2020-04-15T00:29:36.924Z] 00:29:36 INFO - PID 1496 | ###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-04-15T00:29:36.925Z] 00:29:36 INFO - PID 1496 |
[task 2020-04-15T00:29:36.925Z] 00:29:36 INFO - PID 1496 | DEBUG: Starting phase web-workers-shutdown
[task 2020-04-15T00:29:36.926Z] 00:29:36 INFO - PID 1496 | DEBUG: Spinning the event loop
[task 2020-04-15T00:29:36.926Z] 00:29:36 INFO - PID 1496 | DEBUG: Completed blocker OS.File: flush pending requests, warn about unclosed files, shut down service. for phase web-workers-shutdown
[task 2020-04-15T00:29:36.926Z] 00:29:36 INFO - PID 1496 | DEBUG: Finished phase web-workers-shutdown
[task 2020-04-15T00:30:09.508Z] 00:30:09 INFO - TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/set_window_rect/user_prompts.py | expected OK
[task 2020-04-15T00:30:09.508Z] 00:30:09 INFO - TEST-INFO took 365004ms
[task 2020-04-15T00:30:09.589Z] 00:30:09 INFO - Closing logging queue
[task 2020-04-15T00:30:09.589Z] 00:30:09 INFO - queue closed
[task 2020-04-15T00:30:09.589Z] 00:30:09 INFO - Starting runner
[task 2020-04-15T00:30:10.019Z] 00:30:10 INFO - PID 2596 | 1586910610002 geckodriver DEBUG Listening on 127.0.0.1:49240
[task 2020-04-15T00:30:10.505Z] 00:30:10 INFO - WebDriver HTTP server listening at http://127.0.0.1:49240/
[task 2020-04-15T00:30:10.505Z] 00:30:10 INFO - TEST-START | /webdriver/tests/status/status.py
[task 2020-04-15T00:30:10.796Z] 00:30:10 INFO - STDOUT: ============================= test session starts =============================
[task 2020-04-15T00:30:10.796Z] 00:30:10 INFO - STDOUT: platform win32 -- Python 2.7.15, pytest-unknown, py-1.5.2, pluggy-0.5.3.dev -- Z:\task_1586905291\build\venv\Scripts\python.exe
[task 2020-04-15T00:30:10.796Z] 00:30:10 INFO - STDOUT: rootdir: Z:\task_1586905291\build, inifile:
[task 2020-04-15T00:30:10.796Z] 00:30:10 INFO - STDOUT: collecting ...
[task 2020-04-15T00:30:10.796Z] 00:30:10 INFO - STDOUT: collected 2 items
[task 2020-04-15T00:30:10.797Z] 00:30:10 INFO - STDOUT: tests/web-platform/tests/webdriver/tests/status/status.py::test_get_status_no_session
[task 2020-04-15T00:30:10.798Z] 00:30:10 INFO - PID 2596 | 1586910610795 webdriver::server DEBUG -> GET /status
[task 2020-04-15T00:30:10.799Z] 00:30:10 INFO - PID 2596 | 1586910610795 webdriver::server DEBUG <- 200 OK {"value":{"message":"","ready":true}}
[task 2020-04-15T00:30:10.799Z] 00:30:10 INFO - STDOUT: PASSED
[task 2020-04-15T00:30:10.799Z] 00:30:10 INFO - STDOUT: tests/web-platform/tests/webdriver/tests/status/status.py::test_status_with_session_running_on_endpoint_node
[task 2020-04-15T00:30:10.806Z] 00:30:10 INFO - PID 2596 | 1586910610798 webdriver::server DEBUG -> POST /session {"capabilities": {"alwaysMatch": {"moz:firefoxOptions": {"binary": "Z:\task_1586905291\build\application\firefox\firefox.exe", "prefs": {"network.process.enabled": false, "toolkit.asyncshutdown.log": true, "network.dns.localDomains": "www1.xn--n8j6ds53lwwkrqhv28a.web-platform.test,op88.web-platform.test,op36.not-web-platform.test,op53.not-web-platform.test,op50.not-web-platform.test,xn--lve-6lad.www.web-platform.test,op98.web-platform.test,op24.not-web-platform.test,op31.not-web-platform.test,op95.not-web-platform.test,op85.web-platform.test,op83.not-web-platform.test,www2.not-web-platform.test,xn--lve-6lad.www.not-web-platform.test,op73.not-web-platform.test,op8.web-platform.test,www2.www2.not-web-platform.test,op89.web-platform.test,op66.web-platform.test,xn--lve-6lad.web-platform.test,op19.not-web-platform.test,www1.www2.web-platform.test,op72.web-platform.test,op24.web-platform.test,op21.not-web-platform.test,xn--lve-6lad.not-web-platform.test,op41.web-platform.test,op79.web-platform.test,op81.not-web-platform.test,op70.not-web-platform.test,xn--n8j6ds53lwwkrqhv28a.xn--lve-6lad.not-web-platform.test,op78.not-web-platform.test,op6.not-web-platform.test,www1.www.not-web-platform.test,op40.not-web-platform.test,op25.not-web-platform.test,op3.not-web-platform.test,op65.not-web-platform.test,op91.web-platform.test,www.www2.web-platform.test,op80.not-web-platform.test,op59.web-platform.test,op52.not-web-platform.test,xn--lve-6lad.xn--lve-6lad.web-platform.test,op68.not-web-platform.test,op45.not-web-platform.test,op71.not-web-platform.test,op72.not-web-platform.test,xn--n8j6ds53lwwkrqhv28a.www2.web-platform.test,op39.web-platform.test,op90.not-web-platform.test,op60.web-platform.test,op58.web-platform.test,op28.web-platform.test,www1.web-platform.test,xn--n8j6ds53lwwkrqhv28a.xn--lve-6lad.web-platform.test,op14.web-platform.test,op89.not-web-platform.test,op69.web-platform.test,op49.not-web-platform.test,op40.web-platform.test,op2.not-web-platform.test,op5.not-web-platform.test,www.www2.not-web-platform.test,op77.not-web-platform.test,www.xn--n8j6ds53lwwkrqhv28a.web-platform.test,op7.web-platform.test,op74.web-platform.test,op79.not-web-platform.test,op82.not-web-platform.test,www.www1.web-platform.test,op12.not-web-platform.test,op39.not-web-platform.test,op31.web-platform.test,www.not-web-platform.test,www.www.not-web-platform.test,op44.not-web-platform.test,www1.not-web-platform.test,xn--n8j6ds53lwwkrqhv28a.www1.web-platform.test,op58.not-web-platform.test,op14.not-web-platform.test,op30.not-web-platform.test,op62.not-web-platform.test,op61.not-web-platform.test,op92.not-web-platform.test,www2.xn--lve-6lad.web-platform.test,op29.not-web-platform.test,op18.web-platform.test,op73.web-platform.test,xn--n8j6ds53lwwkrqhv28a.xn--n8j6ds53lwwkrqhv28a.web-platform.test,op77.web-platform.test,op12.web-platform.test,op54.web-platform.test,op63.web-platform.test,op71.web-platform.test,www2.www1.not-web-platform.test,op95.web-platform.test,op16.web-platform.test,op36.web-platform.test,op27.web-platform.test,www.www.web-platform.test,op98.not-web-platform.test,op64.not-web-platform.test,op29.web-platform.test,op9.web-platform.test,op26.not-web-platform.test,op94.web-platform.test,xn--n8j6ds53lwwkrqhv28a.www2.not-web-platform.test,op44.web-platform.test,op94.not-web-platform.test,op33.web-platform.test,op38.not-web-platform.test,op33.not-web-platform.test,op84.web-platform.test,www1.www1.not-web-platform.test,op23.not-web-platform.test,op57.not-web-platform.test,op54.not-web-platform.test,op85.not-web-platform.test,www2.www2.web-platform.test,op46.not-web-platform.test,op97.not-web-platform.test,op32.web-platform.test,op61.web-platform.test,op70.web-platform.test,www2.web-platform.test,op32.not-web-platform.test,op60.not-web-platform.test,op4.web-platform.test,op43.web-platform.test,op7.not-web-platform.test,op78.web-platform.test,op26.web-platform.test,xn--lve-6lad.xn--n8j6ds53lwwkrqhv28a.web-platform.test,op96.not-web-platform.test,op51.not-web-platform.test,op41.not-web-platform.test,op76.web-platform.test,op52.web-platform.test,op99.web-platform.test,op35.not-web-platform.test,op99.not-web-platform.test,op86.web-platform.test,not-web-platform.test,op42.not-web-platform.test,op46.web-platform.test,op67.not-web-platform.test,op17.web-platform.test,op90.web-platform.test,op93.web-platform.test,op37.not-web-platform.test,op48.not-web-platform.test,op10.web-platform.test,op55.not-web-platform.test,op4.not-web-platform.test,www1.xn--n8j6ds53lwwkrqhv28a.not-web-platform.test,op55.web-platform.test,xn--lve-6lad.www2.web-platform.test,op47.web-platform.test,op51.web-platform.test,op45.web-platform.test,op80.web-platform.test,op68.web-platform.test,op49.web-platform.test,op57.web-platform.test,www2.xn--n8j6ds53lwwkrqhv28a.web-platform.test,www.xn--n8j6ds53lwwkrqhv28a.not-web-platform.test,op56.not-web-platform.test,op35.web-platform.test,op84.not-web-platform.test,xn--n8j6ds53lwwkrqhv28a.not-web-platform.test,xn--lve-6lad.xn--n8j6ds53lwwkrqhv28a.not-web-platform.test,op34.not-web-platform.test,op6.web-platform.test,web-platform.test,op67.web-platform.test,op69.not-web-platform.test,op11.not-web-platform.test,op93.not-web-platform.test,www1.www.web-platform.test,op86.not-web-platform.test,op8.not-web-platform.test,www2.xn--n8j6ds53lwwkrqhv28a.not-web-platform.test,op92.web-platform.test,xn--lve-6lad.www1.not-web-platform.test,op15.web-platform.test,op13.not-web-platform.test,op13.web-platform.test,xn--n8j6ds53lwwkrqhv28a.web-platform.test,xn--n8j6ds53lwwkrqhv28a.www.web-platform.test,op75.web-platform.test,op20.not-web-platform.test,op76.not-web-platform.test,op64.web-platform.test,op97.web-platform.test,op37.web-platform.test,op56.web-platform.test,op62.web-platform.test,op82.web-platform.test,op25.web-platform.test,op11.web-platform.test,www.xn--lve-6lad.not-web-platform.test,www2.www1.web-platform.test,op27.not-web-platform.test,op50.web-platform.test,op17.not-web-platform.test,op38.web-platform.test,www2.www.not-web-platform.test,xn--lve-6lad.www1.web-platform.test,op75.not-web-platform.test,op83.web-platform.test,op81.web-platform.test,op15.not-web-platform.test,xn--n8j6ds53lwwkrqhv28a.www.not-web-platform.test,op20.web-platform.test,op3.web-platform.test,www1.www2.not-web-platform.test,xn--n8j6ds53lwwkrqhv28a.xn--n8j6ds53lwwkrqhv28a.not-web-platform.test,op2.web-platform.test,op21.web-platform.test,op23.web-platform.test,op42.web-platform.test,op47.not-web-platform.test,www1.www1.web-platform.test,op18.not-web-platform.test,op22.web-platform.test,xn--lve-6lad.xn--lve-6lad.not-web-platform.test,op63.not-web-platform.test,op28.not-web-platform.test,op22.not-web-platform.test,op65.web-platform.test,www.www1.not-web-platform.test,www1.xn--lve-6lad.web-platform.test,op43.not-web-platform.test,op66.not-web-platform.test,www2.www.web-platform.test,op96.web-platform.test,op91.not-web-platform.test,www.xn--lve-6lad.web-platform.test,op1.web-platform.test,op74.not-web-platform.test,op87.web-platform.test,op59.not-web-platform.test,op19.web-platform.test,xn--n8j6ds53lwwkrqhv28a.www1.not-web-platform.test,op9.not-web-platform.test,op88.not-web-platform.test,op53.web-platform.test,www2.xn--lve-6lad.not-web-platform.test,op87.not-web-platform.test,op30.web-platform.test,op10.not-web-platform.test,op48.web-platform.test,op16.not-web-platform.test,op34.web-platform.test,op1.not-web-platform.test,www.web-platform.test,op5.web-platform.test,www1.xn--lve-6lad.not-web-platform.test,xn--lve-6lad.www2.not-web-platform.test", "media.peerconnection.mtransport_process": false}}}}}
[task 2020-04-15T00:30:10.811Z] 00:30:10 INFO - PID 2596 | 1586910610803 mozrunner::runner INFO Running command: "Z:\task_1586905291\build\application\firefox\firefox.exe" "-marionette" "-foreground" "-no-remote" "-profile" "C:\Users\task_1586905291\AppData\Local\Temp\rust_mozprofileCej2NI"
[task 2020-04-15T00:30:10.811Z] 00:30:10 INFO - PID 2596 | 1586910610804 geckodriver::marionette DEBUG Waiting 60s to connect to browser on 127.0.0.1:2828
[task 2020-04-15T00:30:11.220Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker OS.File: flush pending requests, warn about unclosed files, shut down service. for phase web-workers-shutdown
[task 2020-04-15T00:30:11.220Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker OS.File: flush I/O queued before profileBeforeChange for phase profile-before-change
[task 2020-04-15T00:30:11.225Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker AddonManager: shutting down. for phase profile-before-change
[task 2020-04-15T00:30:11.312Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker ClientManagerService: start destroying IPC actors early for phase xpcom-will-shutdown
[task 2020-04-15T00:30:11.497Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker Flush WebExtension StartupCache for phase profile-before-change
[task 2020-04-15T00:30:11.785Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker JSON store: writing data for phase profile-before-change
[task 2020-04-15T00:30:11.919Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker JSON store: writing data for phase AddonManager: Waiting for providers to shut down.
[task 2020-04-15T00:30:11.919Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker Update add-on blocklist state into add-on DB for phase AddonManager: Waiting to start provider shutdown.
[task 2020-04-15T00:30:11.919Z] 00:30:11 INFO - PID 2596 | DEBUG: Completed blocker Update add-on blocklist state into add-on DB for phase AddonManager: Waiting to start provider shutdown.
[task 2020-04-15T00:30:11.941Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker XPIProvider shutdown for phase quit-application-granted
[task 2020-04-15T00:30:11.941Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker XPIProvider for phase AddonManager: Waiting for providers to shut down.
[task 2020-04-15T00:30:11.941Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker PluginProvider for phase AddonManager: Waiting for providers to shut down.
[task 2020-04-15T00:30:11.960Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker ServiceWorkerRegistrar: Flushing data for phase profile-before-change
[task 2020-04-15T00:30:11.960Z] 00:30:11 INFO - PID 2596 | DEBUG: Adding blocker CrashMonitor: Writing notifications to file after receiving profile-before-change for phase OS.File: Waiting for clients before profileBeforeChange
[task 2020-04-15T00:30:12.087Z] 00:30:12 INFO - PID 2596 | DEBUG: Adding blocker TelemetryController: shutting down for phase profile-before-change-telemetry
[task 2020-04-15T00:30:12.117Z] 00:30:12 INFO - PID 2596 | DEBUG: Adding blocker EnvironmentAddonBuilder for phase AddonManager: Waiting to start provider shutdown.
[task 2020-04-15T00:30:12.156Z] 00:30:12 INFO - PID 2596 | DEBUG: Adding blocker RemoteSettingsClient - finish IDB access. for phase profile-before-change
[task 2020-04-15T00:30:12.156Z] 00:30:12 INFO - PID 2596 | 1586910612149 Marionette TRACE Marionette enabled
[task 2020-04-15T00:30:12.322Z] 00:30:12 INFO - PID 2596 | 1586910612319 Marionette TRACE Received observer notification toplevel-window-ready
[task 2020-04-15T00:30:12.711Z] 00:30:12 INFO - PID 2596 | DEBUG: Adding blocker MediaShutdownManager: shutdown for phase profile-before-change
[task 2020-04-15T00:30:12.957Z] 00:30:12 INFO - PID 2596 | DEBUG: Adding blocker ServiceWorkerShutdownBlocker: shutting down Service Workers for phase profile-change-teardown
[task 2020-04-15T00:30:13.059Z] 00:30:13 INFO - PID 2596 | DEBUG: Adding blocker JSON store: writing data for phase profile-before-change
[task 2020-04-15T00:30:13.102Z] 00:30:13 INFO - PID 2596 | DEBUG: Adding blocker JSON store: writing data for phase profile-before-change
[task 2020-04-15T00:30:13.102Z] 00:30:13 INFO - PID 2596 | DEBUG: Adding blocker Flush AddonRepository for phase profile-before-change
[task 2020-04-15T00:30:13.145Z] 00:30:13 INFO - PID 2596 | DEBUG: Adding blocker GMPProvider for phase AddonManager: Waiting for providers to shut down.
[task 2020-04-15T00:30:13.145Z] 00:30:13 INFO - PID 2596 | DEBUG: Adding blocker PluginFinder: finding plugins to load for phase profile-change-teardown
[task 2020-04-15T00:30:13.296Z] 00:30:13 INFO - PID 2596 | DEBUG: Adding blocker Remote Settings profile-before-change for phase profile-before-change
[task 2020-04-15T00:30:13.785Z] 00:30:13 INFO - PID 2596 | DEBUG: Adding blocker MediaShutdownManager: shutdown for phase xpcom-will-shutdown
[task 2020-04-15T00:30:14.172Z] 00:30:14 INFO - PID 2596 | DEBUG: Adding blocker Places Clients shutdown for phase profile-change-teardown
[task 2020-04-15T00:30:14.172Z] 00:30:14 INFO - PID 2596 | DEBUG: Adding blocker Places Connection shutdown for phase profile-before-change
[task 2020-04-15T00:30:14.231Z] 00:30:14 INFO - PID 2596 | DEBUG: Adding blocker PageActions: purging unregistered actions from cache for phase profile-before-change
[task 2020-04-15T00:30:14.453Z] 00:30:14 INFO - PID 2596 | DEBUG: Adding blocker Sqlite.jsm shutdown blocker for phase profile-before-change

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

There is a 1 minute shutdown delay after phase web-workers-shutdown:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=297631286&repo=mozilla-central&lineNumber=78424-78425

[task 2020-04-15T00:28:28.248Z] 00:28:28 INFO - PID 1496 | DEBUG: Finished phase web-workers-shutdown
[task 2020-04-15T00:29:28.890Z] 00:29:28 INFO - PID 1496 | 1586910568880 geckodriver::marionette DEBUG Browser process stopped: exit code: 0

Andrea, any idea what this could be?

Comment 2

[Andrea Marchesini [:baku]]

asuth do you have time to take a look? Thanks!

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I just noticed that this failure is for a ccov build. Maybe there is some special logic, and Marco might know.

Comment 4

[Intermittent Failures Robot]

1 failures in 4528 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1630162&startday=2020-04-13&endday=2020-04-19&tree=all

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

The web workers (sub-)phase happens during RuntimeService::Shutdown being invoked by the "xpcom-shutdown" notification. This suggests something else in "xpcom-shutdown" is still doing things? Ideally a crash would be induced in the browser to see what was on the stack.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sadly we cannot detect crashes yet for these type of tests due to bug 1490906. That means we might not have a chance to continue investigation for now.

Comment 7

[Marco Castelluccio [:marco]]

Ccov is pretty slow at shutdown, as that's when it writes out the coverage counters. When we'll be able to induce a crash we can check if the process was in the process of writing coverage files.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This is not only happening for ccov builds but also shippable opt. See the recently duped bug.

Comment 10

[Intermittent Failures Robot]

2 failures in 5488 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• macosx1014-64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1630162&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 13

[Atila Butkovits]

New occurrence: https://treeherder.mozilla.org/logviewer?job_id=331384444&repo=autoland&lineNumber=33680

Comment 14

[Intermittent Failures Robot]

1 failures in 4327 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2021-02-22&endday=2021-02-28&tree=all

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The shutdown hang as port of the failure logged as bug 1698259 is even longer and takes nearly 3 minutes.

Gabriele, isn't the shutdown hang monitor to kick in and crash Firefox eventually? Why doesn't that happen here? If you are the wrong person to ask please forward appropriately.

Comment 17

[Gabriele Svelto [:gsvelto]]

It depends on the build. The normal timeout is 60 seconds but it can be bumped to 180 or even 360 depending on the build, see here. However IIRC every time we start a new shutdown phase the timer is reset to 0 and we have quite a few phases, so if they're all slow the over shutdown may last for significantly longer than the timeout w/o causing a crash.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Gabriele Svelto [:gsvelto] from comment #17)

However IIRC every time we start a new shutdown phase the timer is reset to 0 and we have quite a few phases, so if they're all slow the over shutdown may last for significantly longer than the timeout w/o causing a crash.

Huh, is that really the case? We are using toolkit.asyncshutdown.crash_timeout in Marionette/geckodriver to determine when to actually force-kill the application if time exceeded by extra 10s on top. Do you know who can say for sure how that timer is working?

Comment 19

[Gabriele Svelto [:gsvelto]]

Looking at the code we compare gHeartbeat to the running timer. Whenever we advance to a new shutdown phase we invoke UpdateHeartbeat() that resets the heartbeat to zero.

Or at least that's how I understand this code. BTW I just noticed that this code was recently changed in bug 1689953.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hm ok. Thinking a bit more of it I believe we should keep that force-kill after the 60s. If there is indeed a long shutdown due to some hanging phases we would like to know about it. And killing the process would at least give us a chance to see how often it happens.

Would there be a way to kill the process and get a proper crash stack with the hanging phase? If yes, that would be perfect.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Maybe we need mozcrash.kill_and_get_minidump() under such a condition and not just call mozrunner.stop().

Comment 22

[Gabriele Svelto [:gsvelto]]

That would do the trick.

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks! I filed bug 1698937 to get this implemented.

Comment 24

[Intermittent Failures Robot]

2 failures in 4257 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64: 1
  • opt: 1
• windows10-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 25

[Intermittent Failures Robot]

1 failures in 3125 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2021-03-29&endday=2021-04-04&tree=all

Comment 26

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 28

[Intermittent Failures Robot]

1 failures in 3626 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2021-10-04&endday=2021-10-10&tree=all

Comment 29

[Intermittent Failures Robot]

1 failures in 3315 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2021-10-11&endday=2021-10-17&tree=all

Comment 30

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 32

[Intermittent Failures Robot]

2 failures in 2819 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • opt: 1
• windows10-64-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2021-11-22&endday=2021-11-28&tree=all

Comment 33

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 35

[Intermittent Failures Robot]

5 failures in 2286 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 2
• try: 2

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1
• linux1804-64-asan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2021-12-20&endday=2021-12-26&tree=all

Comment 37

[Intermittent Failures Robot]

2 failures in 2801 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-01-03&endday=2022-01-09&tree=all

Comment 38

[Intermittent Failures Robot]

5 failures in 3060 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 4
  • opt: 4
• windows10-64-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-01-10&endday=2022-01-16&tree=all

Comment 40

[Intermittent Failures Robot]

2 failures in 3445 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-01-24&endday=2022-01-30&tree=all

Comment 41

[Intermittent Failures Robot]

4 failures in 3165 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 3

Platform and build breakdown:

• linux1804-64-ccov-qr: 1
  • opt: 1
• linux1804-64-asan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-01-31&endday=2022-02-06&tree=all

Comment 42

[Intermittent Failures Robot]

1 failures in 3711 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-02-07&endday=2022-02-13&tree=all

Comment 45

[Intermittent Failures Robot]

3 failures in 3281 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 2
  • opt: 2
• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-02-14&endday=2022-02-20&tree=all

Comment 46

[Intermittent Failures Robot]

2 failures in 2992 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 1
  • opt: 1
• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-02-21&endday=2022-02-27&tree=all

Comment 49

[Intermittent Failures Robot]

1 failures in 3063 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-02-28&endday=2022-03-06&tree=all

Comment 51

[Intermittent Failures Robot]

2 failures in 3304 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-03-14&endday=2022-03-20&tree=all

Comment 52

[Intermittent Failures Robot]

1 failures in 3341 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-03-21&endday=2022-03-27&tree=all

Comment 53

[Intermittent Failures Robot]

1 failures in 3323 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-03-28&endday=2022-04-03&tree=all

Comment 54

[Intermittent Failures Robot]

2 failures in 2922 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-04-04&endday=2022-04-10&tree=all

Comment 55

[Intermittent Failures Robot]

2 failures in 2948 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 1

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 1
  • opt: 1
• windows10-32-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-04-11&endday=2022-04-17&tree=all

Comment 56

[Intermittent Failures Robot]

1 failures in 3159 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-04-18&endday=2022-04-24&tree=all

Comment 57

[Intermittent Failures Robot]

3 failures in 3289 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 1

Platform and build breakdown:

• linux1804-64-shippable-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 1
  • opt: 1
• linux1804-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-04-25&endday=2022-05-01&tree=all

Comment 60

[Intermittent Failures Robot]

4 failures in 3540 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• try: 1

Platform and build breakdown:

• linux1804-64-ccov-qr: 3
  • opt: 3
• macosx1015-64-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-05-02&endday=2022-05-08&tree=all

Comment 64

[Intermittent Failures Robot]

4 failures in 3317 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-qr: 2
  • opt: 2
• linux1804-64-shippable-qr: 1
  • opt: 1
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-05-09&endday=2022-05-15&tree=all

Comment 66

[Intermittent Failures Robot]

1 failures in 2917 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-05-30&endday=2022-06-05&tree=all

Comment 68

[Intermittent Failures Robot]

2 failures in 3336 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-central: 1

Platform and build breakdown:

• macosx1015-64-shippable-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-06-06&endday=2022-06-12&tree=all

Comment 70

[Intermittent Failures Robot]

3 failures in 3170 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-central: 1
• autoland: 1

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 1
  • opt: 1
• windows10-32-2004-qr: 1
  • debug: 1
• windows10-64-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-06-13&endday=2022-06-19&tree=all

Comment 72

[Intermittent Failures Robot]

3 failures in 3312 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr91: 1
• mozilla-beta: 1
• try: 1

Platform and build breakdown:

• macosx1015-64-shippable-qr: 1
  • opt: 1
• windows10-64-2004-devedition-qr: 1
  • opt: 1
• windows10-64-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-06-27&endday=2022-07-03&tree=all

Comment 74

[Intermittent Failures Robot]

2 failures in 3072 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-07-04&endday=2022-07-10&tree=all

Comment 75

[Intermittent Failures Robot]

3 failures in 3341 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2
• try: 1

Platform and build breakdown:

• macosx1015-64-devedition-qr: 2
  • opt: 2
• windows10-64-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-07-11&endday=2022-07-17&tree=all

Comment 76

[Intermittent Failures Robot]

1 failures in 3245 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1015-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-07-25&endday=2022-07-31&tree=all

Comment 77

[Intermittent Failures Robot]

21 failures in 3057 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2
• mozilla-central: 18
• try: 1

Platform and build breakdown:

• macosx1015-64-devedition-qr: 1
  • opt: 1
• macosx1015-64-shippable-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 18
  • opt: 18
• windows10-64-2004-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-08-01&endday=2022-08-07&tree=all

Comment 78

[Henrik Skupin [:whimboo][⌚️UTC+2]]

• windows10-64-2004-ccov-qr: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-08-01&endday=2022-08-07&tree=all

These failures are actually all about bug 1783217 and should be re-classified.

Comment 79

[Andreea Pavel [:apavel]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #78)

• windows10-64-2004-ccov-qr: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-08-01&endday=2022-08-07&tree=all

These failures are actually all about bug 1783217 and should be re-classified.

Those have been reclassified.

Comment 80

[Intermittent Failures Robot]

11 failures in 3339 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7
• try: 1
• autoland: 2
• mozilla-beta: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 7
  • opt: 7
• windows10-64-2004-devedition-qr: 2
  • opt: 2
• windows10-64-2004-qr: 1
  • debug: 1
• macosx1015-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-08-08&endday=2022-08-14&tree=all

Comment 81

[Intermittent Failures Robot]

7 failures in 2916 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 6
  • opt: 6
• windows10-32-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-08-15&endday=2022-08-21&tree=all

Comment 84

[Intermittent Failures Robot]

17 failures in 3293 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 15
• autoland: 2

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 9
  • opt: 9
• macosx1015-64-qr: 1
  • opt: 1
• windows10-32-2004-shippable-qr: 6
  • opt: 6
• windows10-64-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-08-22&endday=2022-08-28&tree=all

Comment 85

[Intermittent Failures Robot]

16 failures in 3537 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 14
• autoland: 2

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 13
  • opt: 13
• windows10-32-2004-qr: 1
  • debug: 1
• windows10-32-2004-shippable-qr: 1
  • opt: 1
• windows10-64-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-08-29&endday=2022-09-04&tree=all

Comment 86

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Intermittent Failures Robot from comment #85)

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 13
  • opt: 13

All these failures are wrongly classified and should have classified against bug 1783217. I asked the sheriffs to correct that.

Comment 88

[Intermittent Failures Robot]

19 failures in 3616 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 16
• autoland: 3

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 14
  • opt: 14
• windows10-64-2004-qr: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 1
  • debug: 1
• macosx1015-64-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-09-05&endday=2022-09-11&tree=all

Comment 89

[Intermittent Failures Robot]

11 failures in 3023 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10
• autoland: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 9
  • opt: 9
• android-em-7-0-x86_64-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-09-12&endday=2022-09-18&tree=all

Comment 90

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Intermittent Failures Robot from comment #89)

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 9

These failures are all about bug 1783217. I requested to re-classify the failures.

Comment 91

[Marian-Vasile Laza]

Failures are reclassified.

Comment 92

[Intermittent Failures Robot]

4 failures in 2909 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 2
• try: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 2
  • opt: 2
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-09-19&endday=2022-09-25&tree=all

Comment 93

[Intermittent Failures Robot]

4 failures in 2041 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• mozilla-beta: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 3
  • opt: 3
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-09-26&endday=2022-10-02&tree=all

Comment 94

[Intermittent Failures Robot]

9 failures in 2449 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6
• mozilla-beta: 3

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 6
  • opt: 6
• macosx1015-64-shippable-qr: 1
  • opt: 1
• windows10-64-2004-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-10-03&endday=2022-10-09&tree=all

Comment 95

[Intermittent Failures Robot]

5 failures in 3034 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 3
  • opt: 3
• windows10-64-2004-shippable-qr: 1
  • opt: 1
• windows10-32-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-10-10&endday=2022-10-16&tree=all

Comment 96

[Intermittent Failures Robot]

8 failures in 2793 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 6
• mozilla-esr102: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 6
  • opt: 6
• windows10-64-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-10-17&endday=2022-10-23&tree=all

Comment 97

[Intermittent Failures Robot]

7 failures in 3743 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7

Platform and build breakdown:

• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-10-24&endday=2022-10-30&tree=all

Comment 98

[Intermittent Failures Robot]

15 failures in 3626 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 14
• autoland: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 13
  • opt: 13
• windows10-32-2004-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-10-31&endday=2022-11-06&tree=all

Comment 99

[Intermittent Failures Robot]

8 failures in 3445 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6
• try: 2

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 6
  • opt: 6
• macosx1015-64-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-11-07&endday=2022-11-13&tree=all

Comment 100

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Most of the failures from the last two weeks are CCOV related and as such have been wrongly classified again. I've asked the sheriffs to correct those entries to bug 1783217.

Comment 101

[Intermittent Failures Robot]

15 failures in 3589 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 5
• try: 7
• mozilla-central: 2
• autoland: 1

Platform and build breakdown:

• macosx1015-64-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 7
  • debug-isolated-process: 7
• windows10-64-2004-ccov-qr: 2
  • opt: 2
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1
• macosx1015-64-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-lite-qr: 2
  • opt: 2
• windows10-64-2004-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-11-14&endday=2022-11-20&tree=all

Comment 102

[Intermittent Failures Robot]

17 failures in 3888 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 1
• try: 3
• mozilla-beta: 9

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 4
  • opt: 4
• android-em-7-0-x86_64-qr: 5
  • opt: 1
  • debug-isolated-process: 4
• android-em-7-0-x86_64-shippable-lite-qr: 3
  • opt: 3
• windows10-64-2004-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-qr: 3
  • opt: 3
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-11-21&endday=2022-11-27&tree=all

Comment 103

[Intermittent Failures Robot]

15 failures in 3938 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• mozilla-beta: 7
• mozilla-central: 3
• autoland: 2

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 9
  • debug-isolated-process: 8
  • opt: 1
• android-em-7-0-x86_64-shippable-qr: 3
  • opt: 3
• windows10-64-2004-ccov-qr: 2
  • opt: 2
• windows10-64-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-11-28&endday=2022-12-04&tree=all

Comment 104

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Intermittent Failures Robot from comment #103)

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 9
  • debug-isolated-process: 8
  • opt: 1
• android-em-7-0-x86_64-shippable-qr: 3
  • opt: 3

All of these were wrongly classified and mostly are bug 1804027. I've re-classified.

Comment 105

[Intermittent Failures Robot]

12 failures in 3510 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7
• try: 2
• autoland: 2
• mozilla-release: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 7
  • opt: 7
• linux1804-64-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1
• macosx1015-64-qr: 1
  • opt: 1
• windows10-64-2004-qr: 1
  • opt: 1
• windows10-32-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-12-05&endday=2022-12-11&tree=all

Comment 106

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Intermittent Failures Robot from comment #105)

12 failures in 3510 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 7
  • opt: 7

All these CCOV related failures from the last week are actually related to bug 1783217. Can those be re-classified please? Thanks.

Comment 107

[BugBot [:suhaib / :marco/ :calixte]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:whimboo, since the bug has recent activity, could you please find another way to get the information or close the bug as INCOMPLETE if it is not actionable?

For more information, please visit auto_nag documentation.

Comment 108

[Marco Castelluccio [:marco]]

Sorry, the bot was not aware of the [e-mail] account.

Comment 109

[Iulian Moraru]

The failures are re-classified.

Comment 110

[Intermittent Failures Robot]

12 failures in 3558 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 8
• mozilla-central: 2
• mozilla-beta: 2

Platform and build breakdown:

• android-em-7-0-x86_64-shippable-lite-qr: 3
  • opt: 3
• linux1804-64-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 5
  • debug-isolated-process: 5
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1
• macosx1015-64-devedition-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-12-12&endday=2022-12-18&tree=all

Comment 111

[Intermittent Failures Robot]

4 failures in 2999 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 1
• mozilla-esr102: 1
• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1
• macosx1015-64-qr: 1
  • opt: 1
• windows10-64-2004-shippable-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-12-19&endday=2022-12-25&tree=all

Comment 112

[Intermittent Failures Robot]

7 failures in 1328 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-beta: 3
• mozilla-central: 1
• try: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 3
  • debug-isolated-process: 3
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1
• windows10-32-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2022-12-26&endday=2023-01-01&tree=all

Comment 113

[Intermittent Failures Robot]

15 failures in 2607 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• try: 3
• mozilla-release: 1
• mozilla-beta: 4
• mozilla-central: 5

Platform and build breakdown:

• windows10-32-2004-qr: 2
  • opt: 2
• android-em-7-0-x86_64-qr: 5
  • debug-isolated-process: 5
• linux1804-64-tsan-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 4
  • opt: 4
• windows10-32-2004-shippable-qr: 1
  • opt: 1
• windows10-64-2004-devedition-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-01-02&endday=2023-01-08&tree=all

Comment 114

[Intermittent Failures Robot]

18 failures in 3530 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7
• try: 7
• mozilla-release: 1
• autoland: 3

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 2
  • debug-isolated-process: 2
• android-em-7-0-x86_64-shippable-lite-qr: 3
  • opt: 3
• windows10-64-2004-ccov-qr: 4
  • opt: 4
• linux1804-64-asan-qr: 1
  • opt: 1
• windows10-32-2004-qr: 3
  • debug: 2
  • opt: 1
• linux1804-64-tsan-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-01-09&endday=2023-01-15&tree=all

Comment 115

[Intermittent Failures Robot]

24 failures in 3717 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 10
• mozilla-beta: 8
• mozilla-central: 1
• autoland: 4
• mozilla-release: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 10
  • debug-isolated-process: 10
• android-em-7-0-x86_64-shippable-qr: 3
  • opt: 3
• windows10-64-2004-ccov-qr: 1
  • opt: 1
• macosx1015-64-qr: 2
  • opt: 2
• linux1804-64-tsan-qr: 6
  • opt: 6
• windows10-64-2004-qr: 1
  • opt: 1
• windows10-32-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-01-16&endday=2023-01-22&tree=all

Comment 116

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Intermittent Failures Robot from comment #115)

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 10
  • debug-isolated-process: 10
• android-em-7-0-x86_64-shippable-qr: 3
  • opt: 3

All these Android specific failures including others from former weeks are actually related to bug 1804027. Could someone please re-classify? Thanks.

Comment 117

[Noemi Erli[:noemi_erli]]

All the above are now re-classified. Sorry for the inconvenience.

Comment 118

[Intermittent Failures Robot]

14 failures in 3914 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• mozilla-beta: 3
• mozilla-central: 6
• mozilla-release: 1
• autoland: 1

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 5
  • debug-isolated-process: 5
• windows10-64-2004-ccov-qr: 5
  • opt: 5
• linux1804-64-tsan-qr: 2
  • opt: 2
• windows10-32-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-01-23&endday=2023-01-29&tree=all

Comment 119

[Intermittent Failures Robot]

7 failures in 3606 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• mozilla-beta: 3
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-tsan-qr: 5
  • opt: 5
• windows10-32-2004-shippable-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-01-30&endday=2023-02-05&tree=all

Comment 121

[Intermittent Failures Robot]

6 failures in 3766 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 3
• try: 1
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-tsan-qr: 2
  • opt: 2
• android-em-7-0-x86_64-qr: 2
  • debug-isolated-process: 2
• windows10-64-2004-ccov-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-02-06&endday=2023-02-12&tree=all

Comment 122

[Intermittent Failures Robot]

9 failures in 3856 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• mozilla-central: 5
• try: 2
• mozilla-beta: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1
• windows10-64-2004-shippable-qr: 2
  • opt: 2
• linux1804-64-tsan-qr: 2
  • opt: 2
• windows10-64-2004-ccov-qr: 3
  • opt: 3
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-02-13&endday=2023-02-19&tree=all

Comment 123

[BMO Automation]

Moving bug to Remote Protocol::Marionette component per bug 1815831.

Comment 124

[Intermittent Failures Robot]

4 failures in 3643 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• mozilla-central: 2

Platform and build breakdown:

• windows10-64-2004-devedition-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1
• windows10-32-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-02-20&endday=2023-02-26&tree=all

Comment 126

[Intermittent Failures Robot]

12 failures in 3752 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• try: 2
• mozilla-beta: 3
• autoland: 4

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 3
  • opt: 3
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 2
  • debug-isolated-process: 2
• windows10-64-2004-qr: 2
  • debug: 1
  • opt: 1
• linux1804-64-tsan-qr: 3
  • opt: 3
• windows11-32-2009-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-02-27&endday=2023-03-05&tree=all

Comment 127

[Intermittent Failures Robot]

3 failures in 3966 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• try: 2

Platform and build breakdown:

• linux1804-64-tsan-qr: 2
  • opt: 2
• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-03-06&endday=2023-03-12&tree=all

Comment 131

[Intermittent Failures Robot]

11 failures in 3761 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 7
• mozilla-release: 1
• autoland: 2
• try: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 3
  • debug-isolated-process: 3
• linux1804-64-tsan-qr: 7
  • opt: 7
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-03-13&endday=2023-03-19&tree=all

Comment 132

[Intermittent Failures Robot]

14 failures in 4161 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• mozilla-release: 1
• mozilla-esr102: 2
• autoland: 4
• mozilla-beta: 5

Platform and build breakdown:

• linux1804-64-tsan-qr: 7
  • opt: 7
• android-em-7-0-x86_64-qr: 3
  • debug-isolated-process: 2
  • debug: 1
• windows10-64-2004-qr: 1
  • debug: 1
• windows10-32-2004-qr: 1
  • debug: 1
• windows11-64-2009-qr: 1
  • opt: 1
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-03-20&endday=2023-03-26&tree=all

Comment 133

[Intermittent Failures Robot]

10 failures in 3863 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 4
• autoland: 2
• mozilla-central: 3
• try: 1

Platform and build breakdown:

• linux1804-64-tsan-qr: 4
  • opt: 4
• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows11-64-2009-qr: 2
  • debug: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1
• windows11-64-2009-devedition-qr: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-03-27&endday=2023-04-02&tree=all

Comment 134

[Intermittent Failures Robot]

1 failures in 3386 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-04-03&endday=2023-04-09&tree=all

Comment 136

[Intermittent Failures Robot]

11 failures in 3206 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-beta: 3
• autoland: 2
• try: 3
• mozilla-release: 1

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 1
  • opt: 1
• linux1804-64-tsan-qr: 3
  • opt: 3
• windows11-32-2009-qr: 2
  • opt: 1
  • debug: 1
• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows11-64-2009-devedition-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 2
  • debug-isolated-process: 2
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-04-10&endday=2023-04-16&tree=all

Comment 137

[Intermittent Failures Robot]

7 failures in 4004 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 3
• mozilla-central: 1
• try: 2
• mozilla-release: 1

Platform and build breakdown:

• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• linux1804-64-tsan-qr: 3
  • opt: 3
• windows11-64-2009-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-04-17&endday=2023-04-23&tree=all

Comment 138

[Intermittent Failures Robot]

8 failures in 3177 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 5
• mozilla-beta: 1

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 1
  • opt: 1
• linux1804-64-tsan-qr: 3
  • opt: 3
• windows11-32-2009-shippable-qr: 2
  • opt: 2
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-04-24&endday=2023-04-30&tree=all

Comment 139

[Intermittent Failures Robot]

3 failures in 3411 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 2
  • opt: 2
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-05-01&endday=2023-05-07&tree=all

Comment 140

[Intermittent Failures Robot]

7 failures in 4077 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1
• mozilla-beta: 3
• try: 1

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • opt: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 1
  • debug: 1
• linux1804-64-tsan-qr: 3
  • opt: 3
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-05-08&endday=2023-05-14&tree=all

Comment 142

[Intermittent Failures Robot]

13 failures in 3578 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• try: 6
• mozilla-central: 3
• mozilla-beta: 2

Platform and build breakdown:

• windows11-32-2009-qr: 4
  • debug: 4
• linux1804-64-tsan-qr: 2
  • opt: 2
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 2
  • opt: 2
• windows11-32-2009-shippable-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• windows11-64-2009-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-05-15&endday=2023-05-21&tree=all

Comment 143

[Intermittent Failures Robot]

9 failures in 3946 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 4
• mozilla-beta: 3
• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows11-32-2009-qr: 2
  • opt: 1
  • debug: 1
• windows11-64-2009-qr: 3
  • opt: 2
  • debug: 1
• windows11-64-2009-devedition-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1
• macosx1015-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-05-22&endday=2023-05-28&tree=all

Comment 144

[Intermittent Failures Robot]

15 failures in 3471 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 2
• mozilla-central: 7
• try: 1
• autoland: 5

Platform and build breakdown:

• linux1804-64-tsan-qr: 5
  • opt: 5
• windows11-64-2009-qr: 2
  • debug: 1
  • opt: 1
• windows11-32-2009-shippable-qr: 2
  • opt: 2
• macosx1015-64-qr: 2
  • opt: 2
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• linux1804-64-ccov-qr: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-05-29&endday=2023-06-04&tree=all

Comment 145

[Intermittent Failures Robot]

19 failures in 3885 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 7
• try: 6
• mozilla-central: 1
• autoland: 5

Platform and build breakdown:

• linux1804-64-tsan-qr: 5
  • opt: 5
• macosx1015-64-devedition-qr: 1
  • opt: 1
• windows11-32-2009-shippable-qr: 3
  • opt: 3
• linux1804-64-ccov-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows11-64-2009-devedition-qr: 2
  • opt: 2
• windows11-32-2009-qr: 2
  • opt: 1
  • debug: 1
• windows11-64-2009-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-06-05&endday=2023-06-11&tree=all

Comment 146

[Intermittent Failures Robot]

31 failures in 4079 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 3
• mozilla-central: 9
• try: 8
• autoland: 10
• mozilla-release: 1

Platform and build breakdown:

• linux1804-64-tsan-qr: 16
  • opt: 16
• windows11-32-2009-qr: 3
  • opt: 3
• macosx1015-64-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 2
  • opt: 2
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1
• windows11-64-2009-devedition-qr: 2
  • opt: 2
• windows11-32-2009-shippable-qr: 3
  • opt: 3
• windows11-64-2009-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-06-12&endday=2023-06-18&tree=all

Comment 147

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Intermittent Failures Robot from comment #146)

31 failures in 4079 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Most of these failures have been wrongly classified and should actually have been flagged for bug 1540367. I corrected all the classifications.

Comment 148

[Intermittent Failures Robot]

13 failures in 3880 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• mozilla-beta: 9
• mozilla-central: 1
• try: 1
• mozilla-esr102: 1

Platform and build breakdown:

• linux1804-64-tsan-qr: 7
  • opt: 7
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1
• macosx1015-64-devedition-qr: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-32-2009-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows10-64-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-06-19&endday=2023-06-25&tree=all

Comment 150

[Intermittent Failures Robot]

7 failures in 3594 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 2
• autoland: 2
• mozilla-central: 1
• try: 2

Platform and build breakdown:

• linux1804-64-tsan-qr: 5
  • opt: 5
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-64-2009-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-06-26&endday=2023-07-02&tree=all

Comment 151

[Intermittent Failures Robot]

11 failures in 2956 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1
• autoland: 3
• mozilla-central: 3
• mozilla-beta: 2
• try: 2

Platform and build breakdown:

• linux1804-64-tsan-qr: 3
  • opt: 3
• macosx1015-64-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 2
  • opt: 2
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1
• windows11-64-2009-devedition-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-07-03&endday=2023-07-09&tree=all

Comment 152

[Intermittent Failures Robot]

18 failures in 589 pushes (0.031 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 2
• autoland: 3
• mozilla-central: 1
• mozilla-release: 5
• mozilla-esr115: 7

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 2
  • opt: 2
• windows11-64-2009-devedition-qr: 1
  • opt: 1
• macosx1015-64-qr: 1
  • opt: 1
• linux1804-64-tsan-qr: 12
  • opt: 12
• linux1804-64-qr: 1
  • opt: 1
• windows11-32-2009-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-07-10&endday=2023-07-10&tree=all

Comment 153

[Intermittent Failures Robot]

27 failures in 3309 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 5
• autoland: 7
• mozilla-release: 5
• mozilla-central: 1
• mozilla-esr115: 8
• mozilla-esr102: 1

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 2
  • opt: 2
• windows11-64-2009-devedition-qr: 2
  • opt: 2
• macosx1015-64-qr: 1
  • opt: 1
• linux1804-64-tsan-qr: 15
  • opt: 15
• linux1804-64-qr: 1
  • opt: 1
• windows11-32-2009-qr: 2
  • opt: 2
• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1
• windows10-32-2004-qr: 1
  • debug: 1
• windows11-64-2009-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-07-10&endday=2023-07-16&tree=all

Comment 154

[Intermittent Failures Robot]

14 failures in 3520 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1
• mozilla-central: 5
• try: 4
• autoland: 3
• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-tsan-qr: 5
  • opt: 5
• android-em-7-0-x86_64-qr: 1
  • debug: 1
• windows11-64-2009-qr: 1
  • opt: 1
• windows11-32-2009-qr: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 3
  • opt: 3
• windows11-64-2009-devedition-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 1
  • opt: 1
• windows11-32-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-07-17&endday=2023-07-23&tree=all

Comment 155

[Intermittent Failures Robot]

22 failures in 3403 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 2
• try: 9
• mozilla-esr115: 1
• mozilla-beta: 2
• mozilla-release: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 2
  • debug: 2
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• android-5-0-x86_64: 1
  • pgo: 1
• linux1804-64-tsan-qr: 5
  • opt: 5
• windows11-32-2009-shippable-qr: 5
  • opt: 5
• windows11-64-2009-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• android-em-7-0-x86_64-shippable-lite-qr: 2
  • opt: 2
• macosx1015-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-07-24&endday=2023-07-30&tree=all

Comment 157

[Intermittent Failures Robot]

13 failures in 3276 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 1
• mozilla-release: 2
• try: 1
• mozilla-beta: 5

Platform and build breakdown:

• windows11-32-2009-qr: 2
  • opt: 2
• linux1804-64-tsan-qr: 4
  • opt: 4
• windows11-64-2009-shippable-qr: 3
  • opt: 3
• macosx1015-64-shippable-qr: 1
  • opt: 1
• windows11-32-2009-shippable-qr: 1
  • opt: 1
• macosx1015-64-qr: 1
  • opt: 1
• windows11-64-2009-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-07-31&endday=2023-08-06&tree=all

Comment 159

[Henrik Skupin [:whimboo][⌚️UTC+2]]

There are quite a few TSAN related shutdown hangs recently. With the fix on bug 1848121 we should see them most likely go away or at least show up with a way lower rate.

Comment 160

[Intermittent Failures Robot]

22 failures in 3341 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• autoland: 11
• mozilla-central: 7
• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-tsan-qr: 11
  • opt: 11
• windows11-32-2009-qr: 3
  • opt: 3
• windows11-64-2009-qr: 2
  • opt: 2
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• windows11-32-2009-shippable-qr: 2
  • opt: 2
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-08-07&endday=2023-08-13&tree=all

Comment 161

[Intermittent Failures Robot]

25 failures in 3076 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-beta: 8
• mozilla-esr115: 4
• mozilla-central: 6
• try: 2

Platform and build breakdown:

• windows11-64-2009-qr: 4
  • opt: 4
• linux1804-64-devedition-qr: 1
  • opt: 1
• linux1804-64-tsan-qr: 4
  • opt: 4
• android-em-7-0-x86_64-shippable-qr: 1
  • opt: 1
• windows11-32-2009-shippable-qr: 4
  • opt: 4
• windows11-64-2009-shippable-qr: 6
  • opt: 6
• android-em-7-0-x86_64-qr: 1
  • debug: 1
• windows11-64-2009-devedition-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-08-14&endday=2023-08-20&tree=all

Comment 162

[Intermittent Failures Robot]

20 failures in 2218 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #48 most frequent failure this week.

Repository breakdown:

• autoland: 10
• mozilla-central: 4
• mozilla-release: 4
• try: 2

Platform and build breakdown:

• windows11-64-2009-qr: 5
  • opt: 5
• windows11-64-2009-ccov-qr: 3
  • opt: 3
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1
• windows11-32-2009-qr: 5
  • debug: 2
  • opt: 3
• windows11-32-2009-shippable-qr: 1
  • opt: 1
• macosx1015-64-shippable-qr: 2
  • opt: 2
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• windows11-64-2009-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-08-21&endday=2023-08-27&tree=all

Comment 164

[Intermittent Failures Robot]

20 failures in 3019 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 5
• mozilla-esr115: 2
• autoland: 6
• mozilla-central: 4
• mozilla-beta: 3

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 2
  • opt: 2
• windows11-64-2009-devedition-qr: 5
  • opt: 5
• linux1804-64-tsan-qr: 1
  • opt: 1
• windows11-32-2009-qr: 6
  • debug: 1
  • opt: 5
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• windows11-64-2009-qr: 2
  • debug: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 1
  • debug: 1
• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-08-28&endday=2023-09-03&tree=all

Comment 165

[Intermittent Failures Robot]

10 failures in 3267 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 5
• autoland: 1
• mozilla-central: 4

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 2
  • opt: 2
• windows11-64-2009-shippable-qr: 4
  • opt: 4
• android-em-7-0-x86_64-qr: 2
  • debug: 1
  • debug-isolated-process: 1
• linux1804-64-ccov-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-09-04&endday=2023-09-10&tree=all

Comment 166

[Intermittent Failures Robot]

16 failures in 3337 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• mozilla-beta: 4
• autoland: 6
• try: 1

Platform and build breakdown:

• windows11-64-2009-shippable-qr: 4
  • opt: 4
• linux1804-64-tsan-qr: 2
  • opt: 2
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1
• windows11-32-2009-qr: 4
  • opt: 4
• windows11-64-2009-qr: 1
  • opt: 1
• linux1804-64-qr: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-64-2009-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1630162&startday=2023-09-11&endday=2023-09-17&tree=all