2012405 - Intermittent Windows Debug Mn [taskcluster:error] Aborting task... due to invalid maxRunTime

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: hskupin [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=545325593&repo=autoland&task=CKNdoyZXTGqJoUqndJFaKA.0
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/CKNdoyZXTGqJoUqndJFaKA/runs/0/artifacts/public/logs/live_backing.log

---

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Based on the intermittent failure views list (when querying for windows11.*debug.*marionette) this task timeout is already happening for quite some time on Windows Debug builds. Nearly all the time its the integration job that is failing.

We should check if we may have to add a new chunk because lots of new tests were added or if there is a specific job which causes the delay.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As it looks like the first failure as reported is from 2025-10-03 09:29:20.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

This bug has been marked as a regression. Setting status flag for Nightly to affected.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For integration tests there is a clear pattern of very long running tests (here for a debug build):

Duration (ms) Status       Test Name
====================================================================================================
61471        TEST-PASS    browser/components/backup/tests/marionette/test_backup.py BackupTest.test_backup
51293        TEST-PASS    toolkit/xre/test/marionette/test_fission_autostart.py TestFissionAutostart.test_fission_precedence
38350        TEST-PASS    browser/components/customizableui/test/marionette/test_assert_no_toolbar_changes.py TestNoToolbarChanges.test_no_toolbar_changes
34247        TEST-PASS    dom/cache/test/marionette/test_caches_delete_cleanup_after_shutdown.py CachesDeleteCleanupAtShutdownTestCase.test_ensure_cache_cleanup_after_unclean_restart
34186        TEST-PASS    browser/components/sessionstore/test/marionette/test_restore_windows_after_close_last_tabs.py TestSessionStoreEnabledAllWindows.test_with_variety
34102        TEST-PASS    browser/components/sessionstore/test/marionette/test_restore_windows_after_restart_and_quit.py TestSessionStoreEnabledAllWindows.test_with_variety
33541        TEST-PASS    browser/components/sessionstore/test/marionette/test_restore_windows_after_restart_and_quit.py TestSessionStoreDisabled.test_restore_with_restart
33286        TEST-PASS    browser/components/sessionstore/test/marionette/test_restore_manually_with_tab_groups.py TestSessionRestoreWithTabGroups.test_open_groups_become_permanently_saved_after_two_quits_bug1954488
32837        TEST-PASS    browser/components/sessionstore/test/marionette/test_restore_sidebar.py TestSessionRestore.test_restore_for_always_show
32790        TEST-PASS    browser/components/sessionstore/test/marionette/test_restore_sidebar.py TestSessionRestore.test_upgrade_profile_and_restore_sidebar_from_backup_pref
32382        TEST-PASS    browser/components/sessionstore/test/marionette/test_restore_windows_after_close_last_tabs.py TestSessionStoreEnabledAllWindows.test_close_tabs
32292        TEST-PASS    browser/components/sessionstore/test/marionette/test_restore_sidebar.py TestSessionRestore.test_restore_sidebar_open_from_backup_pref
32179        TEST-PASS    browser/components/sessionstore/test/marionette/test_tabgroups_restore.py TestAutoRestoreWithTabGroups.test_saved_groups_restored_after_quit
32132        TEST-PASS    browser/components/sessionstore/test/marionette/test_tabgroups_restore.py TestAutoRestoreWithTabGroups.test_saved_groups_restored_after_closing_last_window
31992        TEST-PASS    browser/components/sessionstore/test/marionette/test_restore_windows_after_restart_and_quit.py TestSessionStoreDisabled.test_no_restore_with_quit
31911        TEST-PASS    browser/components/sessionstore/test/marionette/test_restore_sidebar.py TestSessionRestore.test_restore_for_hide_sidebar
31030        TEST-PASS    browser/components/sidebar/tests/marionette/test_initialize_vertical_tabs.py TestInitializeVerticalTabs.test_preserve_visibility_pref_after_restart
30792        TEST-PASS    browser/components/sidebar/tests/marionette/test_initialize_vertical_tabs.py TestInitializeVerticalTabs.test_restore_tabstrip_customizations
30447        TEST-PASS    toolkit/xre/test/marionette/test_win32k_enrollment.py TestWin32kAutostart.test_25
30345        TEST-PASS    browser/components/sidebar/tests/marionette/test_default_launcher_visible.py TestDefaultLauncherVisible.test_first_use_default_visible_pref_false

And here results for a job running the Marionette unit tests (as well for a debug. build):

Duration (ms) Status       Test Name
====================================================================================================
26242        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_prefs_enforce.py TestEnforcePreferences.test_restart_with_clean_profile_after_enforce_prefs
25527        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_prefs_enforce.py TestEnforcePreferences.test_change_enforced_preference
21521        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestSwitchProfileWithoutWorkspace.test_replace_with_external_profile
20941        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestSwitchProfileWithoutWorkspace.test_new_named_profile
20351        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestSwitchProfileWithWorkspace.test_new_named_profile
20020        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_cli_arguments.py TestCommandLineArguments.test_startup_timeout
19848        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestSwitchProfileWithoutWorkspace.test_new_named_profile_unicode_escape_characters
19784        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestSwitchProfileWithoutWorkspace.test_new_named_profile_unicode
19681        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestSwitchProfileWithWorkspace.test_clone_existing_profile
19394        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestSwitchProfileWithWorkspace.test_new_random_profile_name
18920        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestSwitchProfileWithoutWorkspace.test_clone_existing_profile
18375        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestSwitchProfileWithoutWorkspace.test_new_random_profile_name
17757        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_quit_restart.py TestQuitRestart.test_restart_safe_mode
17632        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_cli_arguments.py TestCommandLineArguments.test_websocket_url
17627        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithoutWorkspace.test_quit_keeps_same_profile_safe
17619        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_prefs_enforce.py TestEnforcePreferences.test_preferences_are_set
17326        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_cli_arguments.py TestCommandLineArguments.test_start_page_about_blank
17146        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_prefs_enforce.py TestEnforcePreferences.test_restart_preserves_requested_capabilities
17003        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithWorkspace.test_quit_clean_creates_new_profile
16896        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestSwitchProfileWithoutWorkspace.test_replace_with_current_profile
16835        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithoutWorkspace.test_restart_keeps_same_profile_safe
16717        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithWorkspace.test_restart_keeps_same_profile_safe
16595        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithoutWorkspace.test_restart_clean_creates_new_profile
16514        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithoutWorkspace.test_quit_clean_creates_new_profile
16369        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithoutWorkspace.test_restart_keeps_same_profile_forced
16362        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithWorkspace.test_quit_keeps_same_profile_forced
16359        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithoutWorkspace.test_quit_keeps_same_profile_forced
16044        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithWorkspace.test_restart_clean_creates_new_profile
15957        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py TestBackForwardNavigation.test_timeout_error
15699        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithWorkspace.test_quit_keeps_same_profile_safe
15662        TEST-PASS    testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py TestQuitRestartWithWorkspace.test_restart_keeps_same_profile_forced

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

All of these long-running tests restart Firefox one or more times. Those restarts account for most of the execution time, and test runs are further slowed down in CI due to less powerful machines. For this reason, we should increase the task timeouts for both unit and integration tests on debug and ASan builds.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hm, so for integration tests for a debug build the task definition actually shows a maxRuntime of 2700s. This is the default value for integration tests but not what we have defined for debug tests, which should have 3600s.

There was bug 1947310 landed about 10 months ago which did the following change:

-            .*asan.*: 3600
-            .*tsan.*: 3600
-            .*ccov.*: 3600
-            .*debug.*: 3600
+            .*-asan.*: 3600
+            .*-tsan.*: 3600
+            .*-ccov.*: 3600
+            .*-debug.*: 3600

The debug entry most likely needs to be .*/debug. For unit tests we fallback to the default entry which is 1800s as for opt builds.

Given that this mainly affects only debug builds lets make this bug specific to these builds. We can care about ASAN in a different bug if it becomes necessary.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 2012405 - [marionette] Extend maxRunTime for unit and integration tests on Windows Debug.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Henrik Skupin [:whimboo][⌚️UTC+1] from comment #6)

There was bug 1947310 landed about 10 months ago which did the following change:

This bug indeed caused the regression.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/5be70b934474
https://hg.mozilla.org/integration/autoland/rev/3ebd581a0216
[marionette] Extend maxRunTime for unit and integration tests on Windows Debug. r=ci-and-tooling,bhearsum,aryx DONTBUILD

Comment 10

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/3ebd581a0216

Comment 11

[amarc]

There has been another occurence after this landed, could you please take a look?
https://treeherder.mozilla.org/logviewer?job_id=546240003&repo=mozilla-central&task=Yq3X0jx4RHeSd-52c0xF8g.0

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to amarc from comment #11)

There has been another occurence after this landed, could you please take a look?
https://treeherder.mozilla.org/logviewer?job_id=546240003&repo=mozilla-central&task=Yq3X0jx4RHeSd-52c0xF8g.0

Yes, task timeouts can still happen after this fix as it always can be. But the number of failures is now drastically reduced. Lets classify again against bug 1809667 and I'll keep an eye on it how often it still happens. If the failure rate is still high we might have to add more chunks.

Comment 13

[Intermittent Failures Robot]

37 failures were associated with this bug in the last 7 days.

This failure happened more than 30 times this week! Resolving this bug is a high priority.

Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled.

Repository breakdown:

• mozilla-central: 14
• mozilla-beta: 4
• autoland: 17
• try: 1
• mozilla-esr140: 1

Table

	no_variant	swr
windows11-x86/debug	12	5
windows11-x86/opt	1
windows11-x86_64/asan	1
windows11-x86_64/debug	13	4
windows11-x86_64/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2012405&startday=2026-01-26&endday=2026-02-01&tree=all