1916456 - allow marionette to run in chunks instead of a single task

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, task, P2)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

marionette only runs as a single task right now. In order to split this up to reduce runtime on windows/asan, we need to run in chunks. There are a few ways to do this:

1. add cli params --total-chunks X --this-chunk Y - this lets the harness do the work to split the tests up
2. use taskgraph chunking - this sends in a list of manifests to the harness to run, a different list for each chunk

I am proposing solving this with #2, as almost all test harnesses work this way now. But in doing this, we will switch to a model where the taskgraph decides which manifests to run and will effectively run them in alphanumerical order, this is not how tests are run now, as they are ordered in the unit-tests.toml file.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1916456 - split marionette asan into two chunks. r=whimboo

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We need to fix the test on bug 1898477 before we can use multiple chunks.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The junking will help to see lesser task timeouts as covered on bug 1809667.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Joel, can you please try again after pulling and merging latest mozilla-central? It should hopefully work now.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I have a try push:
https://treeherder.mozilla.org/jobs?repo=try&revision=507cb9e7f1e18232ea83fd69df4bbd6685f4c281

the exit codes test is still failing on linux (waiting on windows).

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That's interesting. There must be some other test which as well seems to not correctly reset the state of Firefox on teardown. I might have to dig into that. For now here the list of tests as run before the failing test:

browser/components/sessionstore/test/marionette/test_restore_windows_after_windows_shutdown.py
browser/components/sessionstore/test/marionette/test_persist_closed_tabs_restore_manually.py TestSessionRestoreClosedTabs.test_restore
browser/components/sessionstore/test/marionette/test_restore_loading_tab.py TestRestoreLoadingPage.test_by_script
browser/components/sessionstore/test/marionette/test_restore_loading_tab.py TestRestoreLoadingPage.test_target_blank
browser/components/sessionstore/test/marionette/test_restore_loading_tab.py TestRestoreLoadingPage.test_target_other
browser/components/sessionstore/test/marionette/test_restore_manually.py TestSessionRestoreManually.test_restore
browser/components/sessionstore/test/marionette/test_restore_manually_with_pinned_tabs.py TestSessionRestoreWithPinnedTabs.test_no_restore_with_quit
browser/components/sessionstore/test/marionette/test_restore_sidebar_automatic.py TestSessionRestore.test_restore
browser/components/sessionstore/test/marionette/test_restore_sidebar.py TestSessionRestore.test_restore_for_always_show
browser/components/sessionstore/test/marionette/test_restore_sidebar.py TestSessionRestore.test_restore_for_hide_sidebar
browser/components/sessionstore/test/marionette/test_restore_sidebar.py TestSessionRestore.test_restore_sidebar_closed
browser/components/sessionstore/test/marionette/test_restore_sidebar.py TestSessionRestore.test_restore_sidebar_open
browser/components/sessionstore/test/marionette/test_restore_windows_after_close_last_tabs.py TestSessionStoreEnabledAllWindows.test_close_tabs
browser/components/sessionstore/test/marionette/test_restore_windows_after_close_last_tabs.py TestSessionStoreEnabledAllWindows.test_with_variety
browser/components/sessionstore/test/marionette/test_restore_windows_after_restart_and_quit.py TestSessionStoreDisabled.test_no_restore_with_quit
browser/components/sessionstore/test/marionette/test_restore_windows_after_restart_and_quit.py TestSessionStoreDisabled.test_restore_with_restart
browser/components/sessionstore/test/marionette/test_restore_windows_after_restart_and_quit.py TestSessionStoreEnabledAllWindows.test_with_variety
browser/components/sessionstore/test/marionette/test_restore_windows_after_restart_and_quit.py TestSessionStoreEnabledNoPrivateWindows.test_with_variety
dom/cache/test/marionette/test_cacheapi_encryption_PBM.py CacheAPIEncryptionPBM.test_encrypted_request_response_ondisk
dom/cache/test/marionette/test_cacheapi_encryption_PBM.py CacheAPIEncryptionPBM.test_request_response_ondisk
dom/cache/test/marionette/test_caches_delete_cleanup_after_shutdown.py CachesDeleteCleanupAtShutdownTestCase.test_ensure_cache_cleanup_after_clean_restart
dom/cache/test/marionette/test_caches_delete_cleanup_after_shutdown.py CachesDeleteCleanupAtShutdownTestCase.test_ensure_cache_cleanup_after_unclean_restart
dom/quota/test/marionette/test_private_repository_cleanup.py PBM.test_ensure_cleanup
dom/quota/test/marionette/test_private_repository_cleanup.py PBM.test_ensure_cleanup_after_crash
dom/quota/test/marionette/test_private_repository_cleanup.py PBMAutoStart.test_ensure_cleanup
dom/quota/test/marionette/test_private_repository_cleanup.py PBMAutoStart.test_ensure_cleanup_after_crash
extensions/pref/autoconfig/test/marionette/test_autoconfig.py TestAutoConfig.test_autoconfig
netwerk/test/marionette/test_purge_http_cache_at_shutdown.py PurgeHTTPCacheAtShutdownTestCase.test_asetup
netwerk/test/marionette/test_purge_http_cache_at_shutdown.py PurgeHTTPCacheAtShutdownTestCase.test_ensure_cache_purge_after_forced_restart
netwerk/test/marionette/test_purge_http_cache_at_shutdown.py PurgeHTTPCacheAtShutdownTestCase.test_ensure_cache_purge_after_in_app_quit
netwerk/test/marionette/test_purge_http_cache_at_shutdown.py PurgeHTTPCacheAtShutdownTestCase.test_longstanding_cache_purge_after_in_app_quit
toolkit/components/antitracking/bouncetrackingprotection/test/marionette/test_bouncetracking_storage_persistence.py BounceTrackingStoragePersistenceTestCase.test_state_after_restart
toolkit/xre/test/marionette/test_exitcode.py TestFissionAutostart.test_exit_code
toolkit/xre/test/marionette/test_exitcode.py TestFissionAutostart.test_normal_exit

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

so in this case wouldn't it be something that is run in chunk1 that is responsible for putting the browser in a state that allows test_exitcode.py to run properly?

keep in mind the way I am chunking ends up making everything in alphanumeric format

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I was able to reproduce and with the fix on bug 1917243 it should finally work.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Joel, please rebase and try again. Now it should hopefully work without issues.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/714f6b3f0620
split marionette asan into two chunks. r=whimboo,webdriver-reviewers,taskgraph-reviewers,releng-reviewers,ahal

Comment 11

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0b0060941fe9
Backed out changeset 714f6b3f0620 for causing marionette Mn failures. CLOSED TREE

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

this was backed out, but a try push yields green. The failure appears to be it cannot find a manifest.toml:
[task 2024-09-20T21:02:21.776Z] 21:02:21 INFO - runtests.py: error: Test file(s) not found: /builds/worker/workspace/build/tests/marionette/tests/toolkit/modules/tests/marionette/manifest.toml

I am confused because this file exists in the tree. Maybe something changed on autoland and hasn't made it to trunk yet?

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #12)

[task 2024-09-20T21:02:21.776Z] 21:02:21 INFO - runtests.py: error: Test file(s) not found: /builds/worker/workspace/build/tests/marionette/tests/toolkit/modules/tests/marionette/manifest.toml

I am confused because this file exists in the tree. Maybe something changed on autoland and hasn't made it to trunk yet?

This file is not present in the common test archive where all the Marionette tests are contained. Also when I check recent Mn jobs on mozilla-central I cannot see that this test is actually run? It should cause a problem there as well.

Nevertheless it looks like the test was never added to:
https://searchfox.org/mozilla-central/source/testing/marionette/harness/marionette_harness/tests/unit-tests.toml#44-48

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Oh so chunking is most likely based on the manifest files, which means we currently just silently ignore the test and don't run it at all. With chunking we now actually see that some test is not run!

Should chunking maybe done by the MARIONETTE_MANIFESTS settings instead of the .toml file? We basically should get rid of the toml at some point.

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

yes, we effectively deprecate the unit-tests.toml file with this change. Which is probably a good thing, but needs other changes as well.

I am still confused why this fails on autoland and works on try.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Did it always fail on autoland? At least for wdspec tests we know that not all tests are running but each time a number of tests is selected. So maybe that is similar here and this extra logic causes us to fail on autoland?

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

on try, we ignored manifests there were run-if = 'buildapp == 'browser', I added that condition to the guess_mozinfo_from_task, and I can reproduce the problem.

now I can work on a fix!

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

I have confirmed that this one single manifest is all that is missing. Given that, I need a fix to ensure it is copied into common.tests.tar.gz and I suspect we should be good.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Building the common test archive happens in test_archive.py and our Marionette related configuration is here:
https://searchfox.org/mozilla-central/rev/af15acc88dfa0b8567cef768bd7da56cb65dfa18/python/mozbuild/mozbuild/action/test_archive.py#112

Note that if the one manifest file is missing in our unit-tests.toml file it will not be copied to the test archive.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8ce6008b274b
split marionette asan into two chunks. r=whimboo,webdriver-reviewers,taskgraph-reviewers,releng-reviewers,ahal

Comment 21

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/8ce6008b274b