2026593 - Split browser/components/extensions/test/browser/browser.toml in smaller parts

Status: RESOLVED FIXED

(WebExtensions :: General, task, P2)

Description

[Rob Wu [:robwu]]

The browser/components/extensions/test/browser/browser.toml test manifest contains many desktop-specific browser tests relating to extensions.

Florian noted that the many tests here cause the whole test run to take a lot of time, and that we should split the manifest to get the individual test manifests to take less time. He shared some context in https://phabricator.services.mozilla.com/D289375#10061180 , pasted below.

The patch there proposes an arbitrary alphabetical partitioning of test manifests. We (WebExtensions engineering) prefer to partition by related features (e.g. separate browser action + page action + extension button, context menus, sidebars, content scripts, etc.).

You can see manifest runtime data at https://tests.firefox.dev/manifests.html
If you filter on browser/components/extensions/test/browser/browser.toml, you'll see that manifest currently takes 41min on tsan, 32min on Windows asan, 29min on Mac debug, 26min on Windows debug, 23min on Linux asan, 22min on Linux debug.
The chunk runtime target for dynamic chunking is 20 minutes, so any manifest that lasts longer than 20 minutes on any of these tier1 configurations will cause outliers in terms of chunk runtime (ie. will delay the end-to-end time of getting results from CI). The outliers are clearly visible on https://tests.firefox.dev/mochitest-jobs.html (although browser/components/preferences/tests/browser.toml is so far off that it kinda hides everything else).

The reason why I filed this bug and made this patch in the first place is that I'm trying to introduce Mochitest standalone jobs, that will enforce tests remain independent from each other. For that, we restart the browser between every test, which adds significant overhead. Having these jobs finish in 20 minutes is quite unrealistic, and I accept that. Currently these jobs are configured to timeout in 90minutes. So any manifest that takes longer than 90 min to complete with a restart between every test will cause perma fails. On Windows debug, the overhead of a browser restart is about 15s.

You can see the runtime of individual tests at https://tests.firefox.dev/xpcshell-timings.html?kind=mochitest, so for the extensions manifest: https://tests.firefox.dev/xpcshell-timings.html?kind=mochitest#date=2026-03-24&q=browser%2Fcomponents%2Fextensions%2Ftest%2Fbrowser&view=list

The raw data files behind these dashboards are at:
https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.try.latest.source.test-info-manifest-timings/artifacts/public/manifests.json
https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.source.test-info-mochitest-timings/artifacts/public/mochitest-2026-03-24.json

Their formats are documented at:

• testing/runtimes/JSON_FORMAT.md
• testing/timings/JSON_FORMAT.md

Comment 1

[Rob Wu [:robwu]]

Luca will prepare a patch to split the test manifest, and I will serve as reviewer.

Comment 2

[Florian Quèze [:florian]]

Bug 2010766 landed with

[DEFAULT]
skip-if = [
  "debug && standalone", # Bug 2026593
]

added to browser/components/extensions/test/browser/browser.toml

When this bug will be fixed, that skip-if can be removed.

Comment 3

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: Bug 2026593 - Split browser/components/extensions/test/browser/browser.toml in smaller per-area toml manifests.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/752616db5ccf
https://hg.mozilla.org/integration/autoland/rev/9f266a284b94
Split browser/components/extensions/test/browser/browser.toml in smaller per-area toml manifests. r=robwu

Comment 5

[rperta]

https://hg.mozilla.org/mozilla-central/rev/9f266a284b94

Comment 6

[Luca Greco [:rpl] [:luca] [:lgreco]]

We'll wait a few more days of baking time, but we intend to request an uplift to beta of this test-only change (mainly to make it easier to deal with uplift to ESR in the future months) and so we will open the uplift request next Monday.

Comment 7

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: Bug 2026593 - Split browser/components/extensions/test/browser/browser.toml in smaller per-area toml manifests.

Original Revision: https://phabricator.services.mozilla.com/D292610

Comment 8

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined/Reason for urgency: No user impact, this is a test-only change.

Our main interest in uplifting this patch to 153 beta is to make it easier to uplift patches to future 153 ESR dot releases.

• Code covered by automated testing?: yes
• Fix verified in Nightly?: yes
• Needs manual QE testing?: no
• Steps to reproduce for manual QE testing:
• Risk associated with taking this patch: low
• Explanation of risk level: This is a test-only change, and so risks are low.
• String changes made/needed?: None
• Is Android affected?: no

Comment 9

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/9bbe8136416c
https://hg.mozilla.org/releases/mozilla-beta/rev/de0970341918