2072711 - [meta] Fix the flakiness in browser/extensions/newtab/test/browser

Status: ASSIGNED

(Firefox :: New Tab Page, task)

Description

[Florian Quèze [:florian]]

Meta bug tracking the flakiness burndown of browser/extensions/newtab/test/browser, and the blocking target for the individual fixes.

Over the 21 days to 2026-09-15 the folder produced 1,528 issues -- 757 fail and 771 timeout -- in 23 of its 29 test files, (fx-tests issues --path browser/extensions/newtab/test/browser --group-by test --harness mochitest to get this data from a CLI). browser_topsites_section.js (494), browser_newtab_side_by_side_layout.js (309) and browser_newtab_spaces.js (228) are most of it. Ten of the per-test tracking bugs had been closed for inactivity rather than fixed.

Every issue in the window now has a named cause. Largest first:

• waitForPreloaded in the folder's head.js waits for a load event that has already been dispatched, so the test hangs until the harness kills it. 705 of the 771 timeouts, over seven tests. Fix in bug 2064559; bug 2025752, bug 2031557, bug 1970996, bug 1960245 and bug 2023930 each hold a share of the volume.
• The first tile in the top sites row is not the one the test configured: seven places address a tile by position and get either the @amazon search shortcut TopSitesFeed pins into browser.newtabpage.pinned -- a real pref, which no pref environment pops -- or the add-shortcut tile, which has no context menu. bug 2031620, bug 1958336, bug 2011958, bug 2061014, bug 2025752, bug 2031557. Frecency and pins left by earlier tests in the same job displace the row as well.
• A dismissed top site stays in the rendered row: TopSitesFeed.refresh's generation guard discards a superseded refresh's broadcast intent along with its stale links, so TOP_SITES_UPDATED reaches the preloaded tab alone. 53 failures, the folder's largest single signature, product-side, fixed under bug 2031557.
• A preloaded about:newtab built before the test's pushPrefEnv is handed over anyway, blank or with stale state: bug 1970996, bug 2023930.
• panel-list sets its open attribute two animation frames before it has positioned itself and moved focus: bug 2025752, bug 1960245.
• AboutNewTab.activityStream is read before onBrowserReady() assigns it, in two tests. No existing bug carries that signature.
• A train-hopped XPI is missing newly-landed Glean metrics and the recording throws before the newtab ping is submitted: bug 2067207.
• The thumbnail expiry timer deletes the file browser_getScreenshots.js is about to read: bug 1781344.
• A timing-out test makes the leak check blame an earlier, innocent test: 68 failures, filed against Testing :: Mochitest.
• waiting for vsync to be disabled, 20 failures over 13 tests, is two things, and not the infinite CSS placeholder animations it was put down to. Thirteen are a requestAnimationFrame outstanding in a hidden document, which spins the refresh driver at vsync rate without ever running it -- bug 2022285, plus a new Core :: Layout bug for the general behaviour. The other seven are one degraded Linux job that spent 6.07 s in a single parent-process Composite, and need no fix.
• Two singletons: a content process that died during launch, which BrowserTestUtils.browserLoaded turns into a 100-second timeout because it has no crash path, and a poll for a CSS transition's end state in a job that cannot paint.
• A newtab feature-gate flip during a preloaded page's mount leaves the wrong search bar until reload, because an in-flight loadComponent() sets the ref the creating effect guards on.
• Nineteen checks in the folder cannot fail, including every assertion in browser_discovery_render.js, which reports Subtests passed 1/1 with nothing failable in it: TestUtils.waitForCondition emits no assertion on success, so none of them appeared in any log. That is how a product defect four profiles show plainly reached a reader as a tile-counting problem.

Two lessons are about the folder rather than single tests. Address tiles and menu items by name (href, data-l10n-id) and never by position -- five bugs over five tests were all the same positional hazard. And the XPI ships generated, host-version-specific content that a train-hop can leave missing or stale: runtime-metrics-<N>.json (bug 2067207) and an en-US Fluent snapshot (bug 2031557 comment 21).

Comment 1

[Florian Quèze [:florian]]

Attachment: Bug 2072711 - Restore the diagnostic on newtab's #root render wait, r=#home-newtab-reviewers.

TestUtils.waitForCondition rejects immediately when its condition throws, so
an absent #root used to name itself in the failure message. The optional
chaining bug 2016217 added makes the condition evaluate to undefined instead,
so the wait now reports a bare timeout, the same message a #root that exists
but never gets children produces. #root is in the prerendered markup, so
there is nothing for the check to guard.

Comment 2

[Florian Quèze [:florian]]

Attachment: Bug 2072711 - Remove unreachable skip-if conditions from the newtab browser test manifest, r=#home-newtab-reviewers.

socketprocess_networking can no longer be true: bug 2004381 removed the variant
that set it, and bug 2057871, which the remaining condition cites, is fixed.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/52cab01dfd77
https://hg.mozilla.org/integration/autoland/rev/5296620fe01b
Restore the diagnostic on newtab's #root render wait, r=mconley.
https://github.com/mozilla-firefox/firefox/commit/7ce5ab13c147
https://hg.mozilla.org/integration/autoland/rev/d7810ab9eda0
Remove unreachable skip-if conditions from the newtab browser test manifest, r=mconley.

Comment 4

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/5296620fe01b
https://hg.mozilla.org/mozilla-central/rev/d7810ab9eda0