2083786 - Update browsertime 29.x.x+ / latest

Status: NEW

(Testing :: Performance, enhancement, P2)

Description

[Kash Shampur [:kshampur] ⌚EST]

There have been a lot of changes to browsertime since we last updated. We're currently on 26.3.1, which bundles geckodriver 0.36.0. Skimming the changelog https://github.com/sitespeedio/browsertime/blob/e834ee32896e51463a3cb160d075821fe7280386/CHANGELOG.md there are a lot of beneficial changes, and some breaking ones we need to test first.

The bundled geckodriver still enables system access by passing --remote-allow-system-access to Firefox, but Firefox 159+ only supports the MOZ_REMOTE_ALLOW_SYSTEM_ACCESS env var. So local ./mach raptor runs fail with "System access is required" (bug 2083658). CI works because it passes the geckodriver built from the tree via --browsertime-geckodriver.

Upstream browsertime 28.3.1 bumped geckodriver to 0.37.1, which sets the env var, so updating should fix local runs without any raptor change.

Other stuff we'd pick up between 26.3.1 and 28.4.0 (latest as of writing):

• geckodriver with native aarch64 builds
• Firefox network idle timeout fixes
• gecko profile and log post processing is now awaited

Possible breaking changes:

• click commands now use Selenium Actions instead of JavaScript click, so elements need to be visible and interactable. We need to review our scripts that use clicks (mostly the perftest ones in testing/performance)
• --timeouts.elementWait default changed from 0 to 6000ms
• Node 22+ is required (should be fine I believe)

And probably expect some metric shifts in Chrome/Firefox as usual.

If the full update is too disruptive, we could use npm overrides to just pin geckodriver to 0.37.1.

Comment 1

[Kash Shampur [:kshampur] ⌚EST]

there seem to be possible vismet related improvements too which would benefit tp6 + navbench

• https://github.com/sitespeedio/browsertime/pull/2545 (one failed video no longer gives later pages in the iteration wrong metrics)
• https://github.com/sitespeedio/browsertime/pull/2443 (ffmpeg stderr drain, fixes dropped filmstrip frames and possible deadlock)
• https://github.com/sitespeedio/browsertime/pull/2554 (no more sliver crop when viewport detection fails)
• https://github.com/sitespeedio/browsertime/pull/2537 (VisualComplete99 reset fix)
• https://github.com/sitespeedio/browsertime/pull/2444 (SSIM failure no longer drops all other visual metrics)
• https://github.com/sitespeedio/browsertime/pull/2528 (optional videoParams.noiseTolerance for re-raster noise)
• https://github.com/sitespeedio/browsertime/pull/2548 (Android video cleanup actually removes recordings now)
• https://github.com/sitespeedio/browsertime/pull/2489 (navigation script failures include URL and iteration)