2090090 - nav-bench visual metrics include browsertime's FOG reset before each navigation

Status: ASSIGNED

(Testing :: Raptor, defect, P1)

Description

[Andrew Creskey [:acreskey]]

nav-bench starts each measurement with measure.start(alias) and no URL, then navigates in a separate command. In that path browsertime starts the video recording first and then calls the Firefox delegate's beforeEachURL, which runs Services.fog.testFlushAllChildren() and Services.fog.testResetFOG() (browsertime/lib/firefox/webdriver/firefox.js).

https://github.com/sitespeedio/browsertime/blob/1f77635dbc5d022e27b1b4200a3673518d30e04c/lib/firefox/webdriver/firefox.js#L154

The reset runs inside the recorded window, so its cost is added to FirstVisualChange and SpeedIndex for every page.

The reset only exists so that browsertime can read a per-page Glean.power.totalCpuTimeMs afterwards. nav-bench does not use that value.

This became visible with a Glean change that moves storage to SQLite: testResetFOG now closes the database (WAL checkpoint plus fsync) and blocks the parent main thread in glean_core::join_init while Glean reinitializes. On Linux, nav-bench SpeedIndex regressed 3-23% on every site:
https://perf.compare/compare-lando-results?landoInstance=lando-prod-2025&baseLando=114439&newLando=114440&baseRepo=try&newRepo=try&framework=13&sort=median-diff%7Cdesc&initialized=1&search=nav-bench+-profile

The browser was not slower in that comparison: median TTFB, FCP and loadEventEnd are unchanged or slightly better on every site, while FirstVisualChange is 40 ms later on every site (one frame at 25 fps).

Profile (google-nav-load, parent process main thread): the 68.7 ms Perform microtasks block starting with the fog.testResetFOG marker, right before WebDriver:Navigate:
https://share.firefox.dev/4ePHcQc

Options:

• Stop resetting FOG inside the measured window for nav-bench, for example by patching browsertime to call beforeEachURL before _startVideo, or to skip the reset when the CPU metric is not needed.
• Upstream, read totalCpuTimeMs before and after each page and subtract, instead of resetting FOG.

Comment 1

[Andrew Creskey [:acreskey]]

We might want to remove this from upstream Browsertime in the general case.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

@acreskey assigning to you and setting priority/severity during grooming. Please correct if you feel these are wrong.

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

I'll wait until this is in upstream, then we can update browsertime (bug 2083786)

(since a btime upgrade can sometimes be quite disruptive, id prefer to not do it back to back)

Comment 4

[Andrew Creskey [:acreskey]]

Upstream browsertime bug: https://github.com/sitespeedio/browsertime/issues/2654