Define a high-resolution page load time histogram

RESOLVED FIXED in Firefox 65

Status

()

defect
P1
normal
RESOLVED FIXED
6 months ago
3 months ago

People

(Reporter: tdsmith, Assigned: tdsmith)

Tracking

(Blocks 1 bug)

65 Branch
Firefox 65
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox65 fixed)

Details

Attachments

(3 attachments)

Teams have expressed interest in measuring the effect of browser changes on page load time. Recent examples include [1], [2].

The FX_PAGE_LOAD_MS histogram, implemented in Bug 790213, measures the time it takes Firefox to load a page's static content. Unfortunately, it uses only 20 buckets to cover a 10,000ms range, so resolution is quite poor. The modal bucket is 860ms wide. There is at least anecdotal evidence that increasing page load times by increments as small as 100 ms can alter user behavior [3], so much better resolution is required to have confidence in changes we expect to alter page load times.

I propose defining a FX_PAGE_LOAD_MS_V2 histogram, collected identically, but defined to have 200 linear buckets over the same [0, 10,000] ms range, in order to have 50 ms precision over the entire range of the histogram. I prefer linear buckets because we need high resolution at the mode of the data and do not need additional resolution for very short page loads. The probe should also be enabled for release.

Dominik, I'm interested in your opinion as a perceived performance expert; do you buy my argument here?

I anticipate being able to implement the probe. We should also fix Bug 1364957 while we're here.

[1] WebRender MVP release criteria: https://docs.google.com/document/d/1HSe0XCyPsgxdJpQV9HRpdiLdAd4CnXalyw92sWxKkNM/edit?ts=5bad53b2#
[2] Bug 1462158: Tab throbber FPS SHIELD study
[3] http://services.google.com/fh/files/blogs/google_delayexp.pdf
Flags: needinfo?(dstrohmeier)
Measurement dashboard link for the current probe for easy reference: https://mzl.la/2yOz6TZ
Taking a rough guess at a component here (I'm on Core:General triage this week :).
Component: General → Telemetry
Product: Core → Toolkit

Comment 3

6 months ago
Measurement requests go in the component where the measurement code will live. Going by FX_PAGE_LOAD_MS' location in browser.js, and browser.js' log having 2 out of the most recent 5 commits' bugs coming from Firefox::Tabbed Browser, let's try there :)
Component: Telemetry → Tabbed Browser
Product: Toolkit → Firefox
I agree with Tim's point from a perceptual perspective. There is more than one case which shows that small changes to page load performance had an impact on user churn, conversion rates etc. Although the identified improvements differ from case to case, I agree that buckets of smaller size like the proposed 50ms buckets make sense.
Flags: needinfo?(dstrohmeier)
Assignee: nobody → tdsmith
Status: NEW → ASSIGNED
See Also: → 1501470
Priority: -- → P1
Just a quick note and fyi, I don't consider this bug to be a "blocking beta" bug for WebRender.  Typically P1 WebRender bugs blocking Bug 1386669 are beta blockers for WR, but this bug is outside the WR component where the triage process is a little different.
Posted file Data collection review β€”
Attachment #9023143 - Flags: review?(francois)
(In reply to Maire Reavy [:mreavy] Plz needinfo from comment #5)
> Just a quick note and fyi, I don't consider this bug to be a "blocking beta"
> bug for WebRender.  Typically P1 WebRender bugs blocking Bug 1386669 are
> beta blockers for WR, but this bug is outside the WR component where the
> triage process is a little different.

As such, I think it's probably clearer for this not to block bug 1386669.
No longer blocks: stage-wr-trains
The FX_PAGE_LOAD_MS histogram, implemented in Bug 790213, measures the time it takes
Firefox to load a page's static content. Unfortunately, it uses only 20 buckets to cover
a 10,000ms range, so resolution is quite poor. The modal bucket is 860ms wide. There is
at least anecdotal evidence that increasing page load times by increments as small as 100 ms
can alter user behavior, so much better resolution is required to have confidence in changes
we expect to alter page load times.

Also fixes Bug 1364957.
Comment on attachment 9023143 [details]
Data collection review

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, in Histograms.json.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, telemetry setting.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Yes, Rebecca Weiss, as the head of the product data science team.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 1 (performance data).

5) Is the data collection request for default-on or default-off?

Default ON, all channels.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No, permanent.
Attachment #9023143 - Flags: review?(francois) → review+
Depends on: 1364957

Comment 10

5 months ago
Pushed by tismith@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a4b64b6bd26b
Define a high-resolution page load time histogram r=dao,chutten

Comment 11

5 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/a4b64b6bd26b
Status: ASSIGNED → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 65
Duplicate of this bug: 1364957
You need to log in before you can comment on or make changes to this bug.