1504247 - Define a high-resolution page load time histogram

Status: RESOLVED FIXED

(Firefox :: Tabbed Browser, defect, P1)

Description

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Teams have expressed interest in measuring the effect of browser changes on page load time. Recent examples include [1], [2].

The FX_PAGE_LOAD_MS histogram, implemented in Bug 790213, measures the time it takes Firefox to load a page's static content. Unfortunately, it uses only 20 buckets to cover a 10,000ms range, so resolution is quite poor. The modal bucket is 860ms wide. There is at least anecdotal evidence that increasing page load times by increments as small as 100 ms can alter user behavior [3], so much better resolution is required to have confidence in changes we expect to alter page load times.

I propose defining a FX_PAGE_LOAD_MS_V2 histogram, collected identically, but defined to have 200 linear buckets over the same [0, 10,000] ms range, in order to have 50 ms precision over the entire range of the histogram. I prefer linear buckets because we need high resolution at the mode of the data and do not need additional resolution for very short page loads. The probe should also be enabled for release.

Dominik, I'm interested in your opinion as a perceived performance expert; do you buy my argument here?

I anticipate being able to implement the probe. We should also fix Bug 1364957 while we're here.

[1] WebRender MVP release criteria: https://docs.google.com/document/d/1HSe0XCyPsgxdJpQV9HRpdiLdAd4CnXalyw92sWxKkNM/edit?ts=5bad53b2#
[2] Bug 1462158: Tab throbber FPS SHIELD study
[3] http://services.google.com/fh/files/blogs/google_delayexp.pdf

Comment 1

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Measurement dashboard link for the current probe for easy reference: https://mzl.la/2yOz6TZ

Comment 2

[Andrew Overholt [:overholt]]

Taking a rough guess at a component here (I'm on Core:General triage this week :).

Comment 3

[Chris H-C :chutten]

Measurement requests go in the component where the measurement code will live. Going by FX_PAGE_LOAD_MS' location in browser.js, and browser.js' log having 2 out of the most recent 5 commits' bugs coming from Firefox::Tabbed Browser, let's try there :)

Comment 4

[:Dominik Strohmeier [:dstrohmeier]]

I agree with Tim's point from a perceptual perspective. There is more than one case which shows that small changes to page load performance had an impact on user churn, conversion rates etc. Although the identified improvements differ from case to case, I agree that buckets of smaller size like the proposed 50ms buckets make sense.

Comment 5

[Maire Reavy [:mreavy]]

Just a quick note and fyi, I don't consider this bug to be a "blocking beta" bug for WebRender.  Typically P1 WebRender bugs blocking Bug 1386669 are beta blockers for WR, but this bug is outside the WR component where the triage process is a little different.

Comment 6

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Attachment: Data collection review

Comment 7

[Bobby Holley (:bholley)]

(In reply to Maire Reavy [:mreavy] Plz needinfo from comment #5)
> Just a quick note and fyi, I don't consider this bug to be a "blocking beta"
> bug for WebRender.  Typically P1 WebRender bugs blocking Bug 1386669 are
> beta blockers for WR, but this bug is outside the WR component where the
> triage process is a little different.

As such, I think it's probably clearer for this not to block bug 1386669.

Comment 8

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Attachment: Bug 1504247 - Define a high-resolution page load time histogram r?dao!,chutten!

The FX_PAGE_LOAD_MS histogram, implemented in Bug 790213, measures the time it takes
Firefox to load a page's static content. Unfortunately, it uses only 20 buckets to cover
a 10,000ms range, so resolution is quite poor. The modal bucket is 860ms wide. There is
at least anecdotal evidence that increasing page load times by increments as small as 100 ms
can alter user behavior, so much better resolution is required to have confidence in changes
we expect to alter page load times.

Also fixes Bug 1364957.

Comment 9

[François Marier [:francois]]

Comment on attachment 9023143 [details]
Data collection review

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, in Histograms.json.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, telemetry setting.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Yes, Rebecca Weiss, as the head of the product data science team.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 1 (performance data).

5) Is the data collection request for default-on or default-off?

Default ON, all channels.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No, permanent.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a4b64b6bd26b
Define a high-resolution page load time histogram r=dao,chutten

Comment 11

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/a4b64b6bd26b

Comment 12

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-batch-view/pull/515