1671729 - Collect telemetry on page load and animation frame callback times

Status: RESOLVED FIXED

(Core :: Performance, enhancement)

Description

[Denis Palmeiro [:denispal]]

FX_PAGE_LOAD_MS_2, and the navigation probes TIME_TO_LOAD_EVENT_START_MS, etc, will collect data for content that is not http/https. This includes opening a new tab, navigating to about:blank, any extension content, etc. FX_PAGE_LOAD_MS_2 will also stop timing when a user navigates off the page through switching tabs or clicking on a link.

There is a need for some metrics that measure page load time on http and https content only, comparing against both navigationStart and responseStart to see if some network noise can be eliminated from the measurement.

Additionally, we should consider measuring the time it takes for an animation frame callback so that it can be used to measure improvements but also identify any pathological cases where it is not performing well.

Comment 1

[Denis Palmeiro [:denispal]]

Attachment: data-review-1671729.md

Comment 2

[Denis Palmeiro [:denispal]]

Attachment: Gather telemetry on page load and request animation frame callback time

Comment 3

[Denis Palmeiro [:denispal]]

Comment on attachment 9182396 [details]
data-review-1671729.md

># Request for data collection review form
>
>1) What questions will you answer with this data?
>
>What is the current page load times for real websites out in the wild, measured both against navigation start and response start, the latter of which is used in an attempt to try and eliminate some network noise.  Furthermore, these metrics, along with the animation frame callback time, will be used to monitor improvements or regressions in the wild caused by changes in the browser.
>
>2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:
>
>This can lead to improved performance for page loads and to investigate whether planned improvements are reflected on real users.
>
>3) What alternative methods did you consider to answer these questions? Why were they not sufficient?
>
>We have some local testing in CI to cover some aspects of this, such as page load, but is very minimal and insufficient.
>
>4) Can current instrumentation answer these questions?
>
>No.
>
>5) List all proposed measurements and indicate the category of data collection for each measurement, using the [Firefox data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) found on the Mozilla wiki.
>
>**Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.**
>
><table>
>  <tr>
>    <td>Measurement Description</td>
>    <td>Data Collection Category</td>
>    <td>Tracking Bug #</td>
>  </tr>
>  <tr>
>    <td>Time in milliseconds from navigationStart to loadEventStart for the foreground http or https root content document.</td>
>    <td>2: Interaction Data</td>
>    <td>1671729</td>
>  </tr>
>  <tr>
>    <td>Time in milliseconds from responseStart to loadEventStart for the foreground http or https root content document.</td>
>    <td>2: Interaction Data</td>
>    <td>1671729</td>
>  </tr>
>  <tr>
>    <td>Time in milliseconds from navigationStart to domContentLoaded for the foreground http or https root content document.</td>
>    <td>2: Interaction Data</td>
>    <td>1671729</td>
>  </tr>
>  <tr>
>    <td>Time in milliseconds from responseStart to domContentLoaded for the foreground http or https root content document.</td>
>    <td>2: Interaction Data</td>
>    <td>1671729</td>
>  </tr>
>  <tr>
>    <td>The time between navigationStart and the first contentful paint of a foreground http or https root content document, in milliseconds. The contentful paint timestamp is taken during display list building and does not include rasterization or compositing of that paint.</td>
>    <td>2: Interaction Data</td>
>    <td>1671729</td>
>  </tr>
>  <tr>
>    <td>The time between responseStart and the first contentful paint of a foreground http or https root content document, in milliseconds. The contentful paint timestamp is taken during display list building and does not include rasterization or compositing of that paint.</td>
>    <td>2: Interaction Data</td>
>    <td>1671729</td>
>  </tr>
>  <tr>
>    <td>Time spent in milliseconds calling a request animation frame callback, collected every 100 calls.</td>
>    <td>2: Interaction Data</td>
>    <td>1671729</td>
>  </tr>
></table>
>
>6) Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.
>
>This collection is documented in its definitions files Histograms.json and metrics.yaml, and in the Probe Dictionary at https://probes.telemetry.mozilla.org.
>
>7) How long will this data be collected?  Choose one of the following:
>
>I, Denis Palmeiro ([e-mail]), want to permanently monitor this data.
>
>8) What populations will you measure?
>
>* Which release channels?
>All
>
>* Which countries?
>All
>
>* Which locales?
>All
>
>* Any other filters?  Please describe in detail below.
>None
>
>9) If this data collection is default on, what is the opt-out mechanism for users?
>Opt-out telemetry.
>
>10) Please provide a general description of how you will analyze this data.
>
>We will be monitoring this data as part of a performance dashboard, to be observed on a routine basis and monitored closely when an expected improvement is expected.
>
>11) Where do you intend to share the results of your analysis?
>
>On the telemetry dashboard.
>
>12) Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:
>
>No.

Comment 4

[Denis Palmeiro [:denispal]]

Comment on attachment 9182396 [details]
data-review-1671729.md

Edit:  Will re-attach a new data review.

Comment 5

[Denis Palmeiro [:denispal]]

Attachment: data-review-1671729.md

Comment 6

[Denis Palmeiro [:denispal]]

Attachment: data-review-1671729.md

Comment 7

[Chris H-C :chutten]

Comment on attachment 9182922 [details]
data-review-1671729.md

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Denis Palmeiro is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for pre-release channels only.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---

Result: datareview+

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/19a68bfd6c80
Gather telemetry on page load and request animation frame callback time r=bas,smaug,Dexter

Comment 9

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/19a68bfd6c80

Comment 10

[Denis Palmeiro [:denispal]]

Comment on attachment 9182397 [details]
Gather telemetry on page load and request animation frame callback time

Beta/Release Uplift Approval Request

• User impact if declined: Crash as in bug 1672503.
• Is this code covered by automated tests?: Unknown
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch fixes a crash in bug 1672503 that is also present in beta. Alongside, it also implements a few new probes we would like to monitor for a performance experiment once 83 enters release.
• String changes made/needed:

Comment 11

[Pascal Chevrel:pascalc]

Comment on attachment 9182397 [details]
Gather telemetry on page load and request animation frame callback time

Approved for 83 beta 5, thanks.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/a0e1e582dde2