1551965 - Gather page reload stats from a user experience perspective

Status: RESOLVED FIXED

(Core :: Performance, defect)

Description

[Sean Feng [:sefeng]]

Attachment: data_collection_review_reload_ux.txt

We'd like to collect data about how users reload a page.

Comment 1

[Benjamin De Kosnik [:bdekoz]]

Comment on attachment 9065126 [details]
data_collection_review_reload_ux.txt

You are just checking function key and toolbar button, or all the combinations?

Comment 2

[Sean Feng [:sefeng]]

Instead of key combinations, how about we collect things like,
NORMAL_F5, SKIP_CACHE_F5, NORMAL_RELOAD_BUTTON, SKIP_CACHE_RELOAD_BUTTON, DUPLICATE_TAB_RELOAD_BUTTON

Comment 3

[Corey Dow-Hygelund [:ccd]]

From a data analysis perspective, either method works fine as they contain the same information: a single probe keyed scalar, or a suite of scalar probes. The two issues I see are the telemetry resource demand of either method, and discovery/documentation of probe(s).

It might be a smaller fingerprint to send four integers probes, than a keyed scalar. I defer to Benjamin for his expertise on this point.

Breaking into individual probes will require good documentation, as it will require analysts to be informed of this probe "suite" or have to rummage through the probe dictionary to "discover" these probes and their relationship. The latter is definitely a large productivity hit for the analyst. In some sense, a keyed scalar is self-documenting, so it is easier to immediately utilize.

Comment 4

[Benjamin De Kosnik [:bdekoz]]

This seems like a resonable set of starting combinations. Also in favor of the keyed approach.

Comment 5

[Benjamin De Kosnik [:bdekoz]]

Sorry if my previous comment was confusing: I'm suggesting a (categorical or enumerated) histogram. If categorical, then proposed labels might be (trying to stay away from "normal", since that involves an individual's own perception and describing the keys pressed only)

"labels": [
  "f5",
  "ctrl_f5",
  "toolbar",
  "ctrl_toolbar",
  "duplicate_tab"
],

I cannot remember off hand, but I think you'd mentioned yet more combinations. Sound good?

Comment 6

[Benjamin De Kosnik [:bdekoz]]

Hey Sean, can you update the data review text with the current key combinations that will be tracked please?

Comment 7

[Sean Feng [:sefeng]]

Attachment: data_collection_review_reload_ux_v2.txt

Hi Ben, please check out the updated review form. Let me know if you have questions.

Comment 8

[Sean Feng [:sefeng]]

Just making sure it doesn't fall off Ben's radar.

Comment 9

[Benjamin De Kosnik [:bdekoz]]

Ok from my end

Comment 10

[Sean Feng [:sefeng]]

Attachment: Bug 1551965 - Add telemetry for page reload key combinations r=bdekoz

Comment 11

[Sean Feng [:sefeng]]

Ben, while I was working on the patch, I figured the correct key combinations should be
["only_f5", "ctrl_f5", "ctrl_r", "ctrl_shift_r", "toolbar", "shift_toolbar", "meta_toolbar", "auxiliary_toolbar"], do I need to update the review form to reflect the change?

Comment 12

[Sean Feng [:sefeng]]

Attachment: data_collection_review_reload_ux_v3.txt

Update the data review form to match the key combinations and tag data review team to review.

Comment 13

[Chris H-C :chutten]

Comment on attachment 9073314 [details]
data_collection_review_reload_ux_v3.txt

Preliminary note:

It is best practice to include regression tests for permanent data collections like these. You may find the histogram utility functions in [TelemetryTestUtils](https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/tests/utils/TelemetryTestUtils.jsm#194) useful, and I hear there are some folks on IRC#telemetry who'd be happy to help out.

---
DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions file [Histograms.json](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Histograms.json) and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Benjamin DeKosnik and Dragana Damjanovic are responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+

Comment 14

[Sean Feng [:sefeng]]

Ben Corey: Chutten pointed whether it needs to be collected forever, I think it's a good point because we probably can stop it once we have cleaned the common user behaviours. thoughts?

Comment 15

[Benjamin De Kosnik [:bdekoz]]

I think it makes sense to collect this data for 2 releases, and then re-evaluate what we have learned and re-assess as to 1) does the collected data change or consolidate code paths in the browser (modifying probes) and 2) how relevant is the original data collection after more is known about reload behaviors.

So, I would be in favor of not collecting forever, but just for the next couple of releases and re-assessing. If it makes sense at that decision point, then perhaps then is a good time to flip the forever switch, Sound good?

Comment 16

[Corey Dow-Hygelund [:ccd]]

I defer to Benjamin. The only suggestion I have is to collect the data for at least a few releases to base a decision of utility upon.

Comment 17

[Sean Feng [:sefeng]]

Let's do 2 releases!

Comment 18

[Vicky Chin [:vchin]]

Can this be closed off now?

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0ecac854be9c
Add telemetry for page reload key combinations r=bdekoz,Gijs,chutten

Comment 20

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/0ecac854be9c

Comment 21

[Sean Feng [:sefeng]]

vchin: Yep and it is closed!