1601993 - Collect Telemetry on the slow startup notification bar

Status: RESOLVED FIXED

(Firefox :: General, task, P3)

Description

[Mike Conley (:mconley) (:⚙️)]

Firefox has a way of detecting slow startup times for profiles more than 90 days old. When it detects a slow startup, it shows a notification bar telling the user that Firefox seems slow to start and that they might want to refresh their profile (linking them to https://support.mozilla.org/en-US/kb/reset-firefox-easily-fix-most-problems).

According to our stats, that page has been our #6 most visited SUMO article over the last several releases (after redirecting to https://support.mozilla.org/en-US/kb/refresh-firefox-reset-add-ons-and-settings).

While we have Telemetry on startup times, we don't have Telemetry in how users interact with slow startup time. This notification bar is really the only piece of UI that tries to empower the user with some ability to improve startup time. Knowing how often it appears and how often users interact with it could help illuminate how much pain our users are having out there with startup time.

The other possibility is that the notification bar is displaying at the wrong times, or too often. That's definitely worth fixing too.

Things I'd want to collect are:

1. Is the user profile configured to display the slow startup notification bar?
2. Has the notification bar been displayed in the current session?
3. What was the user's interaction with the notification bar, if anything? Did they click through to the SUMO page, or did they dismiss it?

Comment 1

[Mike Conley (:mconley) (:⚙️)]

It might be good to collect the number of slow startup samples, as well as the average startup time, when the notification bar is displayed (and maybe even when it's not displayed...)

Comment 2

[:Gijs (he/him)]

Fx::General p3 while this waits for an assignee. :-)

Comment 3

[:Gijs (he/him)]

I assume that we kind of want this from a perf perspective...

Comment 4

[Mike Conley (:mconley) (:⚙️)]

I'm going to add some other items to the things I want to collect in comment 0:

1. The value of the browser.slowStartup.averageTime preference, regardless of whether or not the notification is shown.

Comment 5

[Mike Conley (:mconley) (:⚙️)]

Hey chutten, what's the state of the art on these kinds of measurements these days? I presume I'm still using Telemetry here... since these values are only expected to change once per session, can I presume I want scalars for each of these?

Comment 6

[Chris H-C :chutten]

Yup, we're still in Telemetry which means you are correct that Scalars would be the right choice.

(( do recall the caveat that Scalars are cleared once they're sent, so you'll find them in exactly one of a session's pings, not all of them. ))

Comment 7

[Mike Conley (:mconley) (:⚙️)]

Hm... so one of the things we might want to record is the computed average time that the slow startup notification is using to decide whether or not to display itself. According to bug 1614861, it's sometimes possible that this value is negative. I suspect we cannot record negative values via scalars... what is the preferred way to record a negative value? Also, for recording what action the user took on the notification bar (for example, did they click on the "Refresh Firefox" button, or the "Don't show me this again" button, or the "x" button on the notification bar, or none of the above), is it better to use Events?

Comment 8

[Chris H-C :chutten]

(In reply to Mike Conley (:mconley) (:⚙️) from comment #7)

Hm... so one of the things we might want to record is the computed average time that the slow startup notification is using to decide whether or not to display itself. According to bug 1614861, it's sometimes possible that this value is negative. I suspect we cannot record negative values via scalars... what is the preferred way to record a negative value?

Correct, you can't record negative values in your standard Telemetry types, which leaves us with varying levels of hackery to work around the limitation. You could use a string scalar and encode your number as some sequence of utf8 characters (by calling toString on it, perhaps), a bool scalar that denotes the sign of the measurement (e.g. slow_startup_notification_duration_negative: true/false), an adjustment of encoding so that the true value has UINT_MAX >> 1 added to it when recording (the same amount subtracted during analysis)... each has their drawbacks, and each may hit your personal "ugh, too much of a hack" threshold in different ways.

Also, for recording what action the user took on the notification bar (for example, did they click on the "Refresh Firefox" button, or the "Don't show me this again" button, or the "x" button on the notification bar, or none of the above), is it better to use Events?

Events are best for when you need to know the order things happen. If you don't care about the order you could use some label of the button as a String Scalar (maybe slow.startup.button_pressed: refresh/ignore/dismiss). Collection of exactly one sample of categorical (ie, non-continuous) data is another weak point of Firefox Telemetry, unless it fits into a bool.

So, uh, congratulations for finding two weak points in quick succession : |

Comment 9

[Mike Conley (:mconley) (:⚙️)]

Attachment: Bug 1601993 - Add Telemetry for the slow startup notification bar. r?dthayer, data-review=mmccorquodale

Comment 10

[Mike Conley (:mconley) (:⚙️)]

Attachment: Data review form

Comment 11

[Megan McCorquodale]

Comment on attachment 9129620 [details]
Data review form

1.    Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way? 
Yes, this will be documented in the probe dictionary. 

2.    Is there a control mechanism that allows the user to turn the data collection on and off?
Yes, by opting out of telemetry. 

3.    If the request is for permanent data collection, is there someone who will monitor the data over time?
Not permanent data collection. 

4.    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
Category 2, Interaction Data. 

5.    Is the data collection request for default-on or default-off?
Default on. 

6.    Does the instrumentation include the addition of any new identifiers?
No new identifiers. 

7.    Is the data collection covered by the existing Firefox privacy notice? 
Yes. 

8.    Does there need to be a check-in in the future to determine whether to renew the data? 
Yes, check in at version 78. 

9.    Does the data collection use a third-party collection tool? 
No. 

-----
data-review +

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/03e8a9366a2f
Add Telemetry for the slow startup notification bar. r=jaws, data-review=mmccorquodale

Comment 13

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/03e8a9366a2f