1630384 - Add telemetry for users that see "FirstStartup"

Status: NEW

(Firefox :: Messaging System, enhancement, P3)

Description

[Kate Hudson :k88hudson]

In order to verify if enrolment actually happened during the first start-up phase, we'd like to include in the enrolment ping. We could do this by passing in a value to the onRecipe function (or _run in Normandy actions)

Comment 1

[Nan Jiang [:nanj]]

Looks like we already have some probes in FirstStartup.jsm.

It includes statusCode and elapsed time duration of the first startup, I think those are sufficient for us to understand how many users see the first startup.

Note that those probes will retire in Firefox 80.

Comment 2

[Su-Young Hong]

Nan, I don't think the existing probes are sufficient for what we need, since they're only reported in main pings.

We're going to need this in new profile pings (https://docs.telemetry.mozilla.org/concepts/choosing_a_dataset.html?highlight=new,profile#new-profile-ping), (and would be useful as well in experiment enrollment pings).

The experiments team will need this specifically because we need to know what the number of new profiles are that are on firstStartup to get the baseline population to determine if enrollment is correct.

It will also be useful for general analysis.

Unless there is significant cost to implementing this, we definitely should add implement in new profile pings (and possibly enrollment pings).

Comment 3

[Nan Jiang [:nanj]]

For sure, we can still add probes for this for better accuracy (compared to the existing ones).

The experiments team will need this specifically because we need to know what the number of new profiles are that are on firstStartup to get the baseline population to determine if enrollment is correct.

FYI, we are landing bug 1642455 to improve the enrollment for the first-run experiments. Particularly, this allows us to precisely enroll users only during the first startup other than relying on indirect filters such as profile age and pref value.

Comment 4

[Nan Jiang [:nanj]]

Reset the iteration and priority based on the triage meeting. We will revisit this feature for the Firefox 80 iteration.

Comment 5

[Kate Hudson :k88hudson]

Given that our targeting improvements worked well, let's deprioritize this.

Comment 6

[Punam Dahiya [:pdahiya]]

FirstStartup targeting is stable with fix of https://bugzilla.mozilla.org/show_bug.cgi?id=1642455. NI Nan to help triage and close this if this telemetry is no longer needed.

Comment 7

[Su-Young Hong]

Was there validation for stable FirstStartup targeting working as expected? I imagine this is where this telemetry would be useful.

Comment 8

[Nan Jiang [:nanj]]

Was there validation for stable FirstStartup targeting working as expected? I imagine this is where this telemetry would be useful.

Yes, we validated the FirstStartup targeting in the multi-stage v1/v2/v3 experiments, and the results showed that the user enrollments across branches matched against the first-run impressions with the associated screen id in the experiment.

Multi-stage v1: https://sql.telemetry.mozilla.org/dashboard/messaging-system-experiment-multi-stage-about-welcome-in-firefox-78
Multi-stage v2: https://sql.telemetry.mozilla.org/dashboard/messaging-system-experiment-multi-stage-about-welcome-v2-in-firefox-79
Multi-stage v3: https://analytics.amplitude.com/mozilla-corp/notebook/71nd852

Now that this targeting turns out to be effective, I'd recommend closing this for now.

Comment 9

[Su-Young Hong]

Nan, I'm not sure I'm following the validation logic here. I'm interpreting what your saying to mean: we enrolled people into some experiments, and they saw the treatment about:welcome screen they were supposed to.

But not all users enrolled into the experiment see the correct about:welcome message, or any about:welcome message. Currently I believe a non-negligible part of new user experiment populations still experience this (~10%).

The point of this telemetry is to give us clarity on why they're not seeing this, is it because they're not enrolled early enough? is it a race condition? is it something else?

So I don't think we've actually validated the FirstStartup targeting is working as intended (which this telemetry would accomplish).

Comment 10

[Nan Jiang [:nanj]]

But not all users enrolled into the experiment see the correct about:welcome message, or any about:welcome message. Currently I believe a non-negligible part of new user experiment populations still experience this (~10%).

Right, we just found that around 1-3% of treatment users ended up getting the default first-run experience in all the first-run experiments (multi-stage v1/v2/v3). Note that users in the control group never got the incorrect experience, so we don't believe this was caused by the incorrect enrollment during the first startup.

So I don't think we've actually validated the FirstStartup targeting is working as intended (which this telemetry would accomplish).

Given all the observations so far, we're inclined to believe the user enrollment (including the FirstStartup targeting) is working as expected. The issue(s) are more likely to take place after the experiment enrollment. Specifically, when Firefox tries to render the about:welcome based on the experiment data. We're suspecting that for a certain amount of users (i.e. slow clients), the experiment data (which requires disk io) could be unavailable, so it has to fall back to the default first-run, and that's what causes the above issue.

Punam and I had discussed the possible solutions as follows:

• Record a telemetry event when detecting the experiment data is unavailable and the fallback is happening
• For first-run experiment users, we could unenroll them if the expected experience can't be fulfilled during first-run

What do you think, Su?

Comment 11

[Nan Jiang [:nanj]]

A new telemetry probe called activation will be added to Nimbus in bug 1675104. I believe that should allow us to understand what exactly happens to the experiment users.

Comment 12

[Su-Young Hong]

Hey Nan,

That's interesting about the I/O error. Having more telemetry to detect those experiment data events will be super helpful, I think that's a great idea.

I don't think we should unenroll if the expected experience can't be fulfilled though, because:

• one, we'll be introducing sampling bias at that point, and the experiment results are no longer truly "random" (this breaks the experiment)
• two, if we have some probe that tells us the user got enrolled in the experiment but didn't correct experient, we could potentially correct for it.

It makes sense to measure, but not intervene.

But to the original point about firstStartup, so users seeing the correct about:welcome doesn't indicate that they're on firstStartup, which is the behavior we're trying to validate, right? Like all those users who were enrolled in V1/2/3 experiments, how do we know they're specifically firstStartup users, versus other new profiles (profile manager, etc.).

Confirming that everyone enrolled into those 3 experiments were new profiles that were on the firstStartup (and don't include new profiles that are on other new profile protocols) run, and that the enrollment happend during firstStartup (as opposed to the second checkin with Remote settings) is thing thing we want to know. And about:welcome can happen when either of conditions aren't true, so the presence of the correct about:welcome doesn't validate what we're trying to know.

Comment 13

[Nan Jiang [:nanj]]

(In reply to Su-Young Hong from comment #12)

I don't think we should unenroll if the expected experience can't be fulfilled though, because:

• one, we'll be introducing sampling bias at that point, and the experiment results are no longer truly "random" (this breaks the experiment)
• two, if we have some probe that tells us the user got enrolled in the experiment but didn't correct experient, we could potentially correct for it.

It makes sense to measure, but not intervene.

That's a good point, Su.

But to the original point about firstStartup, so users seeing the correct about:welcome doesn't indicate that they're on firstStartup, which is the behavior we're trying to validate, right? Like all those users who were enrolled in V1/2/3 experiments, how do we know they're specifically firstStartup users, versus other new profiles (profile manager, etc.).

Confirming that everyone enrolled into those 3 experiments were new profiles that were on the firstStartup (and don't include new profiles that are on other new profile protocols) run, and that the enrollment happend during firstStartup (as opposed to the second checkin with Remote settings) is thing thing we want to know. And about:welcome can happen when either of conditions aren't true, so the presence of the correct about:welcome doesn't validate what we're trying to know.

Wanted to clarify that the targeting isFirstStartup allows us to only enroll users to the first-run experiments during the first startup run. To my understanding, the first startup run only takes place when one installs the browser via the Firefox installer on Windows. Since this is the only way to trigger the first startup run, I think the first-run experiment enrollment ping is essentially equivalent to the telemetry of experiment users who see "FirstStartup". Does that make sense to you?

Comment 14

[Punam Dahiya [:pdahiya]]

(In reply to Nan Jiang [:nanj] from comment #10)

But not all users enrolled into the experiment see the correct about:welcome message, or any about:welcome message. Currently I believe a non-negligible part of new user experiment populations still experience this (~10%).

Right, we just found that around 1-3% of treatment users ended up getting the default first-run experience in all the first-run experiments (multi-stage v1/v2/v3). Note that users in the control group never got the incorrect experience, so we don't believe this was caused by the incorrect enrollment during the first startup.

Investigating deeper into 1-3% treatement users seeing default experience , it seems a browser restart right after install and loading about:welcome
seems to be the reason. During this subsequent restart, user stays enrolled but experiment data is not available for about:welcome resulting in fall back to default experience. Created bug https://bugzilla.mozilla.org/show_bug.cgi?id=1678516 to log this investigation and fix the issue