1633047 - Beta 76 pull-factor test-run experiment enrolled too many invalid users

Status: RESOLVED FIXED

(Firefox :: Messaging System, defect, P2)

Description

[Kate Hudson :k88hudson]

• Type of issue: Targeting expression result did not match intent.
• Result: Too many users were enrolled in the experiment that never saw about:welcome.
• Cause: We didn't include profile age filtering in the targeting expression. We identified this during review as a possible issue, but decided to go ahead to see if it would be ok without it since this was a beta dry run – and since we suspected there might be issues with accessing the ASRouter profileDateCreated from disk
• Impact to experiment: We will need to restart. No impact to the actual product goals here since this was a test run.

Comment 1

[Kate Hudson :k88hudson]

Other issues we noticed:

• Reading from disk to get profileAgeCreated is potentially problematic – but we're not sure
• Like with Normandy, writing the targeting expression for the first run case was not easy!

Comment 2

[Kate Hudson :k88hudson]

The problem with profileAgeCreated is that this involves reading from disk, which is slow for some users.

The failure state in the Normandy/Remote Settings filter expression environment:

In Normandy, if the reading profile age is too slow, the failure state will be that it returns undefined immediately. We can work around by adding a ! around the expression:

!(env.telemetry.main.environment.profile.creationDate < (\"2020-04-14\"|date / 1000 / 60 / 60 / 24))

The failure state in ASRouterTargeting filter expression environment:

In ASRouterTargeting, the failure state will be that the expression times out, which will mean the user will never get enrolled.

Comment 3

[Kate Hudson :k88hudson]

We've decided to go with the ASRouter approach for now to see what happens.

See https://github.com/mozilla/messaging-system-inflight-assets/pull/37 for the new version of the targeting expression.

Comment 4

[Su-Young Hong]

Wanted to add: I reviewed the data and the recipe specs as well, and have a number of recommendations for recipes

Learnings from Beta Smoke Test: AW_PULL_FACTOR_PRIVACY_1

The main points are:

• Include PCD in new user targeting (as noted above)
• We need to include explicit channel targeting
• Use randomly generated constant in sampling function (currently using experiment slug)
• Include convention in experiment slug to ensure uniqueness

Comment 5

[Kate Hudson :k88hudson]

After seeing low enrollment (and deciding this is likelt due to the ASRouter creation date implementation timing out), we are switching this to use the Remote Settings environment (env.telemetry.main.environment.profile.creationDate as well as adding channel (env.beta)

Comment 6

[Punam Dahiya [:pdahiya]]

Updated targeting expression https://github.com/mozilla/messaging-system-inflight-assets/pull/42

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:tspurway, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 8

[Punam Dahiya [:pdahiya]]

With fix of bug 1642455 in Fx 78, first run onboarding experiments enroll user with below filters in targeting , that has stabilised new user enrollment
a) isFirstStartup - Enrolled user has seen seen FirstStartup during FirstRun.

b) check for 'trailhead.firstrun.didSeeAboutWelcome' pref as false enrolling only new users that hasn't seen welcome UI

With above fixes its safe to close this issue as resolved. Thanks!