1472367 - SAVANT study new user enrollment extremely low

Status: RESOLVED FIXED

(Shield :: Shield Study, defect, P1)

Description

[Su-Young Hong]

Attachment: new_client_approximations_ms.png

We went live with the SAVANT pref flip study on June 26th.

There are 4 cohorts in the study

* US, existing users
* DE, existing users
* US, new users
* DE, new users

Enrollment in the existing users groups seem fine (7-10k a day), however, enrollment in the new user group is extremely low: 200-300 users a day. 

Source: https://sql.telemetry.mozilla.org/queries/56205/source#147284

By the current normandy settings, we should be enrolling 50% of new users who are:
    - country and locale in US or DE
    - starting on FF version is 61
    - channel is release
    - it is their first run

The enrollment numbers look way too low for those conditions. 

Quick and dirty approximations of the above conditions in main summary show about 10-18k new clients a day in DE and 18-37k new clients a day in US (see screenshot) 


Resources: 
SAVANT study tracking bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1457226#c3
Monitoring dashboard: https://sql.telemetry.mozilla.org/dashboard/savant-data-monitoring?fullscreen
Analysis repo: https://github.com/mozilla/SAVANT-study-analysis (currently private, just ask for access!)

Comment 1

[Chris More [:cmore]]

Thanks for filing the bug, Su.

What is the condition of the "it is their first run"? Is it within a specific amount of time after the new user ping / profile creation? Is the speed at which Shield gets a recipe and executes impacted here? Do we have any conditions on first run that would exclude a user from the new user cohort?

Comment 2

[Michael Cooper [:mythmon]]

Chris: "is their first run" is defined by the first time that Normandy has run on a given profile. It stores this data in a preference, and tends to be very reliable.

We found a different problem in the filter expression related to this. The filter expression tried to use profile creation date as a backup way to make sure we didn't over sample. Unfortunately, it did so in a way that conflicted with isFirstRun, causing the whole expression to be false in almost all cases.

Rob updated the recipe to remove the always false case, and I've filed bug 1472781 to fix this sort of problem in the future.

I'll leave this bug open until we can confirm the increased enrollment rate.

Comment 3

[Chris More [:cmore]]

Makes sense on the conclusion of the targeting condition issue. Thanks for the update!

Comment 4

[Su-Young Hong]

Talked offline with [:rrayborn] and "there was a race condition that prevented enrollment for some "new" users."

The shield team fixed it yesterday, and we're seeing significant improvement in the enrollment rate for new users now: https://sql.telemetry.mozilla.org/queries/56205/source#147284. (2.5k for DE and 4.6k for US). 

This issue is fixed. Thanks guys! :)