Closed Bug 1450090 Opened 4 years ago Closed 4 years ago

Add study branch to Normandy Client unenroll telemetry event


(Firefox :: Normandy Client, enhancement, P1)




Firefox 62
Tracking Status
firefox61 --- verified
firefox62 --- verified


(Reporter: jlockhart, Assigned: mythmon, Mentored)


(Keywords: good-first-bug)


(1 file)

The current Normandy Client enroll telemetry event schema includes the study branch, seen here:

However the 'branch' is not included in the unenroll telemetry event:

This makes it difficult to track unenrollment on a per branch basis, which can occur under certain conditions.  We should add branch to the unenroll event to make it easier to track and assess study health.
Mentor: mcooper
Keywords: good-first-bug
Priority: -- → P3
This issue is the last thing blocking us being able to put real-time per-branch enrollment numbers on our experiment dashboards... any chance we can get this bumped up in priority?
Flags: needinfo?(mgrimes)
Getting our in-flight dashboards up is definitely a P1 as this impacts not only my team, but all of our stakeholders as well. Mythmon - What's the LOE for this and what would we be trading off to make this happen?
Flags: needinfo?(mgrimes)
This is something that should be pretty easy to fix. It is mostly an oversight in the spec of what should be in the events that was reviewed here: Assuming we have this data at unenrollment time (I think we do), it should just be a matter of adding it to the schema and including the data in the event.

As far as getting this bumped in priority: preference rollout still dominates the engineering time on Normandy, but I'll make this a P2 (will work on next)
Priority: P3 → P2
Bump... this is still blocking us in a big way, wondering if we've got a timeframe for getting this implemented?
I know we've (mostly) wrapped work on preference rollout. Mythmon - Do you now have time to pick this up? Expanding our visibility into the Shield ecosystem is extremely important.
Assignee: nobody → mcooper
Priority: P2 → P1
Thanks for the bump. Now that there is some extra time, this was pretty straight forward.
Comment on attachment 8976262 [details]
Bug 1450090 - Add study branch to preference study unenroll telemetry event
Attachment #8976262 - Flags: review?(gijskruitbosch+bugs) → review+
Pushed by
Add study branch to preference study unenroll telemetry event r=Gijs
Comment on attachment 8976262 [details]
Bug 1450090 - Add study branch to preference study unenroll telemetry event

Approval Request Comment
[Feature/Bug causing the regression]: N/A

[User impact if declined]: 
"This change will allow more detailed monitoring of Shield pref flip study health, which helps prevent malformed studies from running for long periods of time unnoticed.  This can help us catch problems sooner so users aren’t being rapidly enrolled and unenrolled from studies, which can appear confusing to them." -jkerim

[Is this code covered by automated tests?]: Yes
[Has the fix been verified in Nightly?]: No
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: Adds existing data to a more convenient place, without affecting behavior.
[String changes made/needed]: None
Attachment #8976262 - Flags: approval-mozilla-beta?
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 62
Comment on attachment 8976262 [details]
Bug 1450090 - Add study branch to preference study unenroll telemetry event

Helps us better monitor the health of SHIELD studies and avoid potentially confusing users if they end up in a bad state. Approved for 61.0b7.
Attachment #8976262 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Flags: qe-verify+
Verified as fixed on: Windows 8 x64, the branch is now displayed in the telemetry events extras.
- 61.0b7 20180521110957
- 62.0a1 20180521220045 

toolkit.telemetry.cachedClientID = ed26014c-77f4-43f2-a655-e8869ea4334f
11379 	normandy 	enroll 	preference_study 	branches_experiment 	{"experimentType": "exp", "branch": "4"}
307098 	normandy 	unenroll 	preference_study 	branches_experiment 	{"didResetValue": "false", "branch": "4", "reason": "user-preference-changed"} 

toolkit.telemetry.cachedClientID = 22bf489d-16f0-4128-98f7-c90d7b06dce2
1521 	normandy 	unenroll 	preference_study 	branches_experiment 	{"didResetValue": "false", "branch": "1", "reason": "user-preference-changed-sideload"}

general opt-out toolkit.telemetry.cachedClientID = 88f32802-cd87-456b-afe7-e0897d5a0a35
4625 	normandy 	unenroll 	preference_study 	branches_experiment 	{"didResetValue": "true", "branch": "2", "reason": "recipe-not-seen"} 

I will check Redash in 1-2days in order to make sure everything actually gets displayed properly. Adding NI for myself for that.
Flags: needinfo?(adrian.florinescu)
Telemetry Redash data looks good as well. The study branch is now listed in the telemetry server data, hence marking this issue as verified.

Mike, I see two columns in Redash that are not populated with data: active_experiment_id &	
active_experiment_branch. Expected or a possible bug?
Flags: qe-verify+
Flags: needinfo?(mcooper)
Flags: needinfo?(adrian.florinescu)
I'm not sure what those columns are. If you have the particular query you're looking at, I could check it. I would guess however that users that have just unenrolled from an experiment would probably have no data for active_experiment_* columns.
Flags: needinfo?(mcooper)
I don't remember seeing those columns populated anywhere. I was looking at the comment 14 client id's, which have enroll/unenroll rollout events:

Let me know if you'd like to have a separate bug if you think that those two columns should have any data and we should track it.
Flags: needinfo?(mcooper)
Looking through that query, all the data I expect to see is represented in some column. I haven't done anything with those columns before, and I can't find any reference to them in mozilla-central. I think it is fine that they are blank.
Flags: needinfo?(mcooper)
You need to log in before you can comment on or make changes to this bug.