Clients repeatedly failed to unenroll from Comcast steering experiment
Categories
(Firefox :: Normandy Client, defect, P3)
Tracking
()
People
(Reporter: tdsmith, Unassigned)
References
Details
Many clients experienced failures attempting to unenroll from the Comcast steering experiment.
Grafana shows a high number of events: https://grafana.telemetry.mozilla.org/d/XspgvdxZz/experiment-enrollment?panelId=22&fullscreen&orgId=1&var-experiment_id=bug-1647305-pref-comcast-steering-experiment-release-78-80&from=1597017600000&to=
Deduping by client shows that clients are each sending many events, all for reason already-unenrolled.
The branches were affected more or less evenly; about 31k users per branch (~4% of clients) were affected: https://sql.telemetry.mozilla.org/queries/74591/source
The filter expression is straightforward, and is notable only for a geolocation clause:
"filter_expression": "(normandy.channel in [\"release\"]) && (normandy.country in [\"US\"]) && ([normandy.userId,\"doh-comcast-rollout\"]|bucketSample(8999,1000,10000)) && ((normandy.version>=\"78\"&&normandy.version<\"79\")||(normandy.version>=\"79\"&&normandy.version<\"80\")||(normandy.version>=\"80\"&&normandy.version<\"81\")||(normandy.version>=\"81\"&&normandy.version<\"82\")||(normandy.version>=\"82\"&&normandy.version<\"83\")||(normandy.version>=\"83\"&&normandy.version<\"84\")||(normandy.version>=\"84\"&&normandy.version<\"85\")||(normandy.version>=\"85\"&&normandy.version<\"86\"))",
The original unenrollments for affected users were overwhelmingly for reason filter-mismatch. https://sql.telemetry.mozilla.org/queries/74591/source
We did not expect clients to unenroll but it's not surprising that some users would have inconsistent geolocation results over time.
The unenrollments began as soon as the experiment launched. There was not a spike associated with the Fx 80 launch, unlike bug 1663971.
|
Reporter | |
Comment 1•4 years ago
|
||
The PDF retention experiment was also notable for generating a large number of unenrollFailed events -- these seemed to be associated with the Fx 80 launch and the recipe didn't target Fx 80, so there was a good reason for clients to unenroll but not a good reason for unenrollments to fail. Unenrollment at the end of the experiment apparently succeeded since the volume of messages dropped. cf https://sql.telemetry.mozilla.org/queries/75911/source#189359
|
||
Comment 2•2 years ago
|
||
Normandy is being deprecated.
Description
•