Investigate `mobile-event` pings for GV/WV experiment
Categories
(Data Science :: Investigation, task, P1)
Tracking
(Not tracked)
People
(Reporter: harter, Assigned: tdsmith)
References
Details
In Bug 1552570 we identified a bias in the enrollment for a recent mobile experiment. Specifically, the control branch has ~25% more enrollments than the treatment branch even for recent periods. We're suspicious that this enrollment issue is caused by users in treatment being unable to send their first core ping due to crashes.
We record a mobile event when trying to load GeckoView. The mobile-event ping is sent more frequently than the core ping, so there's a chance we're still getting the event ping. We should investigate whether we see an unusually high number of GeckoView load events given the number of enrollments we see on the treatment branch of this focus experiment.
|
Reporter | |
Comment 1•6 years ago
|
||
:colee - can you confirm the structure of the event being recorded when loading GeckoView? We discussed this yesterday but my notes are incomplete.
|
Assignee | |
Comment 3•6 years ago
|
||
Clearing ni for now; this is blocked behind identifying next steps for webcompat metrics.
The event on loading GeckoView for the first time looks like the following... Category: action. Method: change. Object: gecko_engine.
|
|
Reporter | |
Updated•6 years ago
|
I just figured out why our WebView and GeckoView experiment populations seemed off by so much. Months ago, a decision was made to avoid languages with multi-byte characters due to a known GeckoView bug handling them. Instead of figuring out all multi-byte locales and using a blacklist, we used a whitelist containing only the top eight, non-multi-byte locales. This filter was never reversed. This clearly explains the real reason why there is enrollment bias.
I'm removing that filter as I believe this bug was fixed long ago.
Here is a list of the 3-letter language codes which should have been included in the experiment. All other languages would give biased data as far as this experiment is concerned.
lang: eng, fra, por, spa, ita, rus, pol, nld
|
||
Comment 7•6 years ago
|
||
Does this mean that we can still use the data by only looking at a smaller set of locales where GeckoView would act similar to WebView?
|
|
||
Comment 8•6 years ago
|
||
If this is the only problem, then yes. We're going to do that investigation next week, finding out if this is the only problem and if so what the real results are. Cross your fingers!
|
|
Reporter | |
Comment 9•6 years ago
|
||
In Bug 1566630 we verified that most of the enrollment bias is due to the locale filtering Colin explains in comment 5. The remaining enrollment bias may be caused by this bug, but the effect is bounded to less than ~1.5%. I'm not going to pursue this further. You can review the full report here.
Description
•