Cryptomining pref-flip study
Categories
(Data Science :: Experiment Collaboration, task)
Tracking
(Not tracked)
People
(Reporter: arthur, Assigned: wbeard, NeedInfo)
Details
Attachments
(3 files, 7 obsolete files)
Brief Description of the request (required):
We want to investigate the effect of cryptomining blocking on retention.
Business purpose for this request (required):
The cryptomining lists may cause breakage; we need to check this breakage isn't severe enough to cause problems for our users.
Requested timelines for the request or how this fits into roadmaps or critical decisions (required):
We would like to run this study during the 67 Release.
Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):
A link to the experimenter entry will be posted here when it's ready.
Name of Data Scientist (If Applicable):
Requesting flawrence.
|
||
Updated•7 years ago
|
|
Reporter | |
Comment 1•7 years ago
|
||
|
||
Comment 2•7 years ago
|
||
Hi just checking in to see if we would be able to secure Felix's time to support this study? He has been helping us with other ETP studies. Thank you!
|
|
||
Comment 3•7 years ago
|
||
(In reply to Tony Cinotto [:tcinotto] (UTC-8) from comment #2)
Hi just checking in to see if we would be able to secure Felix's time to support this study? He has been helping us with other ETP studies. Thank you!
Hi Tony,
Chris is going to be heading up this request (you will see him as the assignee). If needed he can certainly bring in Felix to consult while completing the work. But I would make sure to contact Chris with any questions or concerns you have on this specific request.
|
||
Comment 5•7 years ago
|
||
|
|
||
Updated•7 years ago
|
|
|
||
Comment 7•7 years ago
|
||
|
|
||
Updated•7 years ago
|
|
|
Reporter | |
Comment 8•7 years ago
|
||
Repurposing this ticket for cryptomining only. We will do a fingerprinting study separately.
|
||
Comment 9•7 years ago
|
||
this is for the CM-only study, for signing please.
|
|
||
Comment 10•7 years ago
|
||
|
||
Comment 11•7 years ago
|
||
(In reply to Arthur Edelstein [:arthur] from comment #1)
The link was updated as
https://experimenter.services.mozilla.com/experiments/cryptomining-protections-retention-study/
|
|
||
Comment 12•7 years ago
|
||
Data Science Peer Review
What is the goal of the effort the experiment is supporting?
The goal is to push cryptomining-blocking functionality out into desktop Firefox by default.
Is an experiment a useful next step towards this goal?
Yes, this should let us see whether the functionality would do a large amount of harm to users e.g. by breaking websites.
What is the hypothesis or research question? Are the consequences for the top-level goal clear if the hypothesis is confirmed or rejected?
Yes. The theory is that a meaningful amount of website breakage or otherwise undesired consequences would churn a detectable number of users. The hypothesis is no churn or change to engagement metrics is observed at 2 weeks. If churn is observed then presumably this would be addressed before shipping the feature.
Which measurements will be taken, and how do they support the hypothesis and goal? Are these measurements available in the targeted release channels? Has there been data steward review of the collection?
This will use whether a ping is received, active hours, subsession hours, and total uri count. They are available and no new collection is required.
Is the experiment design supported by an analysis plan? Is it adequate to answer the experimental questions?
Chris listed the specific effects he's looking for and the analysis should follow easily from that. They are a decent fit for the experimental questions: not super reliable but the best we have and they have detected large changes in the past.
Is the requested sample size supported by a power analysis that includes the core product metrics?
The power analysis includes the above stated metrics - it doesn't include "searches"; I forget whether searches is considered a core metric. I would be surprised if it required a notably larger sample size than the other metrics
If the experiment is deployed to channels other than release, is it acceptable that the results will not be representative of the release population?
N/A - it's for release
|
|
||
Comment 13•7 years ago
|
||
V1.1 for signing please, this one does not uninstall on disable.
|
|
||
Comment 14•7 years ago
|
||
signature for this one please
|
|
||
Comment 15•7 years ago
|
||
(In reply to Erica Wright [:ewright] from comment #14)
Created attachment 9064471 [details]
cryptomining_protections_retention_study-1.1.zipsignature for this one please
Hi Michael, just when you get a chance I was going to check to see if you could sign this one please. Thank you.
|
|
||
Comment 16•7 years ago
|
||
|
|
||
Updated•7 years ago
|
|
||
Comment 17•7 years ago
|
||
We have finished testing the Anti -Tracking: Cryptomining Retention experiment on Firefox 67.0b19 unbranded build.
QA’s recommendation: GREEN
Reasoning:
- No issues have been found during testing.
Testing Summary:
- Test Suite: Test Rail;
- Verified that the created recipe for the experiment on the Normandy stage server works as expected in the browser;
- Verified that the add-on doesn’t uninstall itself when it is disabled;
Tested Platforms:
- Windows 10 x64;
- macOS 10.14;
- Arch Linux 4.14.3 x64;
Tested Firefox versions:
- Firefox 67.0b19 unbranded build (US);
- Firefox 67.0b19 unbranded build (DE);
- Firefox 67.0b19 unbranded build (FR);
|
|
||
Comment 18•7 years ago
|
||
|
|
Assignee | |
Comment 19•6 years ago
|
||
I shared this in #anti-tracking, but posting the study results here for posterity
https://docs.google.com/document/d/1qEGpYOdOP82-9Euq-GSM9zkrbuKroEZgpVm739o9cCo/edit?usp=sharing
Description
•