1565118 - Check Current Active Enrollment In Old Pocket Experiment

Status: RESOLVED INACTIVE

(Data Science :: Investigation, task)

Description

[jonathan koren]

Brief Description of the request (required):

While investigating curious behavior in another experiment, we discovered Pocket was continuing to receive requests from an experiment that was terminated a year ago. (pref-flip-activity-stream-pocket-octr-bug-1476815) We engaged Shield and confirmed that the experiment was "off". (Thanks jkerim.)

While we understand that old experiments continue to have enrollments forever at a rate asymptotically close to zero, Pocket is seeing what may be an unusually large number of requests from this completed experiment.

For background, there's a CDN in front of the Pocket servers that caches requests. Unfortunately we don't have number of requests that hit the CDN by URL. Requests that are missed by the CDN (due to cache misses) hit the Pocket servers, and are logged by Pocket. Of these, 0.1% of the requests appear to be from this completed experiment. This translates to thousands of requests every day. Almost a year since the experiment was marked complete, we're still seeing on average 41 requests per hour.

While 0.1% isn't a lot, 41 requests an hour seems more than tens of profiles. We would like to know how many profiles are still enrolled in this experiment as of July 2019.

We should note that Pocket has observed some browser profiles that send an unusually large number of new tab impressions. (For example 45 impressions of a single article in a second according to Activity Stream telemetry.) While we don't understand what is happening here (perhaps it is some sort of Selenium script), it would be useful to know if any of the browsers that are still enrolled in this experiment are exhibiting unusual behavior.

Business purpose for this request (required):

We want to get a better understanding on what is happening with completed experiments.

Requested timelines for the request or how this fits into roadmaps or critical decisions (required):

While not business critical, this is part of a post-mortum investigation.

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

Old experimenter link: https://experimenter.services.mozilla.com/experiments/pref-flip-activity-stream-pocket-octr-bug-1476815/

Example dashboard / query Su-Young suggested: https://sql.telemetry.mozilla.org/dashboard/pktnewtabh12019-2-branch-confirmation-test-monitoring

https://grafana.telemetry.mozilla.org/d/3QA87kliz/experiment-enrollment?orgId=1&from=now-1y&to=now&var-experiment_id=pref-flip-activity-stream-pocket-octr-bug-1476815&var-branch_id=All

Name of Data Scientist (If Applicable):

Normally, Pocket works with Su-Young, but he is out until July 22, so we'd like someone to fill in.

Comment 1

[Su-Young Hong]

I'll be on PTO / leave until Oct 13th, so unassigning myself from this ticket. Can pick this back up when I'm back / have bandwidth, otherwise, feel free to re-assign.

Comment 2

[Jess Mokrzecki [:jmok]]

Work for the DS team is now tracked in Jira. You can search with the Data Science Jira project for the corresponding ticket.