1473652 - Federated Learning shield study

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Florian Hartmann]

- Basic description of experiment:
The current algorithm for ranking history search suggestions in the awesome bar is based on manually selected numbers for weighting  frequency and recency (“frecency”) to determine the relevance of a potential search result. These numbers were not selected in a data-driven way and there might be a configuration of them that leads to much more relevant search results. Our experiment involves an optimization process that selects a configuration based on what search results users actually select. This is done using Federated Learning, a distributed way of doing machine learning where the data stays completely locally. Instead of sharing their data with Mozilla, the data is only used locally to improve the machine learning model. Periodically, users send updates of the model back to the server to improve the model for everyone. These updates are very high-level (e.g. "weight frequency more, weight recency less") and do not contain personal information.

- What are the branches of the study?
Treatment, control, control without decay

- What percentage of users do you want in each branch?
1/3, 1/3, 1/3

- What Channels and locales do you intend to ship to?
Beta channel, all locales

- What is your intended go live date and how long will the study run?
Launch on 7/16, run for 2 weeks

- Are there specific criteria for participants?
Must be on Firefox 62+

- What is the main effect you are looking for and what data will you use to make these decisions?
1. Baseline awesome bar product metrics do not decrease
2. Federated learning system is successfully exercised
3. Stretch: resultant model improves history search results (any of these three should improve, not necessarily all)
We will still decide on exact goals for these metrics.

- Who is the owner of the data analysis for this study?
Florian Hartmann (fhartmann@)

- Who will have access to the data?
Florian Hartmann (fhartmann@), Sunah Suh (ssuh@), Context Graph team

- Do you plan on surveying users at the end of the study?
Yes

- User facing title of the experiment:
Awesome Bar improved history search

- User facing description of the experiment:
The current Awesome Bar suggests history entries based on a number of factors. 
This experiment tries to weight the factors in a way that works better for the searches that users actually perform

- Code Review performed by:
Drew Willcoxon (Client-side), Sunah Suh (Server-side, still to do)

PHD: https://docs.google.com/document/d/1DuZ1nQ2ve7k-98BKUKCZegtVzAxnJBEgYJYv6IUqZck/edit?usp=sharing

Comment 1

[Carmen Fat [:cfat] - Ecosystem QA]

Federated Learning for the Awesome Bar Sign Off
Targeted: Firefox Beta 62.0bx

We have finished testing the Federated Learning for the Awesome Bar experiment.

We have found the following issues which we consider blockers for the experiment:

- #4 (https://goo.gl/49ay6a) and #5 (https://goo.gl/MB3sSc) - The experiment can be re-enabled and re-installed, which is a wrong behavior since this is a Shield Study;

- #8 (https://goo.gl/Cr4onc) - The add-on doesn't expire after the settled period of time for being live ends;

- The survey is not added to the add-on yet.


QA’s recommendation: RED - DON’T SHIP

Reasoning:

- As per previous discussions about this behavior, shield studies were not designed to be disabled and then re-enabled. That would cause weird issues with the data, so once you've opted out of a particular study you cannot re-enroll. Since this behavior is still in progress of new discussions and no final decision was made, we're treating these issues same as on previous shield studies.

- In its current state, the experiment remains installed forever. A workaround was discussed in comment 4 from the GitHub repo(https://goo.gl/8R69Yd), but this method implies a few risks.

- Also, the survey was not tested due to the fact it was not available, so we can't guarantee that there are no issues.


Testing Summary:
- Full Functional test suite: TestRail (https://goo.gl/fJqLnK)
- Verified that the Telemetry pings are correctly sent.

Tested Platforms:
- Windows 10 x64
- Ubuntu 16.04 x64
- macOS 10.13

Tested Firefox versions:
- Firefox Beta 62.0b6
- Firefox Beta 62.0b7