1369427 - figure out how to do a/b testing with updates

Status: RESOLVED WONTFIX

(Release Engineering Graveyard :: Applications: Balrog (backend), enhancement, P3)

Description

[[e-mail] (:bhearsum)]

We do a lot of AB testing these days, most notably with Funnelcakes and SHIELD. The former enables us to do AB testing of anything in the product (but only with new installs), while the latter lets us do AB testing of anything that can be delivered as a SHIELD addon. This means that we're still unable to do AB testing of things that can't be delivered as addons on the vast majority of our user base. Adding the ability to AB test to Balrog would fix this.

There's going to be a few things that make this difficult:
* We have no way to identify a single install across multiple requests. This means that when a user checks in one day we could offer them something we're testing, and then when they check in the next day it could disappear.
* There's no ability to offer something to X users, or X percent of users. We may have to build this counting into Balrog, or rely on Telemetry to halt updates when we hit our goals.

Depending on the types of groups we want to test on, we also may not have the right information in the update ping to do that. Eg: we have very little hardware information in the ping (notably absent: graphics card information).

SHIELD has done a lot of work on this, so we should look at what they've done and see if we can either use it directly or draw some inspiration. https://mozilla.github.io/shield-studies-docs/ may be useful reading.

Comment 1

[Chris AtLee [:catlee]]

I wondered if we could use SHIELD to modify the update URL to indicate which cohort the user should be in?

Comment 2

[[e-mail] (:bhearsum)]

This hasn't come up in a long time, and Normandy (nee SHIELD) has other mechanisms to do this as far as I know.