1324436 - Active shield studies should show up as experiments in the environment data

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect, P3)

Description

[Georg Fritzsche [:gfritzsche]]

We already have `activeExperiment` in the environment data, which is used by Telemetry experiment.
We should probably have shield studies set this value.

Comment 1

[Georg Fritzsche [:gfritzsche]]

Michael, what do you think about this?

Comment 2

[Georg Fritzsche [:gfritzsche]]

`activeExperiment` and the environment data are documented here:
https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/data/environment.html

Comment 3

[Osmose [:osmose, :mkelly]]

Studies should totally have something specifying this, but I think currently we don't have anything stopping a user from being enrolled in multiple studies (except low enough sample rates that being in multiple studies is improbable). They'd also potentially conflict with telemetry experiments, although to be fair the root problem is that we have nothing stopping this, not the collision in the packet.

I think making studies able to set this, and the study utils automatically do so, would be a good thing to do, as long as separately we address the concurrent experiments problem. I suspect we might want to start using the Telemetry Experiment add-on type, or another special one.

ID can probably just be the add-on ID, and we have branch names. If we need more info than that, presumably we can do Analysis Magic (tm) to pull in their study pings?

glind probably has stronger and more-correct opinions than me.

Comment 4

[Georg Fritzsche [:gfritzsche]]

(In reply to Michael Kelly [:mkelly,:Osmose] from comment #3)
> Studies should totally have something specifying this, but I think currently
> we don't have anything stopping a user from being enrolled in multiple
> studies (except low enough sample rates that being in multiple studies is
> improbable). They'd also potentially conflict with telemetry experiments,
> although to be fair the root problem is that we have nothing stopping this,
> not the collision in the packet.

True, to avoid future confusion think we should make sure that only one of Telemetry experiments or Shield studies is active.

Is it a goal to have more than one shield study active at one time?

> ID can probably just be the add-on ID, and we have branch names. If we need
> more info than that, presumably we can do Analysis Magic (tm) to pull in
> their study pings?

I think for analysis it is best if we have:
* id be unique to the active study
* branch be the cohort / experiment branch (it is already used that way)

FWIW, i don't see any problems with changing the environment format here if needed to make analysis easier.

Comment 5

[Osmose [:osmose, :mkelly]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #4)
> Is it a goal to have more than one shield study active at one time?

I suspect not, but glind knows for sure.

> I think for analysis it is best if we have:
> * id be unique to the active study
> * branch be the cohort / experiment branch (it is already used that way)
> 
> FWIW, i don't see any problems with changing the environment format here if
> needed to make analysis easier.

Sounds good to me.

Comment 6

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

I don't think we should force one at atime, especially if what we discuss in 1337081 goes through.  Closing this one in favor of that.

Comment 7

[Georg Fritzsche [:gfritzsche]]

I think we still need this bug to track the work needed for SHIELD to integrate with this.
I'll make it depend on bug 1337081 instead.

Comment 8

[Georg Fritzsche [:gfritzsche]]

I think this happened?

Comment 9

[Osmose [:osmose, :mkelly]]

We started using TelemetryEnvironment.setExperimentActive and friends in https://github.com/mozilla/normandy/pull/674, which has landed, so yes, I think.