1113575 - Experiment setup might skew telemetry data

Status: RESOLVED WONTFIX

(Toolkit :: Telemetry, defect)

Description

[Roberto Agostino Vitillo (:rvitillo)]

While doing the analysis for Bug 1111791, I noticed that the first pings for the experimental branch had a skewed EVENTLOOP_UI_LAG_EXP_MS histogram compared to the control pings. The skew disappears when filtering out the first submission for each client, so my hypothesis is that the setup of the experimental branch can significantly alter the ping.

As Georg noted, disabling temporary Telemetry during setup wouldn't be easy. Nevertheless we could simply mark a client part of an experiment only for the sessions after the one that performs the setup.

Filtering on the server side is certainly possible, once you are aware of it though. Having this in the client would be safer.

Comment 1

[Georg Fritzsche [:gfritzsche]]

With temporarily disabling Telemetry i would be worried about the impact on other metrics as well failsafe recovery from disabling it.

If we need to do something here to fix those sessions, maybe just temporarily disabling that single probe would be better.

(In reply to Roberto Agostino Vitillo (:rvitillo) on PTO from 20/12-2/1 from comment #0)
> Nevertheless we could simply mark a client part of an experiment only for
> the sessions after the one that performs the setup.

We already have experiment activity in the telemetry ping - are you suggesting breaking up sessions at experiment activation or something like that?

Comment 2

[Roberto Agostino Vitillo (:rvitillo)]

Temporarily disabling that single probe isn't going to work as the setup code might affect other probes as well. What I am proposing is to not save in the ping information pertaining the experiment (e.g. activeExperiment*) until the next session, so that the first submission isn't considered as being part of the experiment.

Comment 3

[Benjamin Smedberg]

I don't think it's wise to wait until the next session. We may record information during experiment setup which will be important to analyze later knowing which experiment branch the user was in.

Comment 4

[Roberto Agostino Vitillo (:rvitillo)]

Fair enough, what about a boolean marker that is set only for the setup session?

Comment 5

[Alessio Placitelli [:Dexter]]

We're retiring experiments (see bug 1415284), closing this bug.