Experiment setup might skew telemetry data

RESOLVED WONTFIX

Status

()

Toolkit
Telemetry
RESOLVED WONTFIX
4 years ago
7 months ago

People

(Reporter: rvitillo, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

While doing the analysis for Bug 1111791, I noticed that the first pings for the experimental branch had a skewed EVENTLOOP_UI_LAG_EXP_MS histogram compared to the control pings. The skew disappears when filtering out the first submission for each client, so my hypothesis is that the setup of the experimental branch can significantly alter the ping.

As Georg noted, disabling temporary Telemetry during setup wouldn't be easy. Nevertheless we could simply mark a client part of an experiment only for the sessions after the one that performs the setup.

Filtering on the server side is certainly possible, once you are aware of it though. Having this in the client would be safer.
With temporarily disabling Telemetry i would be worried about the impact on other metrics as well failsafe recovery from disabling it.

If we need to do something here to fix those sessions, maybe just temporarily disabling that single probe would be better.

(In reply to Roberto Agostino Vitillo (:rvitillo) on PTO from 20/12-2/1 from comment #0)
> Nevertheless we could simply mark a client part of an experiment only for
> the sessions after the one that performs the setup.

We already have experiment activity in the telemetry ping - are you suggesting breaking up sessions at experiment activation or something like that?
(Reporter)

Comment 2

4 years ago
Temporarily disabling that single probe isn't going to work as the setup code might affect other probes as well. What I am proposing is to not save in the ping information pertaining the experiment (e.g. activeExperiment*) until the next session, so that the first submission isn't considered as being part of the experiment.

Comment 3

4 years ago
I don't think it's wise to wait until the next session. We may record information during experiment setup which will be important to analyze later knowing which experiment branch the user was in.
(Reporter)

Comment 4

4 years ago
Fair enough, what about a boolean marker that is set only for the setup session?
We're retiring experiments (see bug 1415284), closing this bug.
Status: NEW → RESOLVED
Last Resolved: 7 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.