1383759 - Include all experiments in error_aggregates

Status: RESOLVED FIXED

(Cloud Services :: Mission Control, enhancement, P1)

Description

[Frank Bertsch [:frank]]

experiment_id and experiment_branch should not be dimensions in error_aggregates. The problem is the following:

1. A single user can be part of multiple experiments - if we then multiplex a single user's ping against all dimensions (i.e. each experiment), then that user will be multi-counted when experiment_id is not filtered on. (e.g. if a Beta 55 user is in 3 experiments, and a query does not filter on experiments, that user will be represented 3 times in the aggregated data.)

2. The error_aggregates table attempts to correct this by selecting only the *first* experiment, but this isn't an appropriate solution since then any single experiment may (and probably will) be missing entries.

We will solve this by having a separate dataset that has experiment dimensions.

Comment 1

[Frank Bertsch [:frank]]

Harald - I see that you have some queries using `experiment_id` in `error_aggregates` [0], [1]. Do you have any issues with removing these columns from `error_aggregates`?

[0] https://sql.telemetry.mozilla.org/queries/5882/source#table
[1] https://sql.telemetry.mozilla.org/queries/5813/source#table

Comment 2

[Frank Bertsch [:frank]]

Benjamin - I want to check in with you on these details. Another option is we just move to having telemetry experiments, which a user can only ever be part of 1. Did you have any feedback on this? The concern we had is there might be confusion over which type of experiment is available in this dataset.

Comment 3

[:Harald Kirschner :digitarald]

Removed it from 1 query and archived the other one which was just a test.

Comment 4

[Benjamin Smedberg]

Frank, we're in a transient situation here so I don't have a straightforward answer:

1) telemetry experiments are currently still only using the old environment.addons.activeExperiment annotation, and are not yet using the new environement.experiments map. Bug 1349214 (unowned) is on file for TelEx will use the new annotation.
2) pref experiments are using the new environment.experiments map
3) we are currently running both kinds of experiments
4) I don't know everything that error_aggregates powers. Is that what drives crash-stats and mission control?

In the theoretical future where everything is using environment.experiments, I'd suggest the following solution: keep a map field in error_aggregates with *all* of the experiments, and aggregate on the identify of that field. So e.g. you'd aggregate separately each of the following configs:

* none
* exp A branch 1
* exp A branch 2
* exp B branch 1
* exp B branch 2
* exp A branch 1 + B branch 1
* exp A branch 2 + B branch 1

etc... but as we start scaling to potentially tens of experiments running simultaneously on a channel, this might get combinatorially complex.

Comment 5

[Frank Bertsch [:frank]]

This has changed a bit. Instead of removing it, we're now going to add all experiments for every client, and *every query must include an experiment_id*. If experiment_id = NULL, that just means the entire population (if none is given, clients will be double counted).

M-c will then require selection of an experiment.

Comment 6

[Frank Bertsch [:frank]]

Fix in https://github.com/mozilla/telemetry-streaming/commit/50dba3fd48b9a55e0218d7086e1dde26bd310878.