1483477 - Consider suppressing Environment changes when the values in the Environment don't change

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect, P1)

Description

[Chris H-C :chutten]

There are theoretical situations where the code in TelemetryEnvironment will notify listeners that the Environment has changed when the data in the old and new environment match.

For instance, if the user changes their homepage url from mozilla.org to firefox.com it triggers TelemetryEnvironment's pref listener which then transmits a pref-changed environment change even though the userPref for "browser.startup.homepage" didn't change (it went from "<user-set>" to "<user-set"). There are five such prefs.

It might be worth looking in to how many "environment-change" pings have identical Environment values to see if it is worth reducing the number of pings sent with that reason.

Comment 1

[Chris H-C :chutten]

Discuss

Comment 2

[Alessio Placitelli [:Dexter]]

(In reply to Chris H-C :chutten from comment #0)
> For instance, if the user changes their homepage url from mozilla.org to
> firefox.com it triggers TelemetryEnvironment's pref listener which then
> transmits a pref-changed environment change even though the userPref for
> "browser.startup.homepage" didn't change (it went from "<user-set>" to
> "<user-set"). There are five such prefs.

I think this might be a legit behaviour, as there is a change in the environment even if we cannot report *which* thing changed, due to privacy.
Now, the question is: do we care about reporting *which* pref changed?

> It might be worth looking in to how many "environment-change" pings have
> identical Environment values to see if it is worth reducing the number of
> pings sent with that reason.

good point!

Comment 3

[Georg Fritzsche [:gfritzsche]]

I wonder if there is any need for sending the "main" / "environment-change" ping at all.

We should probably look at whether removing it has impact on:
- data jobs
- data analysis / data scientists
- data latency

Comment 4

[Jan-Erik Rediger [:janerik]]

If there is no additional info carried in the generated ping then I don't see a good reason to send it.
If we can verify the number of those low-information pings in existing data, then let's do that!

Comment 5

[Georg Fritzsche [:gfritzsche]]

The next step required is analysis on frequency of environment-changes and effects on latency.

Comment 6

[Georg Fritzsche [:gfritzsche]]

From calling this out in the monday meeting, for potentially removing "environment-change" it seems that further discussion is required with at least the data platform team.
A point brought up is "every ping should have the exact environment it works in".

Removing the "environment-change" pings where no sent data actually changed is more actionable, so we should probably focus this bug on that part.

Comment 7

[Chris H-C :chutten]

Attachment: Bug 1483477 - Suppress Telemetry Environment changes if the environment didn't actually change

Comment 8

[Chris H-C :chutten]

Attachment: Bug 1483477 - Count how often Telemetry Environment changes aren't r?janerik!

I'm not sure of the ramifications of suppressing them immediately, so let's
just count them for now.

Comment 9

[Chris H-C :chutten]

Attachment: data collection review

Comment 10

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9182997 [details]
data collection review

We discussed in Matrix that querying this from existing telemetry is probably tractable, but data-review+ for the collection while we're chatting:

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in the probe definition files and the Probe Dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

n/a

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, technical data.

5. Is the data collection request for default-on or default-off?

Default-on.

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No

7. Is the data collection covered by the existing Firefox privacy notice?

Yep

8. Does there need to be a check-in in the future to determine whether to renew the data?

It's up to :chutten!

9. Does the data collection use a third-party collection tool?

No.

Comment 11

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

The query we were discussing is https://sql.telemetry.mozilla.org/queries/75736/source, which compares environments between main pings sent with reason environment-change and the main ping for the subsequent subsession.

In a sample of a million environment-change pings, we can identify the subsequent ping for about 95% of them. About 75k (i.e. 7.5%) of the subsequent pings have the same environment as the environment-change ping.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2ea4f55225f2
Count how often Telemetry Environment changes aren't r=janerik

Comment 13

[Chris H-C :chutten]

7.5% is worth a closer look. I'm pushing this measurement to make analysis easier (for instance: if we do decide to suppress these based on the criteria used on the client, what does that do to DAU?) and let's see where that takes us.

Comment 14

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/2ea4f55225f2