1514336 - Separate, or disable, telemetry recorded during UI/integration testing on release builds

Status: RESOLVED WORKSFORME

(Data Platform and Tools :: Glean: SDK, enhancement, P3)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

When testing release candidate builds, applications ideally want to run UI/integration tests on the exact build they'll be releasing. Additionally, some projects test their release configuration every time a PR opens or code merges to master. However, this creates a problem: telemetry logged during testing on release builds will be added to the release telemetry dashboard, injecting unrealistic data. If this happens often enough (e.g. we run the release tests on multiple devices of various configurations multiple times a day and each time there's a new telemetry ID), our data will be skewed.

Ideally, Glean would provide an easy way to configure it to disable telemetry under these conditions and document this as recommended behavior. Presumably, we wouldn't want to modify the builds so this would likely be a start-up flag. The start-up flag should probably persist beyond this application's lifecycle (i.e. if the application is restarted during the test, we wouldn't want to have telemetry be re-enabled).

Comment 1

[Georg Fritzsche [:gfritzsche]]

We see two possible approaches here:
1) Provide a way to (persistently) disable all Telemetry uploads for an apps installation.
2) Provide a way to (persistently) set a "dummy telemetry endpoint" for an apps installation.

I assume tests depending on external services is probably not a good idea anyway.
Option 1) seems like the most straight-forward version.

Comment 2

[Georg Fritzsche [:gfritzsche]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #1)
> Option 1) seems like the most straight-forward version.

Does that sound reasonable? Should we proceed with that option?

For timing, we should solve this in Q1.

Comment 3

[Alessio Placitelli [:Dexter]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #1)
> We see two possible approaches here:
> 1) Provide a way to (persistently) disable all Telemetry uploads for an apps
> installation.

This is orthogonal, but we're also adding a way to dump pings to logcat in bug 1513563

Comment 4

[:sdaswani]

1) seems fine to me. Would this mean we set a flag in the SDK (that we can then expose as a command line startup option)?

Comment 5

[Georg Fritzsche [:gfritzsche]]

Yes, that's probably a flag in the SDK that we'd expose in some way that's convenient for the teams to use.

If you have suggestions on which mechanism works well, please leave them here.
Otherwise we'll reach out when we're picking up this bug.

Comment 6

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #1)
> 2) Provide a way to (persistently) set a "dummy telemetry endpoint" for an
> apps installation.

I see a few use cases related to 2:
- Automated tests that upload to a non-production server (hosted locally?) and the server can verify the received pings. That being said, these are fragile and may never get written
- Allowing QA to upload to production server and separate out their pings for manual verification (like bug 1507921)

If these are addressable with 1) then 1) sounds good to me!

Comment 7

[Georg Fritzsche [:gfritzsche]]

(In reply to Michael Comella (:mcomella) [needinfo or I won't see it] from comment #6)
> (In reply to Georg Fritzsche [:gfritzsche] from comment #1)
> > 2) Provide a way to (persistently) set a "dummy telemetry endpoint" for an
> > apps installation.
> 
> I see a few use cases related to 2:
> - Automated tests that upload to a non-production server (hosted locally?)
> and the server can verify the received pings. That being said, these are
> fragile and may never get written
> - Allowing QA to upload to production server and separate out their pings
> for manual verification (like bug 1507921)
> 
> If these are addressable with 1) then 1) sounds good to me!

Yes, we can address those use cases separately of 1). I think these are better addressed separately as well.

Comment 8

[Chenxia Liu [:liuche]]

We're deprioritizing this bug on Fenix FYI.

Comment 9

[Alessio Placitelli [:Dexter]]

(In reply to Chenxia Liu [:liuche] from comment #8)

We're deprioritizing this bug on Fenix FYI.

Please note that most of the use-cases in this bug are now addressed:

• the GleanTestRule (for unit and integration tests) allows for specifying a custom endpoint for tests;
• QA can inspect pings using the glean debug ping viewer;

Comment 10

[Alessio Placitelli [:Dexter]]

Closing this bug as WORKSFORME: the Glean SDK offers an API to disable metric upload. Fenix can provide a startup parameter that ends up calling that API to disable upload.