1515965 - Ensure telemetry of forks of the application do not end up in the same analysis space as the core application

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect, P3)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

We're open source so sometimes our applications are forked, modified, and released to a large population. However, teams outside of Mozilla are not likely to modify our telemetry code so if a forked application's telemetry uploads to the same place by default, it will skew our numbers.

We should prevent forked applications from uploading to the same place. Some thoughts:
- Use an identifier that will change for forked applications (unclear what those are but we may need to combine several like package name, app name, etc.)
- Include API keys at build time for Mozillian development and release builds; forks will not have these keys

Comment 1

[Alessio Placitelli [:Dexter]]

Hi Michael,

I think this should be covered by bug 1515031 already. We are sending the native application identifier (which changes for forks): talking to Stefan, this should solve the problem.

Does this address your concern?

Comment 2

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

There are several categories of builds derived from our source code that I've identified:
1) Our release builds
2) Our development builds
3) Unofficial, unmodified builds of our source code (e.g. people build APKs of Firefox for Fire TV because we don't provide any atm)
4) Modifications of our source code with the same app ID
5) Modifications of our source code with different app IDs

The solution in bug 1515031 addresses 1, 2, and 5. However, 3 and 4 will be included in our release telemetry numbers (note: it's not clearly wrong to include 3). This is debatably "good enough": it covers the majority use case and has a low probability of being a problem. Our applicationId is only unique on the services we distribute on (e.g. FFTV is only distributed on Amazon and someone can use the same ID on Play [1]) or in a non-app store build. If these forked builds become popular, or highly automated [2], it can unacceptably skew our numbers.

If we wanted to keep 1 in its own bucket (my preference), we could try to aggregate several identifiers that are more likely to get changed or switch to a different method like "API keys". If we want to keep the solution as is, we should talk to our product owners to see if the risk is acceptable.

What do you think we should do, Alessio?

---

Tangential thought: in addition to separating release telemetry from other builds, it's a possibility that we can uniquely identify each of these build types in our telemetry, for which I can see some use cases. However, we should probably wait for a product owner to ask for this before we start collecting that data.

[1]: I wrote an email about addressing this problem: https://groups.google.com/a/mozilla.com/d/msg/mobile-all/ELVmIuophps/VFrRClUvDAAJ
[2]: This is also a problem on our official release builds

Comment 3

[Alessio Placitelli [:Dexter]]

(In reply to Michael Comella (:mcomella) [away until 12/31; needinfo or I won't see it] from comment #2)
> There are several categories of builds derived from our source code that
> I've identified:
> 1) Our release builds
> 2) Our development builds
> 3) Unofficial, unmodified builds of our source code (e.g. people build APKs
> of Firefox for Fire TV because we don't provide any atm)
> 4) Modifications of our source code with the same app ID
> 5) Modifications of our source code with different app IDs
> 
> The solution in bug 1515031 addresses 1, 2, and 5. However, 3 and 4 will be
> included in our release telemetry numbers (note: it's not clearly wrong to
> include 3). This is debatably "good enough": it covers the majority use case
> and has a low probability of being a problem. Our applicationId is only
> unique on the services we distribute on (e.g. FFTV is only distributed on
> Amazon and someone can use the same ID on Play [1]) or in a non-app store
> build. If these forked builds become popular, or highly automated [2], it
> can unacceptably skew our numbers.

I think automation is a different, orthogonal issue. We're tracking that as part of bug 1514336.
 
> If we wanted to keep 1 in its own bucket (my preference), we could try to
> aggregate several identifiers that are more likely to get changed or switch
> to a different method like "API keys". If we want to keep the solution as
> is, we should talk to our product owners to see if the risk is acceptable.
> 
> What do you think we should do, Alessio?

These are all good points, thanks for bringing them up. As for 3, 4, this could be mitigated by providing a way to manually set custom ids at init. These would work like "api keys", and could be injected at build time on our infra.

Georg, any clue on this?

> 
> ---
> 
> Tangential thought: in addition to separating release telemetry from other
> builds, it's a possibility that we can uniquely identify each of these build
> types in our telemetry, for which I can see some use cases. However, we
> should probably wait for a product owner to ask for this before we start
> collecting that data.

This should probably be possible automatically through the "app_build" field sent with each ping (https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings.md). We still have to agree on a format for this, bug 1508305 will deal with it.

> 
> [1]: I wrote an email about addressing this problem:
> https://groups.google.com/a/mozilla.com/d/msg/mobile-all/ELVmIuophps/
> VFrRClUvDAAJ
> [2]: This is also a problem on our official release builds

Comment 4

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

> These would work like "api keys", and could be injected at build time on our infra.

fwiw, we currently have some gradle code to add API keys to the BuildConfig class [1] at build time. I'm hoping we share build code soon in a gradle plugin (after [2][3]) so we can share that "add API keys" task at build time.

[1]: https://github.com/mozilla-mobile/firefox-tv/blob/6b36f79c030b186feba68562fb999415031cae68/app/build.gradle#L211

[2]: https://github.com/mozilla-mobile/android-automation-tools/issues/24
[3]: https://github.com/mozilla-mobile/android-automation-tools/pull/32

Comment 5

[Alessio Placitelli [:Dexter]]

This is addressed: forked applications keeping the original application id won't show up in the play store. If the application id is changed, it will show up in a different analysis space. We fixed handling the application id in bug 1536408.