Telemetry shouldn't be accepted for non-official builds

RESOLVED FIXED in mozilla13

Status

()

Toolkit
Telemetry
RESOLVED FIXED
6 years ago
3 months ago

People

(Reporter: gcp, Assigned: froydnj)

Tracking

(Depends on: 1 bug)

unspecified
mozilla13
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 3 obsolete attachments)

(Reporter)

Description

6 years ago
It looks like we currently accept Telemetry submissions from non-official builds, i.e. development builds make by developers. This is bad for at least 2 reasons:

1) Those builds will be atypical and might have severe bugs or different behavior. They will clutter up real user data, or present outliers/extreme cases that don't really exist.

2) Temporary telemetry or telemetry with typos is showing up. i.e. "1", "7", "URLCLARSIFIER" (yeah, the latter one is my fault!)

Comment 1

6 years ago
We have similar problems on the rest of the data, and it's very hard to "validate" a build. The only thing I can think of is a list of platformbuildid's that we'd check against.

Does anything like that exists?
(Reporter)

Comment 2

6 years ago
Not sure. Maybe taras can clarify if anything extra needs to be sent along. There's always the possibility of not sending Telemetry data unless we're being built with MOZ_OFFICIAL=1, I think?

Comment 3

6 years ago
good point. we currently don't prompt for telemetry on development builds, but if you enable the pref, it'll get sent. We should probably not set the server pref for unofficial builds.
Is it not possible that developers would want to look at telemetry data for these unofficial builds? What about things like partner builds?

What if we accept all the data, but we partition it on our side by adding a build-type field and have a checkbox on the dashboards that says "include unofficial builds"?
(Reporter)

Comment 5

6 years ago
>Is it not possible that developers would want to look at telemetry data for these 
>unofficial builds? 

For their local builds, they can already do so without actually submitting it (about:telemetry). 

Looking at aggregate date for developer builds: can't see much use for that honestly.

>What about things like partner builds?

If this includes things like Iceweasel then I'd really like to filter them out, yes.

>What if we accept all the data, but we partition it on our side by adding a build-type 
>field and have a checkbox on the dashboards that says "include unofficial builds"?

That's fine too, if you're able to tell what an official build is. Comment 1 suggests that this is lacking?
(In reply to Gian-Carlo Pascutto (:gcp) from comment #5)
> >Is it not possible that developers would want to look at telemetry data for these 
> >unofficial builds? 
> 
> For their local builds, they can already do so without actually submitting
> it (about:telemetry). 
> 
> Looking at aggregate date for developer builds: can't see much use for that
> honestly.

I was thinking about the case where a developer might be working on code related to a particular histogram, and would want to be able to see a trend or comparison of builds to determine if there was an improvement or regression.  If this is not a likely use case then I agree we should just change the submit URL for custom builds.  Actually, if there was a developer who wanted to keep the data, they could always spin up a local instance of the collection server for themselves and point the build at that.

> >What about things like partner builds?
> 
> If this includes things like Iceweasel then I'd really like to filter them
> out, yes.

Not so much IceWeasel as things like the Yahoo or Yandex or EUBallot builds that are our core code but customized by partners for their own distribution.
> 
> >What if we accept all the data, but we partition it on our side by adding a build-type 
> >field and have a checkbox on the dashboards that says "include unofficial builds"?
> 
> That's fine too, if you're able to tell what an official build is. Comment 1
> suggests that this is lacking?

If we could use a combination of existing fields such as applicationABI, channel, distribution, then we might be able to figure it out. Otherwise, we would need to build a list of allowed build ids and remove or flag submissions from builds that don't match.
(Assignee)

Comment 7

6 years ago
This is a small patch to Telemetry to consult MOZ_OFFICIAL; I'll work on that.
Assignee: nobody → nfroyd
(Assignee)

Comment 8

6 years ago
Created attachment 593781 [details] [diff] [review]
patch

Here's a simple patch: add isOfficial boolean to the interface and make sure that we only send "idle-daily" reasons for idle pings, "test-ping" for everything else.
Attachment #593781 - Flags: review?(taras.mozilla)

Comment 9

6 years ago
Comment on attachment 593781 [details] [diff] [review]
patch

use MOZ_TELEMETRY_REPORTING

use CanSend.
Attachment #593781 - Flags: review?(taras.mozilla) → review+
(Assignee)

Comment 10

6 years ago
Created attachment 593805 [details] [diff] [review]
patch

Renamed the variable and made it dependent on official-ness and telemetry reporting.

Note that we now test MOZILLA_OFFICIAL, not MOZ_OFFICIAL_BRANDING.  The former is what gets used for all builds (Nightly, Aurora, Beta, Release), whereas the latter only gets used for release builds.
Attachment #593781 - Attachment is obsolete: true
Attachment #593805 - Flags: review+
(Assignee)

Comment 11

6 years ago
Created attachment 593806 [details] [diff] [review]
patch

Er.  It works better if you rebase before attaching patches.
Attachment #593805 - Attachment is obsolete: true
Attachment #593806 - Flags: review+
(Assignee)

Comment 12

5 years ago
Created attachment 595767 [details] [diff] [review]
patch

Rebased to deal with fuzz.  Carrying over r+.
Attachment #593806 - Attachment is obsolete: true
Attachment #595767 - Flags: review+
(Assignee)

Updated

5 years ago
Keywords: checkin-needed
(Reporter)

Comment 13

5 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/1c1ad3dcd782
Keywords: checkin-needed
Group: metrics-private
Component: Data/Backend Reports → Telemetry
Product: Mozilla Metrics → Toolkit
QA Contact: data-reports → telemetry
https://hg.mozilla.org/mozilla-central/rev/1c1ad3dcd782
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla13
(Assignee)

Updated

5 years ago
Blocks: 763991
Depends on: 826998
You need to log in before you can comment on or make changes to this bug.