841978 - Disable telemetry for mozbase dependent test harnesses

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Matthew N. [:MattN]]

Bug 819732 and its dependencies enabled telemetry by default on Nightly and Aurora.  We should disable telemetry in Mozmill to avoid skewing the data.

I think setting toolkit.telemetry.enabled and toolkit.telemetry.enabledPreRelease to false will do this but Taras can correct me if there is a better way.

Comment 1

[Lawrence Mandel [:lmandel] (use needinfo)]

Although I'm not 100% sure that we want to disable Telemetry (I don't know all of the details here), Theo wrote the enabled-by-default code and should be able to advise on the best way to disable Telemetry for Mozmill.

Comment 2

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Lawrence Mandel [:lmandel] from comment #1)
> Although I'm not 100% sure that we want to disable Telemetry (I don't know
> all of the details here), 

Ignore this part of my response. I just read bug 841967.

Comment 3

[Justin Dolske [:Dolske]]

This is kind of a big deal. The probe noted in 841967 is overwhelmed with bogus-looking data, and the conjecture(?) is that it's due to Mozmill machines reporting. That rather spoils the usefulness of Telemerty data. If Mozmill runs are actually causing telemetry data to be reported, that needs to stop ASAP.

Comment 4

[Théo Chevalier]

If you run Mozmill on "default" channel, for instance, you’re safe.

If you run Mozmill on "nightly" channel, as you said you will have to set toolkit.telemetry.enablePreRelease to false. (toolkit.telemetry.enabled is already false)

Comment 5

[Vladan Djeric (:vladan)]

Is Mozmill typically installed as an extension? In other words, can we filter out the Mozmill Telemetry reports at the server by checking the list of installed extensions in the report?

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As I have read on another bug we send telemetry data after about 24h of usage. Is that correct? We do not re-use profiles when running Mozmill tests and we do not exceed 60 minutes of test runs for a single profile.

Do you have machine names or IP addresses you think of that those are from Mozmill runs?

Comment 7

[Matthew N. [:MattN]]

(In reply to Henrik Skupin (:whimboo) from comment #6)
> As I have read on another bug we send telemetry data after about 24h of
> usage. Is that correct? We do not re-use profiles when running Mozmill tests
> and we do not exceed 60 minutes of test runs for a single profile.

Vladan probably knows.

> Do you have machine names or IP addresses you think of that those are from
> Mozmill runs?

No, but we can get that info in bug 841967.  I suspect we would rather not rely on the above behaviour to prevent telemetry from being sent so we should disable telemetry in Mozmill regardless of whether it caused bug 841967.

Comment 8

[Vladan Djeric (:vladan)]

(In reply to Matthew N. [:MattN] from comment #7)
> (In reply to Henrik Skupin (:whimboo) from comment #6)
> > As I have read on another bug we send telemetry data after about 24h of
> > usage. Is that correct? We do not re-use profiles when running Mozmill tests
> > and we do not exceed 60 minutes of test runs for a single profile.
> 
> Vladan probably knows.

Firefox submits Telemetry at most once every 24 hours. However, on first run in a new profile, it will submit Telemetry after 5 minutes of system-wide user inactivity.

> I suspect we would rather not
> rely on the above behaviour to prevent telemetry from being sent so we
> should disable telemetry in Mozmill regardless of whether it caused bug
> 841967.

Sounds like a plan. You can disable Telemetry by setting the toolkit.telemetry.enabled and toolkit.telemetry.enabledPreRelease prefs to false.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Preferences are getting set via mozprofile (mozbase). That means more harnesses than Mozmill can be stopped from reporting telemetry information. I will work on a patch. Thanks for the information Vladan.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1 (master)

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1 (mozmill 1.5)

Comment 12

[Jeff Hammel]

Ultimately preferences will be more distributed, see bug 830430, but in the interim this approach is fine

Comment 13

[Jeff Hammel]

Comment on attachment 715961 [details] [diff] [review]
Patch v1 (master)

WFM, but see previous comment

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 715965 [details] [diff] [review]
Patch v1 (mozmill 1.5)

Review of attachment 715965 [details] [diff] [review]:
-----------------------------------------------------------------

https://github.com/mozilla/mozmill/commit/cd219b0b75ece5b1927f63a8036a0d0413c620e7 (mozmill-1.5)

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 715961 [details] [diff] [review]
Patch v1 (master)

Review of attachment 715961 [details] [diff] [review]:
-----------------------------------------------------------------

https://github.com/mozilla/mozbase/commit/0a8ab18098bcde4fabe486fd39c716fd463f5dd8 (master)

Comment 16

[Vladan Djeric (:vladan)]

Currently, Telemetry only checks its prefs on startup & doesn't react to prefs changed during the current session.

I took a quick look at the git repo, but I just want to make sure: these patches will set the toolkit.telemetry.[..]enabled prefs when the profile is created, right?

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yes, that's exactly what the patches are doing. Those prefs are setup before the application gets started the first time.