1747229 - [Colorway Closet] Investigate new telemetry probes

Status: VERIFIED FIXED

(Firefox :: New Tab Page, task, P2)

Description

[Dão Gottwald [:dao]]

This is in support of the success metrics for Colorway closet:

Primary metrics (our goal is to move them):

• Desktop days of use
• New Tab Desktop Search

Secondary metrics (our theory is that influencing these metrics will directly impact primary metrics):

• Days of use attributable to an increase in colorway/background image engagement
• Conversion of existing users from custom home page to Firefox home page default attributable to the colorway closet
• New Tab Search attributable to an increase in default Home Page settings
• Count of users that enable colorways/background images from the colorway closet

Comment 1

[Bernard Igiri]

Resources to view Telemetry

• Desktop Days of Use on Looker: https://mozilla.cloud.looker.com/dashboards/101?Date=after+2021%2F01%2F01
• Desktop Days of Use char on Airflow: https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_main_summary
• Desktop Days of Use Query: https://sql.telemetry.mozilla.org/queries/84256/source
• Telemetry in Firefox (scroll down to Theme): about:telemetry#environment-data-tab_addons

Documentation

• Adding a Telemetry probe: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/start/adding-a-new-probe.html
• Environment: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/environment.html

Relevant Code

• Method to track search telemetry: https://searchfox.org/mozilla-central/source/browser/components/search/BrowserSearchTelemetry.jsm#177
• Method to track addon telemetry: https://searchfox.org/mozilla-central/source/toolkit/mozapps/extensions/AddonManager.jsm#5023

Comment 2

[kpatenio]

Attachment: Bug 1747229 - implement telemetry probes for colorway closet

Comment 3

[kpatenio]

Attachment: Colorways Data Collection Request.txt

Hi :chutten,
This is a data collection request for Colorway Closet. Could you please take a look when you have a chance?

Comment 4

[Chris H-C :chutten]

Comment on attachment 9286515 [details]
Colorways Data Collection Request.txt

PRELIMINARY NOTES:
dictionary.telemetry.mozilla.org is only available for Glean collections. Since your code (at least the version I looked at) is using Firefox Telemetry only, its public documentation is available at probes.telemetry.mozilla.org/. (( If you'd like to use Glean APIs and tooling and still have your data show up as Telemetry Events we support that and are happy to help. If Telemetry Events alone are working out for you, then great! Just remember its stumbling blocks ))

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Ray Fambro is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

---

Result: datareview+

Comment 5

[kpatenio]

(In reply to Chris H-C :chutten from comment #4)

Comment on attachment 9286515 [details]
Colorways Data Collection Request.txt

PRELIMINARY NOTES:
dictionary.telemetry.mozilla.org is only available for Glean collections. Since your code (at least the version I looked at) is using Firefox Telemetry only, its public documentation is available at probes.telemetry.mozilla.org/. (( If you'd like to use Glean APIs and tooling and still have your data show up as Telemetry Events we support that and are happy to help. If Telemetry Events alone are working out for you, then great! Just remember its stumbling blocks ))

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Ray Fambro is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

---

Result: datareview+

Thanks Chris! I will reach out to Ray to determine if we want to move forward with Glean for these probes or not.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/90d9a155a43f
implement telemetry probes for colorway closet r=bigiri,dao,rpl

Comment 7

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/90d9a155a43f

Comment 8

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/90d9a155a43f

Comment 9

[Carmen Fat [:cfat] - Ecosystem QA]

• I have verified the telemetry probes implementation with Firefox Beta 106.0b4 (Build ID - 20220925185751) using Windows 10 x64, Ubuntu 20.04 x64 and macOS 10.15.7.
• The following telemetry probes were confirmed that they are triggered correctly and sent to a local server we previously set on our machine:

Events:
colorways_modal try_colorways firefoxview
colorways_modal change_colorway firefoxview
colorways_modal try_colorways aboutaddons
colorways_modal change_colorway aboutaddons
colorways_modal set_colorway modal null {"colorwayid": "(theme+intensity)[e-mail]"}
addonsManager enable theme (theme+intensity)[e-mail]
colorways_modal cancel modal
colorways_modal homepage_reset modal
colorways_modal homepage_reset _undo modal

Keyed Scalars:
colorways_modal#try_colorways#firefoxview
colorways_modal#change_colorway#firefoxview
colorways_modal#try_colorways#aboutaddons
colorways_modal#change_colorway#aboutaddons
colorways_modal#set_colorway#modal
colorways_modal#cancel#modal
colorways_modal#homepage_reset#modal
colorways_modal#homepage_reset_undo#modal

Keyed Histograms:
(theme+intensity)[e-mail]