1634555 - Send event telemetry for search default engine changes with change reasons

Status: VERIFIED FIXED

(Firefox :: Search, task, P2)

Description

[Ben Miroglio [:bmiroglio]]

(this request needs proper scoping, below I describe the goal of the added telemetry)

In base telemetry, we send a main ping for a number of reasons, one of which is after a change to the environment (installing an add-on, changing your default search engine, etc.).

I am requesting a similar process for search environment changes specifically, with additional info where possible to help us better track users' search configurations. Main pings aren't enough of a signal to know what is changing without significant effort stringing together a user's collection of in-order pings and doing a diff. This is not a sustainable method for SQL-based analysis.

For example: Say a user changes their default search engine from DDG to Google. One option would be to send an event ping, i.e.

{
    ...
    'field_changed': `default_search_engine`,
    'previous_value`: `DDG`,
    'new_value': 'Google',
    'changed_by': 'user'
    ...
}

where changed_by could be user, add-on, system, etc. This is a good way to look at hijacking for instance, if we look at default_search_engine changes by way of an add-on (if this is possible!).

We'd need to come up with a list of search fields to "listen" to, and enumerate the possible values of some of the response fields. I wanted to file this bug now to get some feedback first before diving into all the specifics.

Comment 1

[Mark Banner (:standard8)]

(In reply to Ben Miroglio [:bmiroglio] from comment #0)

where changed_by could be user, add-on, system, etc. This is a good way to look at hijacking for instance, if we look at default_search_engine changes by way of an add-on (if this is possible!).

Something we can't track with events - which has been one hijacking route - is changes of the search settings from outside of Firefox, e.g. third-party apps changing the profile files, that would take effect outside of Firefox running. Though with this system you might be able to infer that from a change of the environment data without an associated event.

Comment 2

[Ben Miroglio [:bmiroglio]]

(my mistake on filing this in the wrong product!)

(In reply to Mark Banner (:standard8) from comment #1)

Something we can't track with events - which has been one hijacking route - is changes of the search settings from outside of Firefox, e.g. third-party apps changing the profile files, that would take effect outside of Firefox running. Though with this system you might be able to infer that from a change of the environment data without an associated event.

We could very likely infer that using ping sequences, I wouldn't consider it essential for this system. While hijacking is a motivating factor, it isn't the sole purpose of this work. We are interested in looking at user-directed changes too to look at trends in search default modifications.

Is it possible to distinguish an add-on changing search settings versus the user themselves? This alone would be a great start.

Comment 3

[Mark Banner (:standard8)]

(In reply to Ben Miroglio [:bmiroglio] from comment #2)

Is it possible to distinguish an add-on changing search settings versus the user themselves? This alone would be a great start.

Assuming we're just talking about default search engines, then yes, they come in via different routes, so we can add a flag to tell us if it was the user or an add-on that changed it.

We'll probably want to record changes of defaults if you happen to change region or locale where Firefox would then give you a different value. That's definitely possible but should be quite rare at the moment.

Comment 4

[Ben Miroglio [:bmiroglio]]

(In reply to Mark Banner (:standard8) from comment #3)

We'll probably want to record changes of defaults if you happen to change region or locale where Firefox would then give you a different value. That's definitely possible but should be quite rare at the moment.

Yes, this would be great to capture as well

Comment 5

[Mark Banner (:standard8)]

Attachment: Bug 1634555 - Send event telemetry for search default engine changes. r?daleharvey!

Comment 6

[Mark Banner (:standard8)]

Attachment: DefaultEngineDataReview.md

Comment 7

[Mark Banner (:standard8)]

Attachment: Bug 1634555 - Use the testEngine_setup helper in more urlbar xpcshell-tests. r?daleharvey

Comment 8

[Megan McCorquodale]

Comment on attachment 9295486 [details]
DefaultEngineDataReview.md

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
   Yes, this will be documented in the probe dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?
   Yes, users can opt out of telemetry collection.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?
   Yes, this will be monitored by the search engineering team.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
   Category 2, interaction data.

5. Is the data collection request for default-on or default-off?
   Default on.

6. Does the instrumentation include the addition of any new identifiers?
   No new identifiers.

7. Is the data collection covered by the existing Firefox privacy notice?
   Yes.

8. Does the data collection use a third-party collection tool?
   No.

---

data-review +

Comment 9

[Megan McCorquodale]

As Mark pointed out - the load path and submission url are Cat 3 data as they capture partial browsing history. As we already collect this data (as documented in https://bugzilla.mozilla.org/show_bug.cgi?id=1766999#c3 ) this should be ok, but let's get a plus one from Legal to be sure.

Comment 10

[Michael Feldman (use NI)]

Agreed that the load path and submission url are cat 3 data (or something more like category 2.5). This is approved as covered by the privacy notice already as it says "We receive data about how you engage with search in Firefox and the number of searches you request from our search partners."

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2eb54dc77d43
Use the testEngine_setup helper in more urlbar xpcshell-tests. r=daleharvey
https://hg.mozilla.org/integration/autoland/rev/7ffe884ac2f1
Send event telemetry for search default engine changes. r=daleharvey

Comment 12

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/2eb54dc77d43
https://hg.mozilla.org/mozilla-central/rev/7ffe884ac2f1

Comment 13

[Mark Banner (:standard8)]

Comment on attachment 9294894 [details]
Bug 1634555 - Send event telemetry for search default engine changes. r?daleharvey!

Beta/Release Uplift Approval Request

• User impact if declined: Adds telemetry events to to understand the reasons that a default search engine was changed.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce: QA are testing this already.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): QA have been testing with try builds. The search code has been fairly stable over the last few releases, so uplift issues should be limited.

Note: there are a large amount of test changes in here because one of the commonly used function calls was changed to require a second parameter.

• String changes made/needed: None
• Is Android affected?: No

Comment 14

[Mark Banner (:standard8)]

Attachment: Bug 1634555 - Send event telemetry for search default engine changes (Release)

Depends on D158279

Comment 15

[Mark Banner (:standard8)]

Comment on attachment 9296524 [details]
Bug 1634555 - Send event telemetry for search default engine changes (Release)

Beta/Release Uplift Approval Request

• User impact if declined: Adds telemetry events to to understand the reasons that a default search engine was changed.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce: QA are testing this already.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): QA have been testing with try builds. The search code has been fairly stable over the last few releases, so uplift issues should be limited.

Note: there are a large amount of test changes in here because one of the commonly used function calls was changed to require a second parameter.

2nd note: bug 1792663 will be needed to fix a couple of extra items.

• String changes made/needed: None
• Is Android affected?: No

Comment 16

[Pascal Chevrel:pascalc]

Comment on attachment 9294894 [details]
Bug 1634555 - Send event telemetry for search default engine changes. r?daleharvey!

Approved for 106.0b6, thanks.

Comment 17

[Pascal Chevrel:pascalc]

https://hg.mozilla.org/releases/mozilla-beta/rev/20aa78e2f805
https://hg.mozilla.org/releases/mozilla-beta/rev/29cbf057f9e8

Comment 18

[Adrian Florinescu [:aflorinescu]]

Verified on Windows 10, Ubuntu 22.04 and Mac 11 using Nightly 107.0a1 (2022-09-29) and Beta 106.0b6.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9295536 [details]
Bug 1634555 - Use the testEngine_setup helper in more urlbar xpcshell-tests. r?daleharvey

Approved for 105.0.2.

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/acb9ee6e21b8
https://hg.mozilla.org/releases/mozilla-release/rev/fc80157e28c0

Comment 21

[Petruta Horea [:phorea], Desktop Test Engineering]

Verified as fixed with 105.0.2 on Ubuntu 22.04, Mac 11 and Windows 10.