Closed Bug 1634555 Opened 3 years ago Closed 2 months ago

Send event telemetry for search default engine changes with change reasons

Categories

(Firefox :: Search, task, P2)

task

Tracking

()

VERIFIED FIXED
107 Branch
Tracking Status
firefox105 --- verified
firefox106 --- verified
firefox107 --- verified

People

(Reporter: bmiroglio, Assigned: standard8)

References

Details

(Whiteboard: [search-telemetry-backlog][snt])

Attachments

(4 files)

(this request needs proper scoping, below I describe the goal of the added telemetry)

In base telemetry, we send a main ping for a number of reasons, one of which is after a change to the environment (installing an add-on, changing your default search engine, etc.).

I am requesting a similar process for search environment changes specifically, with additional info where possible to help us better track users' search configurations. Main pings aren't enough of a signal to know what is changing without significant effort stringing together a user's collection of in-order pings and doing a diff. This is not a sustainable method for SQL-based analysis.

For example: Say a user changes their default search engine from DDG to Google. One option would be to send an event ping, i.e.

{
    ...
    'field_changed': `default_search_engine`,
    'previous_value`: `DDG`,
    'new_value': 'Google',
    'changed_by': 'user'
    ...
}

where changed_by could be user, add-on, system, etc. This is a good way to look at hijacking for instance, if we look at default_search_engine changes by way of an add-on (if this is possible!).

We'd need to come up with a list of search fields to "listen" to, and enumerate the possible values of some of the response fields. I wanted to file this bug now to get some feedback first before diving into all the specifics.

Blocks: 1632232
Type: enhancement → task
Product: bugzilla.mozilla.org → Firefox

(In reply to Ben Miroglio [:bmiroglio] from comment #0)

where changed_by could be user, add-on, system, etc. This is a good way to look at hijacking for instance, if we look at default_search_engine changes by way of an add-on (if this is possible!).

Something we can't track with events - which has been one hijacking route - is changes of the search settings from outside of Firefox, e.g. third-party apps changing the profile files, that would take effect outside of Firefox running. Though with this system you might be able to infer that from a change of the environment data without an associated event.

Priority: -- → P3

(my mistake on filing this in the wrong product!)

(In reply to Mark Banner (:standard8) from comment #1)

Something we can't track with events - which has been one hijacking route - is changes of the search settings from outside of Firefox, e.g. third-party apps changing the profile files, that would take effect outside of Firefox running. Though with this system you might be able to infer that from a change of the environment data without an associated event.

We could very likely infer that using ping sequences, I wouldn't consider it essential for this system. While hijacking is a motivating factor, it isn't the sole purpose of this work. We are interested in looking at user-directed changes too to look at trends in search default modifications.

Is it possible to distinguish an add-on changing search settings versus the user themselves? This alone would be a great start.

Flags: needinfo?(standard8)

(In reply to Ben Miroglio [:bmiroglio] from comment #2)

Is it possible to distinguish an add-on changing search settings versus the user themselves? This alone would be a great start.

Assuming we're just talking about default search engines, then yes, they come in via different routes, so we can add a flag to tell us if it was the user or an add-on that changed it.

We'll probably want to record changes of defaults if you happen to change region or locale where Firefox would then give you a different value. That's definitely possible but should be quite rare at the moment.

Flags: needinfo?(standard8)

(In reply to Mark Banner (:standard8) from comment #3)

We'll probably want to record changes of defaults if you happen to change region or locale where Firefox would then give you a different value. That's definitely possible but should be quite rare at the moment.

Yes, this would be great to capture as well

Severity: -- → S3
Whiteboard: [search-telemetry-backlog]
No longer blocks: 1632232
Severity: S3 → N/A
Assignee: nobody → standard8
Priority: P3 → P2
Summary: Send search environment change pings with change states → Send event telemetry for search default engine changes with change reasons
Whiteboard: [search-telemetry-backlog] → [search-telemetry-backlog][snt]
Attachment #9294894 - Attachment description: WIP: Bug 1634555 - Send event telemetry for search default engine changes. → Bug 1634555 - Send event telemetry for search default engine changes.
Attachment #9294894 - Attachment description: Bug 1634555 - Send event telemetry for search default engine changes. → Bug 1634555 - Send event telemetry for search default engine changes. r?daleharvey!
Attachment #9295486 - Flags: data-review?(mmccorquodale)

Comment on attachment 9295486 [details]
DefaultEngineDataReview.md

  1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
    Yes, this will be documented in the probe dictionary.

  2. Is there a control mechanism that allows the user to turn the data collection on and off?
    Yes, users can opt out of telemetry collection.

  3. If the request is for permanent data collection, is there someone who will monitor the data over time?
    Yes, this will be monitored by the search engineering team.

  4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
    Category 2, interaction data.

  5. Is the data collection request for default-on or default-off?
    Default on.

  6. Does the instrumentation include the addition of any new identifiers?
    No new identifiers.

  7. Is the data collection covered by the existing Firefox privacy notice?
    Yes.

  8. Does the data collection use a third-party collection tool?
    No.


data-review +

Attachment #9295486 - Flags: data-review?(mmccorquodale) → data-review+

As Mark pointed out - the load path and submission url are Cat 3 data as they capture partial browsing history. As we already collect this data (as documented in https://bugzilla.mozilla.org/show_bug.cgi?id=1766999#c3 ) this should be ok, but let's get a plus one from Legal to be sure.

Flags: needinfo?(mfeldman)

Agreed that the load path and submission url are cat 3 data (or something more like category 2.5). This is approved as covered by the privacy notice already as it says "We receive data about how you engage with search in Firefox and the number of searches you request from our search partners."

Flags: needinfo?(mfeldman)
Pushed by mbanner@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2eb54dc77d43
Use the testEngine_setup helper in more urlbar xpcshell-tests. r=daleharvey
https://hg.mozilla.org/integration/autoland/rev/7ffe884ac2f1
Send event telemetry for search default engine changes. r=daleharvey
Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → 107 Branch

Comment on attachment 9294894 [details]
Bug 1634555 - Send event telemetry for search default engine changes. r?daleharvey!

Beta/Release Uplift Approval Request

  • User impact if declined: Adds telemetry events to to understand the reasons that a default search engine was changed.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce: QA are testing this already.
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): QA have been testing with try builds. The search code has been fairly stable over the last few releases, so uplift issues should be limited.

Note: there are a large amount of test changes in here because one of the commonly used function calls was changed to require a second parameter.

  • String changes made/needed: None
  • Is Android affected?: No
Attachment #9294894 - Flags: approval-mozilla-beta?
Attachment #9295536 - Flags: approval-mozilla-beta?

Comment on attachment 9296524 [details]
Bug 1634555 - Send event telemetry for search default engine changes (Release)

Beta/Release Uplift Approval Request

  • User impact if declined: Adds telemetry events to to understand the reasons that a default search engine was changed.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce: QA are testing this already.
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): QA have been testing with try builds. The search code has been fairly stable over the last few releases, so uplift issues should be limited.

Note: there are a large amount of test changes in here because one of the commonly used function calls was changed to require a second parameter.

2nd note: bug 1792663 will be needed to fix a couple of extra items.

  • String changes made/needed: None
  • Is Android affected?: No
Attachment #9296524 - Flags: approval-mozilla-release?
Attachment #9295536 - Flags: approval-mozilla-release?

Comment on attachment 9294894 [details]
Bug 1634555 - Send event telemetry for search default engine changes. r?daleharvey!

Approved for 106.0b6, thanks.

Attachment #9294894 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9295536 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
See Also: → 1792985

Verified on Windows 10, Ubuntu 22.04 and Mac 11 using Nightly 107.0a1 (2022-09-29) and Beta 106.0b6.

QA Whiteboard: [qa-triaged]

Comment on attachment 9295536 [details]
Bug 1634555 - Use the testEngine_setup helper in more urlbar xpcshell-tests. r?daleharvey

Approved for 105.0.2.

Attachment #9295536 - Flags: approval-mozilla-release? → approval-mozilla-release+
Attachment #9296524 - Flags: approval-mozilla-release? → approval-mozilla-release+

Verified as fixed with 105.0.2 on Ubuntu 22.04, Mac 11 and Windows 10.

Status: RESOLVED → VERIFIED
QA Whiteboard: [qa-triaged]
Flags: qe-verify+
You need to log in before you can comment on or make changes to this bug.