1643490 - Add event telemetry probe to measure the number of words in the input on engagement

Status: RESOLVED FIXED

(Firefox :: Address Bar, task, P2)

Description

[Harry Twyford]

This is to improve the data for the experiment in bug 1643386. Event telemetry depends on UrlbarInput rather than any particular queryContext, so it may be too expensive to tokenize the input value for every event telemetry collection. For the kind of analysis we plan on doing, measuring the number of elements in value.trim().split(" ") should be adequate. It should be noted this will produce some mismeasurements for URLs with spaces in the query parameter, or for queries that use keywords.

Comment 1

[Harry Twyford]

Attachment: Bug 1643490 - Add event telemetry probe to measure the number of words in the input on engagement. r?adw

Comment 2

[Harry Twyford]

Comment on attachment 9154369 [details]
Bug 1643490 - Add event telemetry probe to measure the number of words in the input on engagement. r?adw

1. What questions will you answer with this data?

How many words do users typically enter before selecting a suggestion result in the address bar?

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

If users don't choose suggestions for short/1-word queries, we could explore not fetching suggestions for short queries to improve user privacy.

3. What alternative methods did you consider to answer these questions? Why were they not sufficient?

Counting the number of characters only, which we currently have a probe for. We'd like the additional nuance of how may words they've typed.

4. Can current instrumentation answer these questions?

No, only how many characters were typed.

5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Description: The number of words in the user's query. Measured by trimming the query, then splitting it up by the number of spaces and returning the number of split words. The content of the query is not collected.
Category: 2
Bug for the new probe: bug 1643490
Bug for the experiment: bug 1643386

6. How long will this data be collected?

Permanently a part of event telemetry collection. The analysis of this data is part of a time-limited experiment planned to run 2020-06-23 to 2020-07-07. See https://experimenter.services.mozilla.com/experiments/measure-the-string-length-on-user-selected-suggestions/

7. What populations will you measure?

The experiment targets all English locales in Release in all countries.

8. If this data collection is default on, what is the opt-out mechanism for users?

Disabling event telemetry or opting out of the experiment.

9. Please provide a general description of how you will analyze this data.

Comparing the number of words typed with when suggestions are picked from the address bar dropdown.

10. Where do you intend to share the results of your analysis?

Internally, with the Search team.

11. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:

No.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f13d74051cc3
Add event telemetry probe to measure the number of words in the input on engagement. r=adw

Comment 4

[Harry Twyford]

Ah, I'm sorry. I landed this as a force of habit, before data review completed. I'll request backout. No telemetry should have been collected, since this is all behind the event telemetry pref.

Comment 5

[Razvan Maries]

Backout: https://hg.mozilla.org/integration/autoland/rev/832befa92f299328debefbf7a56c6d52a4c488b9

Comment 6

[Ben Miroglio [:bmiroglio]]

Comment on attachment 9154369 [details]
Bug 1643490 - Add event telemetry probe to measure the number of words in the input on engagement. r?adw

Data Review Form

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

This will be documented in the [Events.yaml file[(https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/event-ping.html). Note this is data specific to an experiment at this moment in time.

2. Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Users can disable this by opt-ing out of Telemetry in the Firefox preferences.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, :harry and the DS team will monitor

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2: Interaction Data

5. Is the data collection request for default-on or default-off?

default-on

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

7. Is the data collection covered by the existing Firefox privacy notice?

Yes.

8. Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**

Yes, this should be reevaluated after a few months of collection.

9. Does the data collection use a third-party collection tool?

No.

data-review: r+

Comment 7

[Ben Miroglio [:bmiroglio]]

This one comment outside of data review:

Would a histogram be more efficient to use here? I recall from past experiments that were sending search events that the volume of events was quite large, and not sustainable for 100% of release.

Comment 8

[Harry Twyford]

It would be nice to get the fine-grained data that event telemetry sends since we're particularly interested in how the number of words typed relates to the type of result picked. For example: are suggestions more likely to be picked when many words are typed?

We don't really need continuous monitoring of this; we're just looking to run a single experiment. I plan to create an experiment add-on that enables event telemetry and filters for the data we need. I have the experiment set up in Experimenter for 0.5% of Release because of the huge volume of data event telemetry produces.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/06798bf879da
Add event telemetry probe to measure the number of words in the input on engagement. r=adw

Comment 10

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/06798bf879da

Comment 11

[Harry Twyford]

Comment on attachment 9154369 [details]
Bug 1643490 - Add event telemetry probe to measure the number of words in the input on engagement. r?adw

Beta/Release Uplift Approval Request

• User impact if declined: We won't be able to run an experiment that's targeting 78. See bug 1643386.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Adds one new parameter to an existing and well-tested event telemetry object. There are several tests for the added probe. It isn't collected by default, but only for those enrolled in an add-on experiment.
• String changes made/needed:

Comment 12

[Julien Cristau [:jcristau]]

Comment on attachment 9154369 [details]
Bug 1643490 - Add event telemetry probe to measure the number of words in the input on engagement. r?adw

approved for 78.0b6

Comment 13

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/970db666ddf6