1560585 - Detail North Start Metric for Implementation

Status: RESOLVED FIXED

(Data Science :: General, task)

Description

[Ryan Harter [:harter]]

In Bug 1518829 we defined a north star metric. We need to refine the definition to make sure engineering can implement the measurement accurately.

Comment 1

[Ryan Harter [:harter]]

Assigning Teon. :mak - can you link to the questions we discussed today?

Thanks all!

Comment 2

[Marco Bonardo [:mak]]

I'll try to sum up most of the outcomes from our discussion.

Basically we confirmed that we want the suggested 2 metrics, but that it's unclear which grouping/stratification will be the best to analyze them. Thus we need more context to better analyze them. The suggested solution is to use Event Telemetry, and in particular consider 2 events:

1. a succesful interaction, recorded when the user confirms a urlbar pick
2. an unsuccesful interaction, recorded when the user action diverges from urlbar results (navigates elsewhere, interrupts the search)
   The recorded event may contain this info:
   a. time elapsed from the first input interaction
   b. number of typed characters
   c. type of input (was it a paste or a typed string)
   d. type of pick (search, autofill, adaptive, frecency...)

The first thing to answer is whether our infrastructure would support such amount of events, and that can be done using the existing urlbar telemetry to count the number of successful interactions. Unfortunately we don't have metrics useful for unsuccesful ones, so it will have to be somehow extracted artificially.
The second thing to answer is whether there are important cases in which we think we may not want to record these events (private browsing is obviously one of these)
The third thing is involving privacy to evaluate eventual risks.

Please, feel free to correct or complete anything I misunderstood or missed.

Comment 3

[Ryan Harter [:harter]]

Thanks, :mak. I think that captures everything.

:sunah, we talked about this briefly in Whistler. We're interested in generating an event for every interaction with the AwesomeBar. You mentioned you had a tool that would help us estimate the downside of implementing with events instead of histograms. Can you link to that here?

Thanks again, all!

Comment 4

[Marco Bonardo [:mak]]

Note, these days Teon is also working on evaluating the possible number of events through the existing histograms.

Comment 5

[bugzilla]

To clarify: I thought about writing a tool after I did the last cost analysis by hand (see bug 1527442 and its dependent bugs, where that team sampled down to 1% of release clients). I'll sync up with Teon re: how many new events we think this will generate and go from there. I'm inclined to say that, for tracking this metric, we should consider doing the same re: sampling down to 1%, and add a pref that can enable the events for any clients in a relevant experiment.

Comment 6

[Marco Bonardo [:mak]]

The current Firefox engineering plan (while we wait for numbers/decision) is in https://bugzilla.mozilla.org/show_bug.cgi?id=1559136. We'll implement event telemetry behind a pref for now. Up to data to decide the population to target.

Comment 7

[Bridget Kaluzny]

Teon- we think this bug is resolved, can you close/resolved this bugzilla please.

Comment 8

[Teon Brooks [:teon]]

tl;dr: estimated around 200 million interactions per day with the URL bar. below is a link to the notebook I used for this estimation.

https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/139351/