Closed Bug 1345080 Opened 7 years ago Closed 7 years ago
Decide what to do with the SEARCH
_SERVICE _ENGINE _COUNT probe
We added this probe in bug 1268424. I think the goal of adding this was to inform UX decisions related to presentation of search plugins in the UI (eg. one-off buttons, search settings). I'm not sure anybody is looking at the collected data, and this probe is expiring with Firefox 55. We should either: - make it permanent (which means cover it with automated tests) - remove it, if nobody intends to use this data - expand it, to collect more detailed/useful data. Here is what the data looks like: https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-25&keys=__none__!__none__!__none__&max_channel_version=release%252F51&measure=SEARCH_SERVICE_ENGINE_COUNT&min_channel_version=null&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-18&table=0&trim=1&use_submission_date=0
This probe makes a lot of sense as a part of our search measurement implementation initiative. I think this is a measurement we want to have. Can we replace it with a Scalar, so that we get the actual non-binned count? We'd want this to be opt-out (calling this out, as it looks like it already is). Also, can we key it by type of search plugin (built-in or user-installed, and visible one-off or hidden)? I'd be particularly interested in the proportion of user-installed vs. built-in. Presumably, this would also answer the question of how many one-off buttons are shown in the dropdown.
(In reply to Dave Zeber [:dzeber] from comment #1) > I'd be particularly interested in the > proportion of user-installed vs. built-in. I would also be interested in knowing about hidden built-in plugins (when the user clicks "Remove" for a built-in engine, we just hide it).
(In reply to Dave Zeber [:dzeber] from comment #1) > Can we replace it with a Scalar, so that we get the actual non-binned count? > We'd want this to be opt-out (calling this out, as it looks like it already > is). I think we should do this, yes. This may require changing the name of the probe. > Also, can we key it by type of search plugin (built-in or user-installed, > and visible one-off or hidden)? I don't know if that's compatible with a scalar. We may need to add a separate value for the count of built-in engines, the count of user-installed engines, the count of hidden one-offs.
(In reply to Florian Quèze [:florian] [:flo] from comment #3) > I don't know if that's compatible with a scalar. We may need to add a > separate value for the count of built-in engines, the count of > user-installed engines, the count of hidden one-offs. Ah yes, I was thinking of this (https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/collection/scalars.html#keyed-scalars), but it does say separate scalars are more appropriate. Could we do counts for all combinations: builtin/visible, builtin/hidden, user-added/visible, user-added/hidden? IIUC, the total count of all visible plugins == the number of one-off buttons in the search dropdown.
Dave chimed in, I'm removing the NI.
Since this probe is already gone in 55 and we established that we need to keep both the old probe and the new proposed one for at least a while, let's use this bug to just extend the lifetime of the existing probe and then file a followup for the better measurement.
Priority: -- → P1
Per the discussion in the previous comments, it seems we want to keep this probe for another few release to give us time to implement a more detailed permanent probe, and compare the results of the two. So I'm bumping the expires_in_version value by 5 releases, and requesting a data review.
Attachment #8847625 - Flags: review?(rweiss)
7 years ago
Assignee: nobody → florian
Status: NEW → ASSIGNED
Comment on attachment 8847625 [details] [diff] [review] bump the max version Given dzeber's answers to the above questions, extending the collection for an additional 5 versions is fine.
Attachment #8847625 - Flags: review?(rweiss) → review+
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/mozilla-inbound/rev/59baa83a1f66 bump the max version of the SEARCH_SERVICE_ENGINE_COUN probe, r=rweiss.
QA Contact: brindusa.tot
Verified that the telemetry collection https://mzl.la/2tQLi5Z contains results for SEARCH_SERVICE_ENGINE_COUNT probe also for Nightly 56 so I can confirm that it hasn't expired in 55. If there are other ways we could manually verify this, please let us know.
You need to log in before you can comment on or make changes to this bug.