1345080 - Decide what to do with the SEARCH_SERVICE_ENGINE_COUNT probe

Status: VERIFIED FIXED

(Firefox :: Search, enhancement, P1)

Description

[Florian Quèze [:florian] (on leave until October 9)]

We added this probe in bug 1268424. I think the goal of adding this was to inform UX decisions related to presentation of search plugins in the UI (eg. one-off buttons, search settings).

I'm not sure anybody is looking at the collected data, and this probe is expiring with Firefox 55.

We should either:
- make it permanent (which means cover it with automated tests)
- remove it, if nobody intends to use this data
- expand it, to collect more detailed/useful data.

Here is what the data looks like: https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-25&keys=__none__!__none__!__none__&max_channel_version=release%252F51&measure=SEARCH_SERVICE_ENGINE_COUNT&min_channel_version=null&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-18&table=0&trim=1&use_submission_date=0

Comment 1

[Dave Zeber [:dzeber]]

This probe makes a lot of sense as a part of our search measurement implementation initiative. I think this is a measurement we want to have.

Can we replace it with a Scalar, so that we get the actual non-binned count? We'd want this to be opt-out (calling this out, as it looks like it already is).

Also, can we key it by type of search plugin (built-in or user-installed, and visible one-off or hidden)? I'd be particularly interested in the proportion of user-installed vs. built-in. Presumably, this would also answer the question of how many one-off buttons are shown in the dropdown.

Comment 2

[Florian Quèze [:florian] (on leave until October 9)]

(In reply to Dave Zeber [:dzeber] from comment #1)
> I'd be particularly interested in the
> proportion of user-installed vs. built-in.

I would also be interested in knowing about hidden built-in plugins (when the user clicks "Remove" for a built-in engine, we just hide it).

Comment 3

[Florian Quèze [:florian] (on leave until October 9)]

(In reply to Dave Zeber [:dzeber] from comment #1)

> Can we replace it with a Scalar, so that we get the actual non-binned count?
> We'd want this to be opt-out (calling this out, as it looks like it already
> is).

I think we should do this, yes. This may require changing the name of the probe.

> Also, can we key it by type of search plugin (built-in or user-installed,
> and visible one-off or hidden)?

I don't know if that's compatible with a scalar. We may need to add a separate value for the count of built-in engines, the count of user-installed engines, the count of hidden one-offs.

Comment 4

[Dave Zeber [:dzeber]]

(In reply to Florian Quèze [:florian] [:flo] from comment #3)
> I don't know if that's compatible with a scalar. We may need to add a
> separate value for the count of built-in engines, the count of
> user-installed engines, the count of hidden one-offs.

Ah yes, I was thinking of this (https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/collection/scalars.html#keyed-scalars), but it does say separate scalars are more appropriate.

Could we do counts for all combinations: builtin/visible, builtin/hidden, user-added/visible, user-added/hidden? IIUC, the total count of all visible plugins == the number of one-off buttons in the search dropdown.

Comment 5

[Javaun Moradi [:javaun]]

Dave chimed in, I'm removing the NI.

Comment 6

[Panos Astithas (he/him) [:past] (please ni?)]

Since this probe is already gone in 55 and we established that we need to keep both the old probe and the new proposed one for at least a while, let's use this bug to just extend the lifetime of the existing probe and then file a followup for the better measurement.

Comment 7

[Florian Quèze [:florian] (on leave until October 9)]

Attachment: bump the max version

Per the discussion in the previous comments, it seems we want to keep this probe for another few release to give us time to implement a more detailed permanent probe, and compare the results of the two. So I'm bumping the expires_in_version value by 5 releases, and requesting a data review.

Comment 8

[[e-mail]]

Comment on attachment 8847625 [details] [diff] [review]
bump the max version

Given dzeber's answers to the above questions, extending the collection for an additional 5 versions is fine.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/59baa83a1f66
bump the max version of the SEARCH_SERVICE_ENGINE_COUN probe, r=rweiss.

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/59baa83a1f66

Comment 11

[Petruta Horea [:phorea], Desktop QA]

Verified that the telemetry collection https://mzl.la/2tQLi5Z contains results for SEARCH_SERVICE_ENGINE_COUNT probe also for Nightly 56 so I can confirm that it hasn't expired in 55.

If there are other ways we could manually verify this, please let us know.