Closed Bug 1345080 Opened 7 years ago Closed 7 years ago

Decide what to do with the SEARCH_SERVICE_ENGINE_COUNT probe

Categories

(Firefox :: Search, enhancement, P1)

54 Branch
enhancement

Tracking

()

VERIFIED FIXED
Firefox 55
Tracking Status
firefox55 --- verified

People

(Reporter: florian, Assigned: florian)

References

(Blocks 1 open bug)

Details

(Whiteboard: [fxsearch])

Attachments

(1 file)

We added this probe in bug 1268424. I think the goal of adding this was to inform UX decisions related to presentation of search plugins in the UI (eg. one-off buttons, search settings).

I'm not sure anybody is looking at the collected data, and this probe is expiring with Firefox 55.

We should either:
- make it permanent (which means cover it with automated tests)
- remove it, if nobody intends to use this data
- expand it, to collect more detailed/useful data.

Here is what the data looks like: https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-25&keys=__none__!__none__!__none__&max_channel_version=release%252F51&measure=SEARCH_SERVICE_ENGINE_COUNT&min_channel_version=null&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-18&table=0&trim=1&use_submission_date=0
Flags: needinfo?(jmoradi)
This probe makes a lot of sense as a part of our search measurement implementation initiative. I think this is a measurement we want to have.

Can we replace it with a Scalar, so that we get the actual non-binned count? We'd want this to be opt-out (calling this out, as it looks like it already is).

Also, can we key it by type of search plugin (built-in or user-installed, and visible one-off or hidden)? I'd be particularly interested in the proportion of user-installed vs. built-in. Presumably, this would also answer the question of how many one-off buttons are shown in the dropdown.
Flags: needinfo?(florian)
(In reply to Dave Zeber [:dzeber] from comment #1)
> I'd be particularly interested in the
> proportion of user-installed vs. built-in.

I would also be interested in knowing about hidden built-in plugins (when the user clicks "Remove" for a built-in engine, we just hide it).
(In reply to Dave Zeber [:dzeber] from comment #1)

> Can we replace it with a Scalar, so that we get the actual non-binned count?
> We'd want this to be opt-out (calling this out, as it looks like it already
> is).

I think we should do this, yes. This may require changing the name of the probe.

> Also, can we key it by type of search plugin (built-in or user-installed,
> and visible one-off or hidden)?

I don't know if that's compatible with a scalar. We may need to add a separate value for the count of built-in engines, the count of user-installed engines, the count of hidden one-offs.
Flags: needinfo?(florian)
Whiteboard: [fxsearch]
(In reply to Florian Quèze [:florian] [:flo] from comment #3)
> I don't know if that's compatible with a scalar. We may need to add a
> separate value for the count of built-in engines, the count of
> user-installed engines, the count of hidden one-offs.

Ah yes, I was thinking of this (https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/collection/scalars.html#keyed-scalars), but it does say separate scalars are more appropriate.

Could we do counts for all combinations: builtin/visible, builtin/hidden, user-added/visible, user-added/hidden? IIUC, the total count of all visible plugins == the number of one-off buttons in the search dropdown.
Dave chimed in, I'm removing the NI.
Flags: needinfo?(jmoradi)
Since this probe is already gone in 55 and we established that we need to keep both the old probe and the new proposed one for at least a while, let's use this bug to just extend the lifetime of the existing probe and then file a followup for the better measurement.
Priority: -- → P1
Per the discussion in the previous comments, it seems we want to keep this probe for another few release to give us time to implement a more detailed permanent probe, and compare the results of the two. So I'm bumping the expires_in_version value by 5 releases, and requesting a data review.
Attachment #8847625 - Flags: review?(rweiss)
Assignee: nobody → florian
Status: NEW → ASSIGNED
Comment on attachment 8847625 [details] [diff] [review]
bump the max version

Given dzeber's answers to the above questions, extending the collection for an additional 5 versions is fine.
Attachment #8847625 - Flags: review?(rweiss) → review+
Pushed by florian@queze.net:
https://hg.mozilla.org/integration/mozilla-inbound/rev/59baa83a1f66
bump the max version of the SEARCH_SERVICE_ENGINE_COUN probe, r=rweiss.
https://hg.mozilla.org/mozilla-central/rev/59baa83a1f66
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 55
Flags: qe-verify+
QA Contact: brindusa.tot
Verified that the telemetry collection https://mzl.la/2tQLi5Z contains results for SEARCH_SERVICE_ENGINE_COUNT probe also for Nightly 56 so I can confirm that it hasn't expired in 55.

If there are other ways we could manually verify this, please let us know.
Status: RESOLVED → VERIFIED
Flags: qe-verify+
You need to log in before you can comment on or make changes to this bug.