Decide what to do with the SEARCH_SERVICE_ENGINE_COUNT probe

RESOLVED FIXED in Firefox 55

Status

()

Firefox
Search
P1
normal
RESOLVED FIXED
4 months ago
2 months ago

People

(Reporter: florian, Assigned: florian)

Tracking

(Blocks: 1 bug)

54 Branch
Firefox 55
Points:
---
Dependency tree / graph
Bug Flags:
qe-verify +

Firefox Tracking Flags

(firefox55 fixed)

Details

(Whiteboard: [fxsearch])

Attachments

(1 attachment)

(Assignee)

Description

4 months ago
We added this probe in bug 1268424. I think the goal of adding this was to inform UX decisions related to presentation of search plugins in the UI (eg. one-off buttons, search settings).

I'm not sure anybody is looking at the collected data, and this probe is expiring with Firefox 55.

We should either:
- make it permanent (which means cover it with automated tests)
- remove it, if nobody intends to use this data
- expand it, to collect more detailed/useful data.

Here is what the data looks like: https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-25&keys=__none__!__none__!__none__&max_channel_version=release%252F51&measure=SEARCH_SERVICE_ENGINE_COUNT&min_channel_version=null&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-18&table=0&trim=1&use_submission_date=0
Flags: needinfo?(jmoradi)

Comment 1

4 months ago
This probe makes a lot of sense as a part of our search measurement implementation initiative. I think this is a measurement we want to have.

Can we replace it with a Scalar, so that we get the actual non-binned count? We'd want this to be opt-out (calling this out, as it looks like it already is).

Also, can we key it by type of search plugin (built-in or user-installed, and visible one-off or hidden)? I'd be particularly interested in the proportion of user-installed vs. built-in. Presumably, this would also answer the question of how many one-off buttons are shown in the dropdown.
Flags: needinfo?(florian)
(Assignee)

Comment 2

4 months ago
(In reply to Dave Zeber [:dzeber] from comment #1)
> I'd be particularly interested in the
> proportion of user-installed vs. built-in.

I would also be interested in knowing about hidden built-in plugins (when the user clicks "Remove" for a built-in engine, we just hide it).
(Assignee)

Comment 3

4 months ago
(In reply to Dave Zeber [:dzeber] from comment #1)

> Can we replace it with a Scalar, so that we get the actual non-binned count?
> We'd want this to be opt-out (calling this out, as it looks like it already
> is).

I think we should do this, yes. This may require changing the name of the probe.

> Also, can we key it by type of search plugin (built-in or user-installed,
> and visible one-off or hidden)?

I don't know if that's compatible with a scalar. We may need to add a separate value for the count of built-in engines, the count of user-installed engines, the count of hidden one-offs.
Flags: needinfo?(florian)
Whiteboard: [fxsearch]

Comment 4

4 months ago
(In reply to Florian Quèze [:florian] [:flo] from comment #3)
> I don't know if that's compatible with a scalar. We may need to add a
> separate value for the count of built-in engines, the count of
> user-installed engines, the count of hidden one-offs.

Ah yes, I was thinking of this (https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/collection/scalars.html#keyed-scalars), but it does say separate scalars are more appropriate.

Could we do counts for all combinations: builtin/visible, builtin/hidden, user-added/visible, user-added/hidden? IIUC, the total count of all visible plugins == the number of one-off buttons in the search dropdown.
Dave chimed in, I'm removing the NI.
Flags: needinfo?(jmoradi)
Since this probe is already gone in 55 and we established that we need to keep both the old probe and the new proposed one for at least a while, let's use this bug to just extend the lifetime of the existing probe and then file a followup for the better measurement.
Priority: -- → P1
Blocks: 1334599
(Assignee)

Comment 7

3 months ago
Created attachment 8847625 [details] [diff] [review]
bump the max version

Per the discussion in the previous comments, it seems we want to keep this probe for another few release to give us time to implement a more detailed permanent probe, and compare the results of the two. So I'm bumping the expires_in_version value by 5 releases, and requesting a data review.
Attachment #8847625 - Flags: review?(rweiss)
(Assignee)

Updated

3 months ago
Assignee: nobody → florian
Status: NEW → ASSIGNED

Comment 8

3 months ago
Comment on attachment 8847625 [details] [diff] [review]
bump the max version

Given dzeber's answers to the above questions, extending the collection for an additional 5 versions is fine.
Attachment #8847625 - Flags: review?(rweiss) → review+

Comment 9

3 months ago
Pushed by florian@queze.net:
https://hg.mozilla.org/integration/mozilla-inbound/rev/59baa83a1f66
bump the max version of the SEARCH_SERVICE_ENGINE_COUN probe, r=rweiss.

Comment 10

3 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/59baa83a1f66
Status: ASSIGNED → RESOLVED
Last Resolved: 3 months ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 55
Flags: qe-verify+
QA Contact: brindusa.tot
You need to log in before you can comment on or make changes to this bug.