1362520 - Derived dataset request: plugin-related tracking for click-to-play

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement)

Description

[Benjamin Smedberg]

One of the key metrics for rollout of click-to-play plugins is the rates of a few plugin metrics:

PLUGINS_NOTIFICATION_SHOWN
PLUGINS_NOTIFICATION_USER_ACTION
PLUGINS_INFOBAR_SHOWN
PLUGINS_INFOBAR_BLOCK
PLUGINS_INFOBAR_ALLOW

For the most part, all I care about are aggregate rates of various metrics:

ratio between the various PLUGINS_NOTIFICATION_USER_ACTION and PLUGIN_NOTIFICATION_SHOWN
If you count showing the infobar as an "error", a MTBF (requires subsession length aggregate, bug 1362474)
ratio of plugins_infobar_shown and the _block and _allow resolutions
(also might consider plugins_infobar_shown / total_uri_count)

But the final few are based on users:

% of active users each week who see plugin_notification
% of active users each week who see plugins_infobar_shown

So I have two inter-related asks:

#1: I'd like a dataset/aggregation for the "% of active users" queries
  doing this approximately using hyperloglog is fine

#2: I can do prerelease rates using the telemetry v2 API, but I need to monitor this for release users on 55, so I would like a way to do that efficiently

Requested timeframe:

For #1, I can do that on nightly "by hand" using ATMO, so it would be good to have a solution in place when this rides to beta on 15-June
For #2, we'd talking 55 release so 8-August

Permanence: we will not be monitoring this permanently. We'll be looking at this through the 55/56 release timeframe. So I'm not sure whether adding these to main-summary is the right solution.

Rollout of CTP plugins is considered a quantum blocker, so I'd like this treated with fairly high priority. I'd like to know by next week whether this ask is reasonable/what timeframe is possible.

Comment 1

[William Lachance (:wlach)]

I can take a look at this. I've talked a bit with :mdoglio about how to go about it, though I don't understand what's needed to solve the problem well enough just at the moment to know exactly how long it will take me. It certainly doesn't seem that big a set of tasks though (my guess would be <2 weeks, depending on how much time I spend on this vs. other things).

Comment 2

[William Lachance (:wlach)]

So I think I managed to get a count of nightly users that are seeing the plugin_notification using hyperloglog inside a zeppelin notebook:

https://www.zeppelinhub.com/viewer/notebooks/aHR0cHM6Ly9wZW9wbGUtbW96aWxsYS5vcmcvfndsYWNoYW5jZS9wbHVnaW4lMjBkaWFsb2cuanNvbg

(not sure if that's the optimal way of expressing this, I'm still pretty new to spark)

So if I understand the ask above, all we need to do is:

* Get the ratio by dividing the above total by total unique users
* Do the same for plugins_infobar_shown
* Calculate this on a weekly basis (or just keep a 7 day moving average?)

I'll try to get some guidance from mdoglio on how to go about this.

Comment 3

[Benjamin Smedberg]

wlach, I was thinking that we could have something equivalent to main_summary for this data. I figured adding these columns to main_summary itself wouldn't be appropriate because this is just temporary, but perhaps I'm wrong? That would allow me to do STMO/redash queries without further assistance. But other ways of solving the problem are also fine.

One additional note about rollout monitoring, from bug 1365714: I need to know which users have the new setting. Since this will be rolled out via pref flips, is pref-flip experiment data already contained in the main summary or another summary table? If not, I'll need that as a correlate in the table. I'm happy to file a separate bug about that if appropriate.

Comment 4

[Mark Reid [:mreid]]

I think adding these fields to main_summary is fine.

It looks like all of them except PLUGINS_NOTIFICATION_USER_ACTION should just be integers representing the total count (should they be split out by process?).

What would be the best representation for PLUGINS_NOTIFICATION_USER_ACTION? Prior art in main_summary would add this data as a field with a nested struct for each expected enum value.

Comment 5

[Benjamin Smedberg]

> It looks like all of them except PLUGINS_NOTIFICATION_USER_ACTION should
> just be integers representing the total count (should they be split out by
> process?).

No (these are all chrome-only metrics)

> What would be the best representation for PLUGINS_NOTIFICATION_USER_ACTION?
> Prior art in main_summary would add this data as a field with a nested
> struct for each expected enum value.

Either a nested tuple with three numbers or splitting it out explicitly into PLUGINS_NOTIFICATION_USER_ACTION_allownow PLUGINS_NOTIFICATION_USER_ACTION_allowalways PLUGINS_NOTIFICATION_USER_ACTION_block would be fine from my perspective.

Comment 6

[William Lachance (:wlach)]

(In reply to Benjamin Smedberg [:bsmedberg] from comment #3)

> One additional note about rollout monitoring, from bug 1365714: I need to
> know which users have the new setting. Since this will be rolled out via
> pref flips, is pref-flip experiment data already contained in the main
> summary or another summary table? If not, I'll need that as a correlate in
> the table. I'm happy to file a separate bug about that if appropriate.

There's an active_experiment_id field here:

https://github.com/mozilla/telemetry-batch-view/blob/381c15c7e1e4beff6aeca8d36e5966b86c3648d6/src/main/scala/com/mozilla/telemetry/views/MainSummaryView.scala#L642

Is that what you're looking for?

Comment 7

[William Lachance (:wlach)]

Attachment: PR

PR to add relevant information to main summary

Comment 8

[Benjamin Smedberg]

> There's an active_experiment_id field here:
> 
> https://github.com/mozilla/telemetry-batch-view/blob/
> 381c15c7e1e4beff6aeca8d36e5966b86c3648d6/src/main/scala/com/mozilla/
> telemetry/views/MainSummaryView.scala#L642
> 
> Is that what you're looking for?

No. This is the "old" telemetry experiment annotation, it is not the new environment.experiments annotation which is used for all the new pref-flipping work. That was bug 1348748. Alessio are the new annotations included in main summary/other derived datasets?

Comment 9

[Mark Reid [:mreid]]

The new experiment annotations are not there yet, this would be a great thing to add too.

Comment 10

[William Lachance (:wlach)]

(In reply to Mark Reid [:mreid] from comment #9)
> The new experiment annotations are not there yet, this would be a great
> thing to add too.

Filed bug 1366253, I can tackle that after this goes in (just want to make sure I'm not missing something about modifying the main summary aggregation before I do more of it).

Comment 11

[William Lachance (:wlach)]

The PR has landed, is there anything left to do here?

Comment 12

[William Lachance (:wlach)]

Looks like we're good as long as they are no bugs. Marking as fixed and NI'ing bsmedberg to verify per his request.

Comment 13

[Benjamin Smedberg]

I made a dashboard from this data on Monday which was working. Then on Tuesday the data disappeared and I filed bug 1367053. So I suspect this is fine, but I'm not 100% sure yet.