1567604 - Enable VIDEO_PLAY_TIME probe for release channel users as an opt-out probe after it is renewed in bug 1356046 .

Status: RESOLVED FIXED

(Core :: Audio/Video, task, P2)

Description

[Mike Conley (:mconley) (:⚙️)]

Something that tdsmith, astevenson and I were talking about in Whistler was whether or not we can determine if Firefox users watch more video when the Picture-in-Picture feature is enabled.

I started digging into what it would take to add a Telemetry probe to measure video playback time, and was pleased to find out that such a probe already exists - it's just expired, and has been since Firefox 63.

This bug is about reviving that probe to help measure the impact (if any) Picture-in-Picture has on video playback time.

Comment 1

[Mike Conley (:mconley) (:⚙️)]

Attachment: Bug 1567604 - Make VIDEO_PLAY_TIME_MS Telemetry probe opt-out. data-review=bmiroglio,r?dminor

This only enables the probe on Firefox Desktop, and only for content processes. We're
reviving this probe mainly as a way of determining if Picture-in-Picture causes users
to watch more video.

Comment 2

[Jean-Yves Avenard [:jya]]

Check bug 1356046 instead.

However, tbh, I seriously doubt this probe will achieve what you describe. What would you be comparing things with? We have no comparison point and as telemetry isn't available by default in release, we have seen in the past that the numbers are so skewed towards one category of users that it makes any analysis impossible

I agree that determining that information would be great, but reviving that telemetry key alone won't do it.

Comment 3

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

telemetry isn't available by default in release

It doesn't answer your critique, but we should also start collecting this in release if we're interested in it.

Note that probe renewal is subject to a new data-collection review.

Comment 4

[Mike Conley (:mconley) (:⚙️)]

However, tbh, I seriously doubt this probe will achieve what you describe. What would you be comparing things with? We have no comparison point and as telemetry isn't available by default in release, we have seen in the past that the numbers are so skewed towards one category of users that it makes any analysis impossible

Thanks, jya. One way we might do this is to try to break some part of either the Beta or Release populations into two groups, one with PiP enabled, and one without, to see whether or not that reveals a signal.

I'm glad to see that dminor is already working on renewing the probe in bug 1356046! Perhaps I'll repurpose this one for requesting a data-review to getting it enabled to release by default.

Comment 5

[Nils Ohlmeier [:drno]]

Yes lets wait for bug 1356046 to land.

From my point of view we want a playback length probe in Release (because otherwise the values are not reliable) - which could be either this or a new one (see below).

jya: is VIDEO_PLAY_TIME_MS good enough going forward or do want/need to update this probe (or create a new one) which records how many frames got rendered (taking into consideration the frame rate of the material)?

Comment 6

[Jean-Yves Avenard [:jya]]

how long things played is hopefully sufficient

Comment 7

[Mike Conley (:mconley) (:⚙️)]

Attachment: Data review request

Comment 8

[Chris H-C :chutten]

Comment on attachment 9084814 [details]
Data review request

Load balancing to :bmiroglio

Comment 9

[Ben Miroglio [:bmiroglio]]

Comment on attachment 9084814 [details]
Data review request

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way? 
Data will be documented in the [probe dictionary](https://telemetry.mozilla.org/probe-dictionary/). 

2. Is there a control mechanism that allows the user to turn the data collection on and off? 
Yes, this can be turned off by disabling telemetry in the Firefox preferences. 

3. If the request is for permanent data collection, is there someone who will monitor the data over time?
Yes: mconley.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
Category 2 - Interaction Data

5. Is the data collection request for default-on or default-off?
Default on. 

6. Does the instrumentation include the addition of any new identifiers?
No new identifiers. 

7. Is the data collection covered by the existing Firefox privacy notice?
Yes.

8. Does there need to be a check-in in the future to determine whether to renew the data? 
No. 

9. Does the data collection use a third-party collection tool? 
No. 
—
Data review +

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a5784d103862
Make VIDEO_PLAY_TIME_MS Telemetry probe opt-out. data-review=bmiroglio,r=dminor

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/a5784d103862