1560590 - Add Telemetry probes to Picture-in-Picture to measure usage

Status: RESOLVED FIXED

(Toolkit :: Video/Audio Controls, task, P2)

Description

[Mike Conley (:mconley) (:⚙️)]

We'd like to answer a few questions by landing some Telemetry probes. Those questions are:

1. Is anybody using Picture-in-Picture?
2. How long is the Picture-in-Picture window being left open for?
3. Does the default placement and size of the Picture-in-Picture window make sense, or are people moving it and resizing it after it opens?
4. How are users accessing Picture-in-Picture? Via the toggle, or via the context menu?
5. How are users closing the Picture-in-Picture window? By clicking X? By clicking the "Unpip" button? By closing the originating tab? By closing the originating window? By shutting down? Some other mechanism (right-clicking on the task bar entry and choosing "close" perhaps... not sure if we can detect that)

This is just a list off of the top of my head, but I'm going to double-check with adamopenweb - there might be some questions here that we actually don't care about, or there might be questions that aren't represented here.

Comment 1

[Mike Conley (:mconley) (:⚙️)]

Hey adamopenweb, any suggestions for additions, removals, or corrections to the above questions?

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a task, but please change it back in case of error.

Comment 3

[Adam Stevenson [:adamopenweb]]

Thanks Mike. Just to clarify (maybe obvious), I'm assuming we want to get a sense of the general usability of the feature in the current state? With data pointing at how to improve our implementation. And that we would consider general discover-ability to be out of scope, as that would require multiple designs and cohorts.

Your questions look great, a few more and expanding on your points:

• How often do they use it? (plays / time window)
• Can we get the time watched / total video length?
• Can we detect the ratio of pip size to desktop screen resolution?
• Is this more useful for short content or long form?
• Can we detect how often live streams are played?

Comment 4

[Andreas Bovens [:abovens]]

All of these questions look good to me.

Comment 5

[Mike Conley (:mconley) (:⚙️)]

Attachment: Bug 1560590 - Add a Telemetry probe to measure how long the Picture-in-Picture window is kept open for. data-review=chutten,r?JSON_Voorhees

Comment 6

[Mike Conley (:mconley) (:⚙️)]

Attachment: Bug 1560590 - Add Telemetry for Picture-in-Picture opening and closing mechanisms. data-review=chutten,r?JSON_Voorhees

Depends on D36358

Comment 7

[Mike Conley (:mconley) (:⚙️)]

Attachment: Bug 1560590 - Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r?JSON_Voorhees

Depends on D36359

Comment 8

[Mike Conley (:mconley) (:⚙️)]

Attachment: Data review request

Comment 9

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 9074843 [details]
Bug 1560590 - Add a Telemetry probe to measure how long the Picture-in-Picture window is kept open for. data-review=chutten,r?JSON_Voorhees

Hi! Adding some probes here - do you have time to do a data review? I've attached the data review form to this bug as an attachment.

Comment 10

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 9074844 [details]
Bug 1560590 - Add Telemetry for Picture-in-Picture opening and closing mechanisms. data-review=chutten,r?JSON_Voorhees

Please see comment 9.

Comment 11

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 9074845 [details]
Bug 1560590 - Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r?JSON_Voorhees

Please see comment 9.

Comment 12

[Mike Conley (:mconley) (:⚙️)]

Hey chutten,

I'm still having trouble with this last set of probes using Event Telemetry in https://phabricator.services.mozilla.com/D36360. Even when I try to manually add an event using the Browser Console, I'm not seeing any information show up in about:telemetry (nor the raw JSON dump). Any ideas what I'm doing wrong?

Comment 13

[Chris H-C :chutten]

Turns out this is a bug on our side. A flag got flipped back during the Fluent rewrite of about:telemetry and we didn't notice (to our defence, JavaScript didn't tell us that it was referencing something that no longer exists).

bug 1562947

I'll get that fixed for you real quick.

Comment 14

[Mike Conley (:mconley) (:⚙️)]

Hey JSON_Voorhees, just a heads up that you'll need to apply the fix in bug 1562947 locally if you want to test the Event Telemetry I just added.

Comment 15

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 9074843 [details]
Bug 1560590 - Add a Telemetry probe to measure how long the Picture-in-Picture window is kept open for. data-review=chutten,r?JSON_Voorhees

Redirecting, since Megan's on PTO.

Comment 16

[Chris H-C :chutten]

Comment on attachment 9075370 [details]
Data review request

Reflagging to appease my sense of order in the universe.

Comment 17

[Mike Conley (:mconley) (:⚙️)]

Whoops - my bad on the feedback? flags - seems like we need to update https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html to refer to the data-review flags.

Comment 18

[Mike Conley (:mconley) (:⚙️)]

Filed bug 1562977 to get the documentation updated.

Comment 19

[Chris H-C :chutten]

Comment on attachment 9075370 [details]
Data review request

Preliminary notes:

That is an excellent list of data collections with a lovely level of detail.

A note about Q8: it is default-on for prerelease users. The opt-out mechanism is the data sharing preference in Firefox's Preferences.

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions files [Histograms.json](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Histograms.json), [Scalars.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Scalars.yaml), and [Events.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Events.yaml) and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire in Firefox 74.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for pre-release channels only.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

Yes. :mconley is responsible for renewing or removing the collection before it expires in Firefox 74.

---
Result: datareview+

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6dfcbc4b9265
Add a Telemetry probe to measure how long the Picture-in-Picture window is kept open for. data-review=chutten,r=JSON_voorhees
https://hg.mozilla.org/integration/autoland/rev/d2fb97301ba3
Add Telemetry for Picture-in-Picture opening and closing mechanisms. data-review=chutten,r=JSON_voorhees

Comment 21

[Mike Conley (:mconley) (:⚙️)]

Hey JSON_Voorhees, if you're looking to test the code for that last patch still waiting on review, here's a try build with all 3 patches applied:

Win64: https://queue.taskcluster.net/v1/task/MxnmdbZnSMeBAa6MNoqQaw/runs/0/artifacts/public/build/target.zip

Comment 22

[Mike Conley (:mconley) (:⚙️)]

Win32: https://queue.taskcluster.net/v1/task/VAn4YVwEQKiBQHwrNVJHUw/runs/0/artifacts/public/build/target.zip

Comment 23

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/6dfcbc4b9265
https://hg.mozilla.org/mozilla-central/rev/d2fb97301ba3

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f92ce352e697
Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r=JSON_voorhees

Comment 25

[Oana Pop-Rus]

Backed out changeset f92ce352e697 (bug 1560590) for ESlint failure

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=f92ce352e697926f1cf5be4fb686310e07ad1e38&selectedJob=255559634

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=255559634&repo=autoland&lineNumber=292

Backout: https://hg.mozilla.org/integration/autoland/rev/005bc170af45995d2cc0099c567bc4cd0d1c16c2

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/29ed46b19db3
Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r=JSON_voorhees

Comment 27

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/29ed46b19db3

Comment 28

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 9074845 [details]
Bug 1560590 - Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r?JSON_Voorhees

Beta/Release Uplift Approval Request

• User impact if declined: None directly, though we'll have more difficulty making data-driven decisions about the Picture-in-Picture feature.

Note that we don't need beta approval for the other two patches in this bug, as they were landed before the merge.

• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This adds a new Telemetry probe for the Picture-in-Picture feature that we're testing on Beta and Developer Edition. The patch is simple, and passive in nature, and neither our automated tests nor manual testing has revealed any problems with it.
• String changes made/needed: None.

Comment 29

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9074845 [details]
Bug 1560590 - Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r?JSON_Voorhees

Adds telemetry for the Picture-in-Picture feature being tested on Beta during the 69 cycle. Low risk since this feature isn't riding the trains to release. Note that only this specific patch requires uplift as the others landed on 69 prior to the merge to Beta. Approved for 69.0b4.

Comment 30

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/d2d91301a2a0