Closed Bug 1560590 Opened 3 months ago Closed 2 months ago

Add Telemetry probes to Picture-in-Picture to measure usage

Categories

(Toolkit :: Video/Audio Controls, task, P2)

68 Branch
task

Tracking

()

RESOLVED FIXED
mozilla70
Tracking Status
firefox69 --- fixed
firefox70 --- fixed

People

(Reporter: mconley, Assigned: mconley, NeedInfo)

References

(Blocks 2 open bugs)

Details

Attachments

(4 files)

We'd like to answer a few questions by landing some Telemetry probes. Those questions are:

  1. Is anybody using Picture-in-Picture?
  2. How long is the Picture-in-Picture window being left open for?
  3. Does the default placement and size of the Picture-in-Picture window make sense, or are people moving it and resizing it after it opens?
  4. How are users accessing Picture-in-Picture? Via the toggle, or via the context menu?
  5. How are users closing the Picture-in-Picture window? By clicking X? By clicking the "Unpip" button? By closing the originating tab? By closing the originating window? By shutting down? Some other mechanism (right-clicking on the task bar entry and choosing "close" perhaps... not sure if we can detect that)

This is just a list off of the top of my head, but I'm going to double-check with adamopenweb - there might be some questions here that we actually don't care about, or there might be questions that aren't represented here.

Hey adamopenweb, any suggestions for additions, removals, or corrections to the above questions?

Flags: needinfo?(astevenson)

Bugbug thinks this bug is a task, but please change it back in case of error.

Type: defect → task
Depends on: 1561389

Thanks Mike. Just to clarify (maybe obvious), I'm assuming we want to get a sense of the general usability of the feature in the current state? With data pointing at how to improve our implementation. And that we would consider general discover-ability to be out of scope, as that would require multiple designs and cohorts.

Your questions look great, a few more and expanding on your points:

  • How often do they use it? (plays / time window)
  • Can we get the time watched / total video length?
  • Can we detect the ratio of pip size to desktop screen resolution?
  • Is this more useful for short content or long form?
  • Can we detect how often live streams are played?
Flags: needinfo?(astevenson)

All of these questions look good to me.

Attached file Data review request

Comment on attachment 9074843 [details]
Bug 1560590 - Add a Telemetry probe to measure how long the Picture-in-Picture window is kept open for. data-review=chutten,r?JSON_Voorhees

Hi! Adding some probes here - do you have time to do a data review? I've attached the data review form to this bug as an attachment.

Attachment #9074843 - Flags: feedback?(mmccorquodale)

Comment on attachment 9074844 [details]
Bug 1560590 - Add Telemetry for Picture-in-Picture opening and closing mechanisms. data-review=chutten,r?JSON_Voorhees

Please see comment 9.

Attachment #9074844 - Flags: feedback?(mmccorquodale)

Comment on attachment 9074845 [details]
Bug 1560590 - Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r?JSON_Voorhees

Please see comment 9.

Attachment #9074845 - Flags: feedback?(mmccorquodale)

Hey chutten,

I'm still having trouble with this last set of probes using Event Telemetry in https://phabricator.services.mozilla.com/D36360. Even when I try to manually add an event using the Browser Console, I'm not seeing any information show up in about:telemetry (nor the raw JSON dump). Any ideas what I'm doing wrong?

Flags: needinfo?(chutten)

Turns out this is a bug on our side. A flag got flipped back during the Fluent rewrite of about:telemetry and we didn't notice (to our defence, JavaScript didn't tell us that it was referencing something that no longer exists).

bug 1562947

I'll get that fixed for you real quick.

Flags: needinfo?(chutten)
Attachment #9074843 - Attachment description: Bug 1560590 - Add a Telemetry probe to measure how long the Picture-in-Picture window is kept open for. data-review=mmccorquodale,r?JSON_Voorhees → Bug 1560590 - Add a Telemetry probe to measure how long the Picture-in-Picture window is kept open for. data-review=chutten,r?JSON_Voorhees
Attachment #9074844 - Attachment description: Bug 1560590 - Add Telemetry for Picture-in-Picture opening and closing mechanisms. data-review=mmccorquodale,r?JSON_Voorhees → Bug 1560590 - Add Telemetry for Picture-in-Picture opening and closing mechanisms. data-review=chutten,r?JSON_Voorhees
Attachment #9074845 - Attachment description: Bug 1560590 - Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=mmccorquodale,r?JSON_Voorhees → Bug 1560590 - Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r?JSON_Voorhees

Hey JSON_Voorhees, just a heads up that you'll need to apply the fix in bug 1562947 locally if you want to test the Event Telemetry I just added.

Comment on attachment 9074843 [details]
Bug 1560590 - Add a Telemetry probe to measure how long the Picture-in-Picture window is kept open for. data-review=chutten,r?JSON_Voorhees

Redirecting, since Megan's on PTO.

Attachment #9074843 - Flags: feedback?(mmccorquodale) → feedback?(chutten)
Attachment #9074844 - Flags: feedback?(mmccorquodale) → feedback?(chutten)
Attachment #9074845 - Flags: feedback?(mmccorquodale) → feedback?(chutten)
Attachment #9074843 - Flags: feedback?(chutten)
Attachment #9074844 - Flags: feedback?(chutten)
Attachment #9074845 - Flags: feedback?(chutten)
Comment on attachment 9075370 [details]
Data review request

Reflagging to appease my sense of order in the universe.
Attachment #9075370 - Flags: data-review?(chutten)

Whoops - my bad on the feedback? flags - seems like we need to update https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html to refer to the data-review flags.

Filed bug 1562977 to get the documentation updated.

Comment on attachment 9075370 [details]
Data review request

Preliminary notes:

That is an excellent list of data collections with a lovely level of detail.

A note about Q8: it is default-on for prerelease users. The opt-out mechanism is the data sharing preference in Firefox's Preferences.

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions files [Histograms.json](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Histograms.json), [Scalars.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Scalars.yaml), and [Events.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Events.yaml) and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire in Firefox 74.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for pre-release channels only.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

Yes. :mconley is responsible for renewing or removing the collection before it expires in Firefox 74.

---
Result: datareview+
Attachment #9075370 - Flags: data-review?(chutten) → data-review+
Keywords: leave-open
Pushed by mconley@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6dfcbc4b9265
Add a Telemetry probe to measure how long the Picture-in-Picture window is kept open for. data-review=chutten,r=JSON_voorhees
https://hg.mozilla.org/integration/autoland/rev/d2fb97301ba3
Add Telemetry for Picture-in-Picture opening and closing mechanisms. data-review=chutten,r=JSON_voorhees

Hey JSON_Voorhees, if you're looking to test the code for that last patch still waiting on review, here's a try build with all 3 patches applied:

Win64: https://queue.taskcluster.net/v1/task/MxnmdbZnSMeBAa6MNoqQaw/runs/0/artifacts/public/build/target.zip

Flags: needinfo?(djustice)
Keywords: leave-open
Assignee: nobody → mconley
Pushed by mconley@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f92ce352e697
Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r=JSON_voorhees
Pushed by mconley@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/29ed46b19db3
Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r=JSON_voorhees
Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70

Comment on attachment 9074845 [details]
Bug 1560590 - Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r?JSON_Voorhees

Beta/Release Uplift Approval Request

  • User impact if declined: None directly, though we'll have more difficulty making data-driven decisions about the Picture-in-Picture feature.

Note that we don't need beta approval for the other two patches in this bug, as they were landed before the merge.

  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This adds a new Telemetry probe for the Picture-in-Picture feature that we're testing on Beta and Developer Edition. The patch is simple, and passive in nature, and neither our automated tests nor manual testing has revealed any problems with it.
  • String changes made/needed: None.
Flags: needinfo?(mconley)
Attachment #9074845 - Flags: approval-mozilla-beta?

Comment on attachment 9074845 [details]
Bug 1560590 - Add Telemetry event probes to determine if users tend to move or resize the Picture-in-Picture window after opening it. data-review=chutten,r?JSON_Voorhees

Adds telemetry for the Picture-in-Picture feature being tested on Beta during the 69 cycle. Low risk since this feature isn't riding the trains to release. Note that only this specific patch requires uplift as the others landed on 69 prior to the merge to Beta. Approved for 69.0b4.

Attachment #9074845 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.