1772546 - Add a separate event for PiP windows displayed with subtitles

Status: VERIFIED FIXED

(Toolkit :: Picture-in-Picture, enhancement)

Description

[Ania]

Let's please add this event as a remedy for pictureinpicture.create event not being usable in detecting PiPs launched with subtitles.

Pictureinpicture.subtitles_shown should be recorded only once per PiP window open.

event category: pictureinpicture
event_method: subtitles_shown
extra_keys: webVTTSubtitles: bool

Comment 1

[Ania]

Attachment: PiP Data Collection Request.txt

Initially, we thought we could detect non-webVTTSubs in the pictureinpicture.create event, but that only worked for webVTT tracks. We would like to add a separate event to track subtitles displayed in the PiP window.

Comment 2

[Chris H-C :chutten]

Comment on attachment 9279680 [details]
PiP Data Collection Request.txt

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Ania Safko is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

---

Result: datareview+

Comment 3

[Niklas Baumgardner [:niklas]]

Attachment: Bug 1772546 - Telemetry for when subtitles are shown. r=#pip-reviewers!

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9f7039b0b86c
Telemetry for when subtitles are shown. r=pip-reviewers,mhowell

Comment 5

[Marian-Vasile Laza]

Backed out changeset 9f7039b0b86c (Bug 1772546) as requested by the dev.
Backout link

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/642e6c336f80
Telemetry for when subtitles are shown. r=pip-reviewers,mhowell

Comment 7

[Niklas Baumgardner [:niklas]]

Comment on attachment 9279682 [details]
Bug 1772546 - Telemetry for when subtitles are shown. r=#pip-reviewers!

Beta/Release Uplift Approval Request

• User impact if declined: We want insight into how many people are using subtitles in Picture-in-Picture and this telemetry event along with some other existing telemetry events gives us insight into subtitles in Picture-in-Picture.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: 1. Make sure media.videocontrols.picture-in-picture.display-text-tracks.enabled is set to true

2. Navigate to https://www.youtube.com/watch?v=XdKzUbAiswE
3. Open a PiP window with original video subtitles on and make sure you see subtitles in the PiP window
4. Navigate to about:telemetry, click "Events" and then "Content" or search "pictureinpicture" and confirm that a pictureinpicture.subtitles_shown event exists with "webVTTSubtitles": "false" under extra

Repeat the steps but use https://twitter.com/TheDailyShow/status/1519497244759707649 for the PiP video. The new pictureinpicture.subtitles_shown event should have "webVTTSubtitles": "true" under extra

Repeat the steps with either video but turn the original video subtitles off before opening the PiP window. Confirm that no subtitles are shown in the PiP window and no new pictureinpicture.subtitles_shown event exists. (There may be previous pictureinpicture.subtitles_shown events)

• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch only adds a telemetry event. This has no impact on user experience.
  We would like to include this in the next dot release.
• String changes made/needed: No
• Is Android affected?: No

Comment 8

[Cristina Cozmuta (:CrissCozmuta)]

The backout requested by the dev. was merged to central https://hg.mozilla.org/mozilla-central/rev/364e8e886a3d4179c3fb5f23a31c26bc646550a5

Comment 9

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/642e6c336f80

Comment 10

[Vlad Lucaci, Desktop QA (:vlucaci)]

Hello,

Confirming this issue as verified fixed with WIndows 11, Ubuntu 21 and macOS 11.6.6 on 103.0a1(20220605213032) using the STR from Comment 7.

Comment 11

[Pascal Chevrel:pascalc (PTO until April 26)]

Comment on attachment 9279682 [details]
Bug 1772546 - Telemetry for when subtitles are shown. r=#pip-reviewers!

Approved for 102 beta 5, thanks.

Comment 12

[Pascal Chevrel:pascalc (PTO until April 26)]

https://hg.mozilla.org/releases/mozilla-beta/rev/612dea49066e

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Is the incoming Telemetry looking sane so far?

Comment 14

[Ania]

https://sql.telemetry.mozilla.org/queries/86265/source
I looked at the recorded events from yesterday for my client id, and this one record looks correct.

Niklas and I launched a bunch of PiPs with regular and WebVTT subs today, and we'll check this query tomorrow to see if they have been recorded correctly.

Comment 15

[Vlad Lucaci, Desktop QA (:vlucaci)]

Hello,

Confirming this issue as verified fixed with Windows 11, Ubuntu 21 and macOS 11.6.6 on 102.0b5-build2(20220607212916) using the STR from Comment 7.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9279682 [details]
Bug 1772546 - Telemetry for when subtitles are shown. r=#pip-reviewers!

Approved for 101.0.1.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/d6382e3cb982

Comment 18

[Ania]

Vlad, thank you for confirming.

Niklas and I verified that events are showing up in the telemetry.events table as expected.

We're still verifying one small edge case that you can perhaps help us validating this?
We have a concern that pictureinpicture.create event with webVTTsubtitles:true attribute might be recorded under these circumstances:

1. Navigate to Youtube
2. Find a video that is preceded by a Youtube ad
3. Turn on the subtitles in the ad
4. Launch PiP, wait until the end of the ad, begin watching the video with subs on.

Because we don't do URL tracking, we would wrongly attribute such an event to the popularity of webVTT format outside (as we're assume that Youtube subtitles ARE NOT webVTT).

Comment 19

[Vlad Lucaci, Desktop QA (:vlucaci)]

Attachment: image.png

Hello,

Confirming this issue as verified fixed in 101.0.1 (20220608170832) with macOS 11.6.6, Ubuntu 20.04 and Windows 10x64 , using steps from comment 7.

(In reply to Ania from comment #18)

We're still verifying one small edge case that you can perhaps help us validating this?

Hello Ania,

Using this video with an ad at the beginning, I have managed to uncover that indeed pictureinpicture.create event is present as well as webVTTsubtitles: but with a false attribute.

Hope this helps!