1680856 - Investigate why we can't see any data for the metrics 'MEDIA_CONTROL_PLATFORM_USAGE'

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, task, P3)

Description

[Alastor Wu [:alwu]]

In bug1668139, I've added some shutdown pings which are used to detect the usage of media control on different platforms.

However, when I checked telemetry ping on dashboard, I am not able to see any data for MEDIA_CONTROL_PLATFORM_USAGE.

Comment 1

[Alastor Wu [:alwu]]

Is that possible that I record those ping too late and the telemetry service is already shutdown? Is there any good way I can use to measure this kind of shutdown ping?
Thank you.

Comment 2

[Chris H-C :chutten]

It's possible it is too late. Anything after profile-before-change (actually, anything that happens later than synchronously during the sendTelemetry AsyncShutdown phase) won't get captured in any ping, and thus won't be persisted or sent.

I have a couple of questions about this instrumentation:

1. What should be collected if I'm on Windows but haven't used the media control?
2. How many of these services are created?

Knowing this I can probably help you come up with an instrumentation approach that'll record the data you need.

(( Also, a point of clarification, "pings" are the HTTP payloads we send to the data pipeline. "probes" or "metrics" are the named pieces of telemetry data that you record to. So, more correctly you "added some shutdown metrics". ))

Comment 3

[Alastor Wu [:alwu]]

MediaControlService is a singleton service which would be created at the first time there is any audible media started in Firefo, and would be destroyed when we shutdown XPCOM.

If a user have ever used media control, which means pressing physical or virtual media key to control media, then it would be remembered and report the corresponding result when destorying the service.

If a user never starts any audible media in the single session life, then we won't need to report anything. If a user has ever started audible media, then we would report EnabledOnXXX if a user never uses media key, or report UsedOnXXX if a user has ever used media key.

Hope above explanation could help, so I think I have to find a time before shutdown telemtry service and report the metrics result.

Thank you.

Comment 4

[Chris H-C :chutten]

Ah, well then I think I can help!

One of the larger unstated concepts of data collection in mozilla projects is that the data collection library's the one that's supposed to hold onto the values for you. You shouldn't have to store data in your own component (unless you're doing something exceedingly perf-sensitive, in which case you might want to wait until you're done, and then hand it to us). Instead, as soon as you have the data, you should give it over.

In this case I would recommend setting the EnabledOn immediately on ctor, and setting UsedOn immediately when it's used.

While we're changing how your collection works, I'd like to recommend one other change: I think you would prefer to use Scalars instead of Histograms here. bool-kind Histograms are designed to count the number of true and false values over time within each client so you can compare them like a ratio. That's not what you're doing here, and so you're paying in analysis for a more complicated type than you need. Instead I recommend a bool Scalar for EnabledOn and either a bool Scalar for UsedOn if you only need to know whether it was used, or a uint Scalar if you want to count how often it was used. The data collection review you already did covers however you prefer to collect it, and I think you'd prefer Scalars.

In either case, recording immediately in the ctor and on use means that you'll have some pings with the values and some pings without values (because the values are reset every time a ping is sent). In order to get per-session information, remember to LOGICAL_OR boolean values or SUM numeric values when you GROUP BY info.session_id for per-session analysis or GROUP BY client_id for per-client analysis. Tools like GLAM count the number of values per client for you, so that might be a nice option for you once the data starts flowing.

Hope this helps!

Comment 5

[Alastor Wu [:alwu]]

Thank for the suggestion, I will change my implementation in order to record those metrics earlier.

However, for the type of probe, I think using Histogram can help me better analysis result than using Scalars. The most important things I would like to know from the telemetry result are that

(1) Know the percentage of EnabledOn and UsedOn in single platform
(2) Compare the result of EnabledOn and UsedOn across different platforms

Because each platform has different implementation of virtual control interface, so I would like to know in how those different implementation would affect the usage of media control.

When using Histogram, it can group all those results in one histrogram like this, so I can easy compare them in one single platform (because same platform result would be put next together) and compare them cross platforms (because all results are put in same histogram)

If I use Scalar, you have to create separate scalar for each platform, so I can't compare them together. If I use keyed Scalar, although it can group them in one page result in Measurement Dashboard, but they would be put in the different histogram like this, which can't help me to clearly see the comparison across different platforms. Eg. knowing what platform has the higest usage.

Comment 6

[Alastor Wu [:alwu]]

Hmm, after some thought, I think using Scalar would be probably better than using Histogram.

When using Histogram, all results would be grouped together, so the highest usage platform would definitely be the platform with most usage, which is Windows, but it can't indicate that users like its implementation most.

So separate the result from different platforms into different histogram can eliminate that, because it shows the percentage of EnabledOn and UsedOn, rather than the total amount of count.

Comment 7

[Alastor Wu [:alwu]]

BTW, I would like to ask a question about GLAM. For the probe MEDIA_CONTROL_SETTING_CHANGE, I can see the result in dashboard, but I can't see anything in GLAM. Do you know why?

Thank you.

Comment 8

[Alastor Wu [:alwu]]

Attachment: Bug 1680856 - part1 : report telemetry result when we start enabling or using media control.

Per suggestion [1], report telemetry result eariler when we know the result in order to prevent the result not being captured because the telemetry service is already shutdown at the time we report the result.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1680856#c4

Comment 9

[Alastor Wu [:alwu]]

Attachment: Bug 1680856 - part2 : change telemetry probe type from Histogram to Scalar.

When using Histogram type, all results would be grouped together, so the highest usage platform would definitely be the platform with most usage, which would be Windows, but that can't indicate that users like its implementation most.

So change the probe type to Scalar which has separate histogram for each key type.

Depends on D99150

Comment 10

[Chris H-C :chutten]

(In reply to Alastor Wu [:alwu] from comment #7)

BTW, I would like to ask a question about GLAM. For the probe MEDIA_CONTROL_SETTING_CHANGE, I can see the result in dashboard, but I can't see anything in GLAM. Do you know why?

Thank you.

If you switch to "Major Version" it shows up, so the data's there. My guess would be that there's so little data in each build id that it's not hitting some minimum threshold. Please file a bug and/or reach out on Slack#glam for direct support.

Comment 11

[Alastor Wu [:alwu]]

Attachment: data-review-request

Per suggestion in comment4, I change the probe to Scalar. Thank you.

Comment 12

[Chris H-C :chutten]

Comment on attachment 9192266 [details]
data-review-request

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire in Firefox 91.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does there need to be a check-in in the future to determine whether to renew the data?

Yes. :alwu is responsible for renewing or removing the collection before it expires in Firefox 91.

---

Result: datareview+

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2649cb72ca04
part1 : report telemetry result when we start enabling or using media control. r=bryce
https://hg.mozilla.org/integration/autoland/rev/08d764beb3dd
part2 : change telemetry probe type from Histogram to Scalar. r=bryce

Comment 14

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/2649cb72ca04
https://hg.mozilla.org/mozilla-central/rev/08d764beb3dd