Closed Bug 1443528 Opened 3 years ago Closed 3 years ago

Extend Telemetry::AUDIOSTREAM_BACKEND_USED telemetry probe

Categories

(Core :: Audio/Video: cubeb, defect, P2)

defect

Tracking

()

RESOLVED FIXED
mozilla60
Tracking Status
firefox60 --- fixed

People

(Reporter: padenot, Assigned: padenot)

References

Details

Attachments

(1 file)

No description provided.
Summary: Extend Telemetry::AUDIOSTREAM_BACKEND_USED → Extend Telemetry::AUDIOSTREAM_BACKEND_USED telemetry probe
I've unfortunately forgot about doing this earlier, and we have no data on Firefox 60, hopefully there are no weirdness coming up.
Comment on attachment 8956482 [details]
Bug 1443528 - Extend Telemetry::AUDIOSTREAM_BACKEND_USED telemetry probe.

https://reviewboard.mozilla.org/r/225398/#review231312
Attachment #8956482 - Flags: review?(jyavenard) → review+
Pushed by paul@paul.cx:
https://hg.mozilla.org/integration/autoland/rev/f9632d331a58
Extend Telemetry::AUDIOSTREAM_BACKEND_USED telemetry probe. r=jya
Rank: 19
Priority: -- → P2
https://hg.mozilla.org/mozilla-central/rev/f9632d331a58
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
Did this receive Data Collection Review? The extension of a probe is a change in data collection and needs to undergo a quick review.
Flags: needinfo?(padenot)
No. Looking through the commit log, I saw bumps in version without review, so I simply bumped it.
Flags: needinfo?(padenot)
Could you copy-paste the questions from https://github.com/mozilla/data-review/blob/master/request.md then answer them and ni?me? Data review can be granted retroactively when necessary :)
Flags: needinfo?(padenot)
Thanks. Here are some answers, please let me know if I can be more precise.

1) What questions will you answer with this data?
In general:
- What is the proportion of users that have telemetry enabled and are using a particular audio backend
- What is the error rate for opening an audio stream, per platform and version. Is it erroring on the first open, or on a subsequent opening.

More precisely (non exhaustive):
- Is it still useful to maintain the WinXP backend, as a fallback, for windows 7+?
- Are there still people using Alsa on Linux
- Is there a new device/OS/patch level/driver/use case combination that is common enough to show an increased error rate.

2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

Monitoring the field for platform and hardware changes, that have a direct impact to quality/
Monitoring the market for trends, that translate into changes in the quantity of work we put into each backend.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

We haven't considered any alternative method.

4) Can current instrumentation answer these questions?

It's been rather successful so far. We have been able to determine the following:
- There has been an increase in errors on Windows, but it's back to normal. The error rate is always lower on 64 bits compared to 32bits. I think it was because of sandboxing and also some Windows changes. We have also been able to make sure Windows 10 improves the situation, on average: less errors there compared to the previous versions.
- Practically nobody uses Alsa on Linux
- It's still useful to maintain a minimally functioning WINMM backend (in theory useless from Vista onward), because it is being used as a fallback in a non trivial amount of case (single digit percent but still).

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki.

We have only one probe, and it's in the "Technical data" category.

6) How long will this data be collected?  Choose one of the following:

I bumped it for 5 versions. However I think this is useful enough to permanently have the probe. In any case, Matthew Gregan and I are listed in the file describing the probes as being point of contact for this one.

7) What populations will you measure?

No filters.

8) If this data collection is default on, what is the opt-out mechanism for users?

It's opt-in. I'd be a bit happier if it would be opt-out (for obvious statistical bias reasons). It's however not mandatory, the data is useful as it is.

9) Please provide a general description of how you will analyze this data.

Once every couple months), I sit down and look at the following, directly on the interface on https://telemetry.mozilla.org :
- Are the repartitions in backend usage per platform what I expect, when looking at them at the same time as the numbers provided by the Firefox Hardware report.
- Are there any differences or trends per channels per platform, for a current point in time and across releases 
- Is there a trend in error rate and their type

For the above, each time I ask myself whether or not there is an explanation (known bug in an OS/with some hardware, platform changes that can explain an increase or reduce error rate, etc.).

10) Where do you intend to share the results of your analysis?

Quite informally on a bug or on IRC usually, by sharing a specific link to the data.
Flags: needinfo?(padenot) → needinfo?(chutten)
Thank you for your very complete request! Here's the Data Steward review portion:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Telemetry histogram.

    Is there a control mechanism that allows the user to turn the data collection on and off? 

Telemetry upload in preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Not permanent.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1

    Is the data collection request for default-on or default-off?

Telemetry rules.

    Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

Nothing new, this is a renewal.

    Is the data collection covered by the existing Firefox privacy notice?

Covered under the existing notice.

    Does there need to be a check-in in the future to determine whether to renew the data?

Yes.

:padenot, could you file a bug targeting 64 or so to remember to review renewal?

---
ni?liuche to check my work (I'm a trainee data steward at the moment)
Flags: needinfo?(chutten) → needinfo?(liuche)
Blocks: 1444878
> :padenot, could you file a bug targeting 64 or so to remember to review renewal?

bug 1444878. We can't target 64 at the minute.
Awesome. Thank you for going through all this! We (Data Stewards) are looking into making probe renewals significantly quicker this year, so hopefully when we get around to bug 1444878 it'll be a quick rubber-stamp.
Great, I also took a look at bug 1280630, done by bsmedberg, and this looks good. In the future, we're discussing having people re-open the original bug to request probe renewal, but we'll be sure to send out an update.
Flags: needinfo?(liuche)
You need to log in before you can comment on or make changes to this bug.