1521507 - [remote-dbg-next] Telemetry: record runtime updates

Status: RESOLVED FIXED

(DevTools :: about:debugging, enhancement, P1)

Description

[Julian Descottes [:jdescottes]]

We should record events when the device/runtime list is updated. This should include:

• when a device is detected,
• when a new runtime is detected
• when a runtime disappears
• when a device disappears
• when a connection to a runtime is established
• when a connection to a runtime is lost

The events should give us an idea of how frequently the list of devices is updated, and should also give insights about the type of devices available to the user. I don't think we can record complex properties (arrays of objects? even if easily serializable?).

One option would be to serialize such an object, but it would probably be hard to query and reuse from amplitude. So alternatively (or additionally) we could have as properties:

• the total number of devices
• number of devices by type
• number of devices by runtime type (GeckoView? Fennec?)
• number of devices by connection status (Connected? Not Connected)

Or we can record one event per device every time the list is updated. E.g. something like device_available, with some basic properties about the device (connection type, runtime, connected...)

Comment 1

[Julian Descottes [:jdescottes]]

Attachment: data_review_1521507.txt

Harald, before I ask for data review, can you check if you would like more data to be logged alongside those events. For now I just log:

• connection_type: how the device was connected ("usb", "network" or "wifi")
• device_name: the manufacturer's device name in case of phones connected via USB (eg "Moto G(4)")
• runtime_name: the name of the detected runtime (eg "Firefox", "Firefox Nightly")

Thanks!

Comment 2

[Julian Descottes [:jdescottes]]

Attachment: Bug 1521507 - Add telemetry events for runtime update;r=daisuke,janerik

Will ask for r? from janerik once product has approved the data collected here.

Comment 3

[Julian Descottes [:jdescottes]]

Attachment: data_review_1521507_v2.txt

Updated information about the extras logged for device_ events.

Comment 4

[Julian Descottes [:jdescottes]]

Attachment: data_review_1521507_v3.txt

Added the runtime id to the list of information collected for runtime events.

Comment 5

[:Harald Kirschner :digitarald]

Comment on attachment 9043658 [details]
data_review_1521507_v3.txt

My take on making this a bit more concrete:

> 1. What questions will you answer with this data?

Which devices are users trying to debug and how?
Which devices are users struggling to connect to?
Is the numbers of developers increasing, that debug GeckoView-based browsers?

> 2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

We can focus on better supporting the devices users actually remote-debug to provide compatibility.

Comment 6

[Julian Descottes [:jdescottes]]

Attachment: data_review_1521507_v4.txt

Applied suggestions from Harald, requesting data-review.

Comment 7

[Chris H-C :chutten]

Comment on attachment 9043901 [details]
data_review_1521507_v4.txt

Network host names are not be eligible for default-on collection in release as they can contain arbitrary strings of unknown Category. I recommend removing that collection and resubmitting for review, or designing and having approved a mitigation for the possibility of those names containing personal data.

Please don't hesitate to reach out to me if you have any questions about this.

Comment 8

[Julian Descottes [:jdescottes]]

Attachment: data_review_1521507_v5.txt

Thanks for the review! I kept a runtime_id data collection to allow us to link together events occurring on a given runtime (and the runtime name is not 1/ specific enough, 2/ always available), but this new "runtime_id" is just a random unique id that should not leak any user information (unique for a given about:debugging session).

I will update the patch so that you and jan-erik can check the implementation as well. Let me know if this approach is ok from a data collection perspective.

Comment 9

[Chris H-C :chutten]

Comment on attachment 9044628 [details]
data_review_1521507_v5.txt

A random id would be perfectly fine from a data collection POV, though I'd love to take a look at the implementation as I give  review.

Comment 10

[Julian Descottes [:jdescottes]]

(In reply to Chris H-C :chutten from comment #9)

Comment on attachment 9044628 [details]
data_review_1521507_v5.txt

A random id would be perfectly fine from a data collection POV, though I'd
love to take a look at the implementation as I give review.

Sure! flagged you for r? on Phabricator.

Comment 11

[Chris H-C :chutten]

Comment on attachment 9044628 [details]
data_review_1521507_v5.txt

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions file ([Events.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Events.yaml)) and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes. Harald Kirschner is responsible (I gather from Comment #5 that he is okay with being named).

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

Yes. It introduces a temporary runtime identifier which is only stable within the Firefox session. It also records the manufacturer-specified device name if the attachment is to a mobile device. Neither of these are of high category or are used in other datasets of high category.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/531c11330f0c
Add telemetry events for runtime update;r=daisuke,janerik,chutten

Comment 13

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/531c11330f0c