Closed Bug 1521507 Opened 5 years ago Closed 5 years ago

[remote-dbg-next] Telemetry: record runtime updates

Categories

(DevTools :: about:debugging, enhancement, P1)

enhancement

Tracking

(firefox67 fixed)

RESOLVED FIXED
Firefox 67
Tracking Status
firefox67 --- fixed

People

(Reporter: jdescottes, Assigned: jdescottes)

References

(Blocks 2 open bugs)

Details

Attachments

(2 files, 4 obsolete files)

We should record events when the device/runtime list is updated. This should include:

  • when a device is detected,
  • when a new runtime is detected
  • when a runtime disappears
  • when a device disappears
  • when a connection to a runtime is established
  • when a connection to a runtime is lost

The events should give us an idea of how frequently the list of devices is updated, and should also give insights about the type of devices available to the user. I don't think we can record complex properties (arrays of objects? even if easily serializable?).

One option would be to serialize such an object, but it would probably be hard to query and reuse from amplitude. So alternatively (or additionally) we could have as properties:

  • the total number of devices
  • number of devices by type
  • number of devices by runtime type (GeckoView? Fennec?)
  • number of devices by connection status (Connected? Not Connected)

Or we can record one event per device every time the list is updated. E.g. something like device_available, with some basic properties about the device (connection type, runtime, connected...)

Priority: -- → P2
Assignee: nobody → jdescottes
Status: NEW → ASSIGNED
Priority: P2 → P1
Blocks: 1469094
Attached file data_review_1521507.txt (obsolete) —

Harald, before I ask for data review, can you check if you would like more data to be logged alongside those events. For now I just log:

  • connection_type: how the device was connected ("usb", "network" or "wifi")
  • device_name: the manufacturer's device name in case of phones connected via USB (eg "Moto G(4)")
  • runtime_name: the name of the detected runtime (eg "Firefox", "Firefox Nightly")

Thanks!

Attachment #9043327 - Flags: feedback?(hkirschner)

Will ask for r? from janerik once product has approved the data collected here.

Attached file data_review_1521507_v2.txt (obsolete) —

Updated information about the extras logged for device_ events.

Attachment #9043327 - Attachment is obsolete: true
Attachment #9043327 - Flags: feedback?(hkirschner)
Attachment #9043591 - Flags: feedback?(hkirschner)
Attached file data_review_1521507_v3.txt (obsolete) —

Added the runtime id to the list of information collected for runtime events.

Attachment #9043591 - Attachment is obsolete: true
Attachment #9043591 - Flags: feedback?(hkirschner)
Attachment #9043658 - Flags: feedback?(hkirschner)
Comment on attachment 9043658 [details]
data_review_1521507_v3.txt

My take on making this a bit more concrete:

> 1. What questions will you answer with this data?

Which devices are users trying to debug and how?
Which devices are users struggling to connect to?
Is the numbers of developers increasing, that debug GeckoView-based browsers?

> 2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

We can focus on better supporting the devices users actually remote-debug to provide compatibility.
Attachment #9043658 - Flags: feedback?(hkirschner) → feedback-
Attached file data_review_1521507_v4.txt (obsolete) —

Applied suggestions from Harald, requesting data-review.

Attachment #9043658 - Attachment is obsolete: true
Attachment #9043901 - Flags: review?(chutten)
Comment on attachment 9043901 [details]
data_review_1521507_v4.txt

Network host names are not be eligible for default-on collection in release as they can contain arbitrary strings of unknown Category. I recommend removing that collection and resubmitting for review, or designing and having approved a mitigation for the possibility of those names containing personal data.

Please don't hesitate to reach out to me if you have any questions about this.
Attachment #9043901 - Flags: review?(chutten) → review-

Thanks for the review! I kept a runtime_id data collection to allow us to link together events occurring on a given runtime (and the runtime name is not 1/ specific enough, 2/ always available), but this new "runtime_id" is just a random unique id that should not leak any user information (unique for a given about:debugging session).

I will update the patch so that you and jan-erik can check the implementation as well. Let me know if this approach is ok from a data collection perspective.

Attachment #9043901 - Attachment is obsolete: true
Attachment #9044628 - Flags: review?(chutten)
Comment on attachment 9044628 [details]
data_review_1521507_v5.txt

A random id would be perfectly fine from a data collection POV, though I'd love to take a look at the implementation as I give  review.
Attachment #9044628 - Flags: review?(chutten)

(In reply to Chris H-C :chutten from comment #9)

Comment on attachment 9044628 [details]
data_review_1521507_v5.txt

A random id would be perfectly fine from a data collection POV, though I'd
love to take a look at the implementation as I give review.

Sure! flagged you for r? on Phabricator.

Comment on attachment 9044628 [details]
data_review_1521507_v5.txt

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions file ([Events.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Events.yaml)) and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes. Harald Kirschner is responsible (I gather from Comment #5 that he is okay with being named).

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

Yes. It introduces a temporary runtime identifier which is only stable within the Firefox session. It also records the manufacturer-specified device name if the attachment is to a mobile device. Neither of these are of high category or are used in other datasets of high category.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+
Attachment #9044628 - Flags: data-review+
Pushed by jdescottes@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/531c11330f0c
Add telemetry events for runtime update;r=daisuke,janerik,chutten
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 67
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: