Closed Bug 1236642 Opened 4 years ago Closed 4 years ago

Adding telemetry data

Categories

(Firefox OS Graveyard :: B2gInstaller, defect)

ARM
Gonk (Firefox OS)
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gerard-majax, Assigned: gerard-majax)

References

Details

(Whiteboard: [systemsfe])

Attachments

(8 files, 2 obsolete files)

I think we should make sure we can track how the addon is being used. Thus, we could leverage telemetry and track:
 - which device got flashed (model)
 - which build got flashed
 - which device got plugged (can help us discover what people want)
 - success/failures of flashing

Any idea to add?
Flags: needinfo?(pdolanjski)
Flags: needinfo?(jlorenzo)
Flags: needinfo?(dale)
I'd also add:
- What was the original build? Android/Firefox OS? If FxOS, what were the build details (see list above).
- What version of b2g-installer and adb helper were used?
- What version of Firefox is used (might already be shared)

> - which build got flashed
Just to confirm, what information do we store here? Something like:
* automatically vs manually downloaded
* userdebug/eng
* buildId
* gecko/gaia/gonk revisions
* taskcluster taskId (if downloaded automatically)?
Flags: needinfo?(jlorenzo)
Ones listed seem like a good call
Flags: needinfo?(dale)
Dominik, can we re-use anything from the reporting used for FTU/app usage for this?  (if not the client side, perhaps the back-end)
Flags: needinfo?(pdolanjski) → needinfo?(dstrohmeier)
Whiteboard: [systemsfe]
We had similar discussions around the FOTA ping (https://bugzilla.mozilla.org/show_bug.cgi?id=1202970). Similar to the FOTA ping, the requirements sound like an adaptation of the FTU ping. There, we collect already device model, buildID...

However, remembering discussion on IRC, this is meant to be a Desktop Browser addon (and please correct me if I am wrong). So the question is how to best expose the information from the device to the browser addon - not necessarily about how to get that information from a server, right?
Flags: needinfo?(dstrohmeier)
ro.cm.device would be a good thing to check, to easily see how many devices already with CM are used
for clarity: CM devices usually spoof other data to look like retail devices - a hack necessary for Play Store to continue working correctly.
After asking on #telemetry, it looks like what we should be making use of is external ping in telemetry. This will require LDAP access to be able to analyze the data, but it allows us to send more arbitrary data.

From the requirements above, I think we should add two external pings: one for device connection to report use, help discover device detection failures and try to detect devices people want to try ; one for device flashing procedure to report use and detect failures.
For device connection ping, we would need to collect:
 - was the device detected as supported?
 - device bootloader
 - device buildid (Android release, like LMY48I)
 - manufacturer
 - device model
 - ro.cm.device value to detect CM devices
Attached file WIP PR
(In reply to Alexandre LISSY :gerard-majax from comment #8)
> For device connection ping, we would need to collect:
>  - was the device detected as supported?
>  - device bootloader
>  - device buildid (Android release, like LMY48I)
>  - manufacturer
>  - device model
>  - ro.cm.device value to detect CM devices

At first, usage of those collected data would only be limited to track addon usage: how many devices flashed successfully, how many failed, how many (and which) unsupported devices got tried. Those generic aggregated values could be exposed publicly, if it makes sense (like a generic status report of the use of the addon).

Depending on the failures, local private exploration might be needed to track potential bugs.
For the flashing ping, I think we would need:
 - flashing success or failure (and if failure, which step failed)
 - device model/manufacturer, bootloader and buildid
 - build flashed: URL for a supported build, or just boolean identifying the use of a local zip file
 - device was running B2G or Android
 - user data kept

This should be enough to track use of the addon and have an overview of what user do flash on which device.

As previously, some of those data might make sense in publicly accessible data report/dashboard, but only on an aggregated basis: number of successfull flash performed (with per device model breakdown?).

Knowing how many devices got flashed from Android or B2G and/or if user data were kept is more useful for studying addon usage.

Knowing which builds gets flashed could get publicized as aggregated data.
In both case, device connected ping or device flashed ping, only generic aggregated values are useful to be made available as dashboard and/or public data exports.
Attachment #8717881 - Flags: review?(benjamin)
Comment on attachment 8717881 [details]
data collection changes

I am the data steward for Firefox. I am not the right person to approve data collection related to Firefox OS or other projects for the connected devices group. The data steward for that organization is Dominik Strohmeier.
Attachment #8717881 - Flags: review?(benjamin)
Attachment #8717881 - Flags: review?(dstrohmeier)
Note that this a Firefox addon and this data would be submitted through Firefox Desktop Telemetry.
I'm unclear how this aligns with the different project ownerships, but it seems to fall into the Firefox data?
Flags: needinfo?(benjamin)
No, unless we ship this addon with Firefox it is separate. The addon may need to have its own privacy policy somewhere, in which case Dominik will need to work with Marshall Erwin.
Flags: needinfo?(benjamin)
Can someone give me context regarding this addon?  What is it?  Is it part of the connected devices workstream? To my knowledge, this is not the type of thing we would collect through telemetry unless, as smedberg suggests, it ships with Firefox.
(In reply to Merwin from comment #17)
> Can someone give me context regarding this addon?  What is it?  Is it part
> of the connected devices workstream? To my knowledge, this is not the type
> of thing we would collect through telemetry unless, as smedberg suggests, it
> ships with Firefox.

It is an addon to easily and legally install B2G on a supported device. See https://bugzilla.mozilla.org/show_bug.cgi?id=1166276 and https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Building_and_installing_Firefox_OS/B2G_installer_add-on
Just notes to be able to test telemetry:
 - a local build must be done with MOZILLA_OFFICIAL=1 and MOZ_TELEMETRY_REPORTING=1
 - get gzipServer from https://github.com/vdjeric/gzipServer
 - follow the README to change toolkit.telemetry.server pref

Then TelemetryController.testPing() should go to your local telemetry server.
Attached file connect_z1c_l.json
Connecting an unsupported Z1 Compact device
Attached file connect_nexus4_kk.json
Connecting a supported Nexus 4
Attached file connect_z3c_kk.json
Connecting a supported Z3 Compact
Failure with a Nexus 4
Successfull flash of a Nexus 4
Comment on attachment 8717845 [details] [review]
WIP PR

Should be good to review now :)
Attachment #8717845 - Flags: review?(dale)
Note: Once this had data steward review and is landing, you need to file a bug in "Cloud Services :: Metrics Pipeline" to white-list the ping type (otherwise it will be sorted into a bucket of "other" pings).
You can flag mreid on that bug.
Comment on attachment 8717881 [details]
data collection changes

I have some questions after reviewing this:

Is this connected to a train in Connected Devices or where does the flashing part live right now? If I understand correctly, this is an addon that will live in Firefox Desktop and allow users to flash devices from Android of B2G to a later version of B2G. Correct?

Further regarding data collection:

1) How will this data be collected? Will the user be notified about the data collection? If yes, when?
2) Is it default on or off, i.e. will the user be given a choice to share this data with Mozilla?

The collected information makes sense to me from the engineering perspective and fulfills the requirements of our lean data policy. It is like a modified activation ping including information specific to flashing. However, I don't see a need to make that collection a 'default on' set of pings right now.

How do you plan to link the information collected in the two pings? Is there a need to identify which pings result from the same flashing process, i.e. are you collecting any identifier in the pings?

It would be great to get some description of when data gets send to us in several cases:

1) successfully connecting and flashing a device
2) connecting, but flashing fails
3) connecting fails
4) any of the action get canceled

I think that it might make sense to combine the pings and send the joined version only if flashing was successful and upon user consent. In terms of failing, user will be given choice to submit data to us about the device connected, but not supported (for connection ping) or about failure (in flashing ping), kind of similar UX like a crash report.

Another question could be: How can we make use of the data which gets collected from the FTU ping after flashing which is default on to support the requirements of these pings? Will FTU pings be enough in case of successfully connecting and flashing a device?

Who is the intended non-public audience? I assume that contributors to the flashing community who are not staff want to have access? 

@Marshall: Do you agree with those points?
Flags: needinfo?(merwin)
Flags: needinfo?(lissyx+mozillians)
(In reply to Dominik Strohmeier from comment #27)
> Comment on attachment 8717881 [details]
> data collection changes
> 
> I have some questions after reviewing this:
> 
> Is this connected to a train in Connected Devices or where does the flashing
> part live right now? If I understand correctly, this is an addon that will
> live in Firefox Desktop and allow users to flash devices from Android of B2G
> to a later version of B2G. Correct?

Correct. I cannot tell any train this is part of right now, but this was started months ago to empower more community people to flash devices legally and easily. Given we still have plans to sunset the smartphone and the addon was mostly ready, I decided to finish it. It was not done earlier because of a lot of reasons I won't explain here.

> 
> Further regarding data collection:
> 
> 1) How will this data be collected? Will the user be notified about the data
> collection? If yes, when?

It's all making use of the current telemetry/ping infrastructure. There is no notification, but we can add a banner if needed.

> 2) Is it default on or off, i.e. will the user be given a choice to share
> this data with Mozilla?

It's sending pings using Telemetry, so if telemetry has been enabled by the user, then it's on.

> 
> The collected information makes sense to me from the engineering perspective
> and fulfills the requirements of our lean data policy. It is like a modified
> activation ping including information specific to flashing. However, I don't
> see a need to make that collection a 'default on' set of pings right now.

The goal is to be able to know if the addon is useful and how much it is being used.

> 
> How do you plan to link the information collected in the two pings? Is there
> a need to identify which pings result from the same flashing process, i.e.
> are you collecting any identifier in the pings?

I don't care about linking both information. Each ping contains enough to be useful on its own (we repeat data we need).

I am not sending any identifier at all in my payload.

You can have a look at JSONs in attachments:
 - attachment 8718351 [details]
 - attachment 8718350 [details]
 - attachment 8718349 [details]
 - attachment 8718348 [details]
 - attachment 8718347 [details]

> 
> It would be great to get some description of when data gets send to us in
> several cases:
> 
> 1) successfully connecting and flashing a device
> 2) connecting, but flashing fails
> 3) connecting fails
> 4) any of the action get canceled

See above.

> 
> I think that it might make sense to combine the pings and send the joined
> version only if flashing was successful and upon user consent. In terms of
> failing, user will be given choice to submit data to us about the device
> connected, but not supported (for connection ping) or about failure (in
> flashing ping), kind of similar UX like a crash report.
> 
> Another question could be: How can we make use of the data which gets
> collected from the FTU ping after flashing which is default on to support
> the requirements of these pings? Will FTU pings be enough in case of
> successfully connecting and flashing a device?

We currently have no way to easily and reliably track flashing result, as documented in bug 1191316.

I guess we can just rely on FTU ping being sent (but what about bug 1236439 ?)

> 
> Who is the intended non-public audience? I assume that contributors to the
> flashing community who are not staff want to have access? 

For the moment, the intended non-public audience would be only me. I don't know the process to grant access to such data collections for community members?
Flags: needinfo?(lissyx+mozillians)
To summup and make it clear: all of this relies on Firefox' Telemetry. It is my understanding that |submitExternalPing()| will do nothing if the user has not enabled telemetry.

If he did enabled telemetry, then we should be clear? We would maybe just need to document exactly what we are sending?
Flags: needinfo?(dstrohmeier)
Comment on attachment 8717845 [details] [review]
WIP PR

Codewise this looks perfectly good to me, I cant comment on the data we collect or how we collect it so leaving review of that to telemetry folks
Attachment #8717845 - Flags: review?(dale) → review+
Attached image Capture du 2016-02-15 14-03-29.png (obsolete) —
UX would look like this
Attachment #8719468 - Flags: review?(dstrohmeier)
Flags: needinfo?(dstrohmeier)
Attachment #8717881 - Flags: review?(dstrohmeier)
Attachment #8719468 - Flags: review?(dstrohmeier)
Attached image Capture du 2016-02-15 18-55-16.png (obsolete) —
Here is a more final wording. The link points to https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Building_and_installing_Firefox_OS/B2G_installer_add-on#Data and there is description of what precisely is being collected.
Attachment #8719468 - Attachment is obsolete: true
Attachment #8719539 - Flags: review?(dstrohmeier)
(In reply to Alexandre LISSY :gerard-majax from comment #32)
> Created attachment 8719539 [details]
> Capture du 2016-02-15 18-55-16.png
> 
> Here is a more final wording. The link points to
> https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/
> Building_and_installing_Firefox_OS/B2G_installer_add-on#Data and there is
> description of what precisely is being collected.

Dominik, I changed a bit the string as https://github.com/mozilla-b2g/b2g-installer/pull/38/files#diff-37065867b0a1a2b428343e5d059c99ddR51

I hope it's good for you.
Attachment #8717881 - Flags: review?(dstrohmeier)
Comment on attachment 8719539 [details]
Capture du 2016-02-15 18-55-16.png

I would recommend to avoid the work "track(ing)". What about:

<b>Help us to improve the addon</b>: By sending anonymous usage data, we will be able to see defects in the addon quickly (device not recognized, flashing failures) and learn about new devices to do ports on. More details are provided on B2G Installer MDN page.
Flags: needinfo?(merwin) → needinfo?(lissyx+mozillians)
Attachment #8719539 - Flags: review?(dstrohmeier) → review-
Comment on attachment 8717881 [details]
data collection changes

After working with Alexandre and Georg on this, let me summarize and then ask Marshall to comment on that, too:

The above data pings will be submitted through the existing Telemetry infrastructure as an external ping. To do this, there are two gates for sending this:

1) The first is that "Basic Telemetry (aka FHR)" has to be enabled. This is a default-on option so shouldn't be a blocker.
2) There will be an opt-in in the addon which will ask the user to share anonymous usage data - see attached screenshot to this bug. There will be more information on the MDN page https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Building_and_installing_Firefox_OS/B2G_installer_add-on#Data

The payload is shown on the MDN page, too, for transparency.

To me, there is no sensitive data in the payload for neither of the two pings. All data is anonymous and also fulfills the requirements of our lean data practices.

One requirement which I will raise here is that I want to see a concept for an open data dashboard regarding the data collected in the two pings soon after launch. I recommend to think about integrating the dashboard to the MDN site.

Flagging this for Marshall to comment.
Flags: needinfo?(lissyx+mozillians) → needinfo?(merwin)
Attachment #8717881 - Flags: review?(dstrohmeier) → review+
Attached file Optin final string
Here is the new string :)
Attachment #8719539 - Attachment is obsolete: true
Attachment #8719751 - Flags: review?(dstrohmeier)
Attachment #8719751 - Flags: review?(dstrohmeier) → review+
Blocks: 1248673
Dominik and I just talked through this. It's good to go from my perspective.
Flags: needinfo?(merwin)
Pings have been whitelisted server-side in bug 1248673. This can land.
https://github.com/mozilla-b2g/b2g-installer/commit/abe6d2e14adfc6df46d3d829fb3ee90e548afa74
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.