1815023 - Send minimal install provenance data via Telemetry

Status: RESOLVED FIXED

(Firefox :: Installer, task, P3)

Description

[Robin Steuber (she/her) [:bytesized]]

Using the data that we read into AttributionCode (Bug 1815021), we ought to send some metadata via Telemetry.

Initially, this will be very limited. Likely something like this:

• Whether the file system supports Alternate Data Streams at all, and potentially the file system (e.g. NTFS, FAT, remote SMB/CIFS, other)
• Whether :Zone.Identifier exists, and if so, what ZoneId is
• Whether ReferrerURL exists
• Whether HostURL exists
• Whether “mozilla.org” and other known fragments/prefixes are in ReferrerURL and/or HostURL

Comment 1

[Nick Alexander :nalexander [he/him]]

This might be a separate ticket, but can we also try to send this in at least the full installer ping? (The stub installer ping might not be sensible due to https://bugzilla.mozilla.org/show_bug.cgi?id=1811648.) It may be too annoying to parse what we want in a re-usable manner for use in both contexts, but let's at least investigate.

Comment 2

[Robin Steuber (she/her) [:bytesized]]

(In reply to Nick Alexander :nalexander [he/him] from comment #1)

This might be a separate ticket, but can we also try to send this in at least the full installer ping? (The stub installer ping might not be sensible due to https://bugzilla.mozilla.org/show_bug.cgi?id=1811648.) It may be too annoying to parse what we want in a re-usable manner for use in both contexts, but let's at least investigate.

I was really hoping to avoid this. If we want to read the provenance data from the INI via NSIS, that's relatively straightforward. But making determinations like "Is this a mozilla URL" from NSIS would actually be quite difficult. And I'm not particularly excited about putting the data in "main ping" telemetry and "full installer ping" telemetry using different parsing mechanisms. Potentially we could make some sort of NSIS plugin that could use code that could be shared with Firefox. But it would probably be very difficult to include any existing code from Firefox without including all of XUL.

Is it going to be a problem not to have it in the full installer ping?

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to Robin Steuber (they/them) [:bytesized] from comment #2)

(In reply to Nick Alexander :nalexander [he/him] from comment #1)

This might be a separate ticket, but can we also try to send this in at least the full installer ping? (The stub installer ping might not be sensible due to https://bugzilla.mozilla.org/show_bug.cgi?id=1811648.) It may be too annoying to parse what we want in a re-usable manner for use in both contexts, but let's at least investigate.

I was really hoping to avoid this. If we want to read the provenance data from the INI via NSIS, that's relatively straightforward. But making determinations like "Is this a mozilla URL" from NSIS would actually be quite difficult. And I'm not particularly excited about putting the data in "main ping" telemetry and "full installer ping" telemetry using different parsing mechanisms. Potentially we could make some sort of NSIS plugin that could use code that could be shared with Firefox. But it would probably be very difficult to include any existing code from Firefox without including all of XUL.

Yeah, it's going to be unpleasant. And yes, I was thinking of sharing some C++ via an NSIS plugin.

Is it going to be a problem not to have it in the full installer ping?

How about we plan to do this in stages? We don't know if we're going to get valuable data here. Let's do this at first for just Firefox main pings to see if there's a signal. If there is, we'll consider lifting the functionality into the installers, potentially by rewriting as a shared C++ component.

This delays some work that may not, in fact, be necessary. The downside is that there is some drop-off between "install" and "running Firefox (and submitting a new-profile ping)" that we won't see immediately. But that seems a reasonable trade-off.

Comment 4

[Nick Alexander :nalexander [he/him]]

This will be most helpful if it's in the new-profile ping and the installation.first_seen event, which we're unifying (a bit) over in Bug 1811374.

Comment 5

[Robin Steuber (she/her) [:bytesized]]

Attachment: Bug 1815023 - Add support for sending attribution provenance data via telemetry r=nalexander!

Add a function that sets scalars to convey what kind of data is present in the :Zone.Identifier Alternate Data Stream. Also adds testing that the scalars get set correctly.

Does not actually call the function that sends the telemetry. This will be done in the next patch since we need to call it in some slightly different places than we usually would since we want to be sure it is included in the new-profile ping and in the ping with the installation.first_seen event.

Depends on D171817

Comment 6

[Robin Steuber (she/her) [:bytesized]]

Attachment: Bug 1815023 - Call function to send attribution provenance telemetry r=nalexander!

Make sure the the attribution telemetry gets sent in the new-profile ping and the installation.first_seen event.In other cases, send the telemetry when the browser is idle.

Depends on D171818

Comment 7

[Robin Steuber (she/her) [:bytesized]]

Attachment: Bug 1815023 data review.md

Comment 8

[Travis Long [:travis_]|PTO]

Comment on attachment 9321539 [details]
Bug 1815023 data review.md

Data Review

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, through the Scalars.yaml file and the Probe Dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, through the telemetry preference in the application settings.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

N/A, collection to end in v121 or be renewed

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction Data

5. Is the data collection request for default-on or default-off?

Default-on

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No

7. Is the data collection covered by the existing Firefox privacy notice?

Yes

8. Does the data collection use a third-party collection tool?

No

Result

data-review+

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fc0d63d3404b
Add support for sending attribution provenance data via telemetry r=nalexander,chutten
https://hg.mozilla.org/integration/autoland/rev/e9af7f1b2a01
Call function to send attribution provenance telemetry r=nalexander,chutten

Comment 10

[Robin Steuber (she/her) [:bytesized]]

Attachment: WIP: Bug 1815023 - testProvenance calls should be awaited upon a=testonly

Comment 11

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/fc0d63d3404b
https://hg.mozilla.org/mozilla-central/rev/e9af7f1b2a01

Comment 12

[Phabricator Automation]

Comment on attachment 9322003 [details]
WIP: Bug 1815023 - testProvenance calls should be awaited upon a=testonly

Revision D172074 was moved to bug 1821406. Setting attachment 9322003 [details] to obsolete.