1635656 - Fetch and store Account Ecosystem Telemetry identifiers in FxA profile data

Status: RESOLVED FIXED

(Firefox :: Firefox Accounts, enhancement)

Description

[Ryan Kelly [:rfkelly]]

The Account Ecosystem Telemetry project needs to include a unique user identifier in its telemetry ping, but without being able to link that identifier back to the user's actual Firefox Account. The proposal for obtaining such an identifier is described in detail in Account Ecosystem Telemetry Identifiers and is mostly the responsibility of the Firefox Accounts service. From the perspective of an FxA client application like Firefox the key points are:

• Each user has an ecosystem_user_id which is derived from their FxA master key. It's OK for client code to know this value but it needs to keep it to itself.
• For recording in AET telemetry pings, the ecosystem_user_id must be encrypted to a public key that's managed by the telemetry pipeline, resulting in a value called ecosystem_anon_id. We need this value in order to send AET telemetry pings.
• When the user signs in to their Firefox Account, the FxA services will use their password to unlock the master key, derive ecosytem_user_id, encrypt it to produce ecosystem_anon_id, and store that in their account profile data for us to fetch.

To support all this we need to add the following in Firefox:

• Support for storing new fields ecosystem_user_id and ecosystem_anon_id as part of the persisted account state.
• Support for obtaining ecosystem_anon_id from the FxA server as part of the user's profile data, and refreshing it as part of refreshing the profile data.
• A public API for exposing the ecosystem_anon_id value so that AET can include it in telemetry pings.

The ability to learn the value of ecosystem_user_id is not part of this bug, but I think it's worth including the storage for it as part of this work.

Comment 1

[Ryan Kelly [:rfkelly]]

Attachment: Bug 1635656 - add support for storing and fetching FxA ecosystem telemetry ids. r?lina

Comment 2

[Ryan Kelly [:rfkelly]]

To help with development and testing here, I've created lightly-modified FxA environment that working implementations of the ecosystem-telemetry-related profile endpoints:

https://aet.dev.lcip.org/

To configure Firefox Desktop to use it:

• Visit about:config
• Search for identity.fxaccounts.autoconfig.uri
• Set it to https://aet.dev.lcip.org
• Sign in or create an account as normal (you can check the URL on the signin page to confirm that you're using the custom dev environment)

I've also made a test account and pre-populated its ecosystemAnonId field:

• Email: [e-mail] (you can check email for this account here)
• Password: testtesttest

Comment 3

[Ryan Kelly [:rfkelly]]

:rfkelly to add details about looking for submitted pings in telemetry

Comment 4

[Ryan Kelly [:rfkelly]]

For testing this and related bugs, it's possible to submit pings to the stage AET telemetry pipeline by posting a JSON blob to https://stage.ingestion.nonprod.dataops.mozgcp.net/submit/firefox-accounts/account-ecosystem/1/${UUID}. Data that's posted them will, after some processing delays be available in one of two places.

For data that fails validation or otherwise can't be processed, an error will be created in the moz-fx-data-shar-nonprod-efed.payload_bytes_error.structured table in bigquery. I have a redash query that lists items from this table here. If you include an X-Debug-ID header in that request, it will be copied into the x_debug_id field of that bigquery table, which can help with tracking down specific failed pings.

For data that validates and is successfully processed, it will show up in the moz-fx-data-shar-nonprod-efed.firefox_accounts_live.account_ecosystem_v1 table in biguqery. I have a redash query that lists items from this table here. IIUC items from this table are processed on a daily basis into some longer-lived summary table, but this is the place to look for relatively fast-turnaround debugging and testing.

Comment 5

[Lougenia Bailey]

Attachment: Bug 1635656 - Added tests for fetching FxA ecosystem telemetry ids

Depends on D77762

Comment 6

[Ryan Kelly [:rfkelly]]

Unless we have a reason not to, I think it would be good to try to squash the two patches here together if possible, so they can land as a single commit (useful for backout, bisection etc purposes). Mark, do you have any wisdom on the right approach to achieve that with phabricator?

Comment 7

[Mark Hammond [:markh] [:mhammond]]

(In reply to Ryan Kelly [:rfkelly] from comment #6)

Unless we have a reason not to, I think it would be good to try to squash the two patches here together if possible, so they can land as a single commit (useful for backout, bisection etc purposes). Mark, do you have any wisdom on the right approach to achieve that with phabricator?

I think you just abandon one of the revisions and ensure the other keeps its "Differential Revision:" line in the commit message - or abandon both then ensure the new commit has no such message, so a new revision ID and line will be generated. And by "adandon" I mean explicitly chose "Abandon Revision" in the "Revision Actions" section of the dropdown where you add comments etc.

Comment 8

[Lougenia Bailey]

(In reply to Mark Hammond [:markh] [:mhammond] from comment #7)

(In reply to Ryan Kelly [:rfkelly] from comment #6)

Unless we have a reason not to, I think it would be good to try to squash the two patches here together if possible, so they can land as a single commit (useful for backout, bisection etc purposes). Mark, do you have any wisdom on the right approach to achieve that with phabricator?

I think you just abandon one of the revisions and ensure the other keeps its "Differential Revision:" line in the commit message - or abandon both then ensure the new commit has no such message, so a new revision ID and line will be generated. And by "adandon" I mean explicitly chose "Abandon Revision" in the "Revision Actions" section of the dropdown where you add comments etc.

In hindsight there are changes that are in Ryan's original revision that aren't in mine and I'm not sure what happens to them if I abandon his revision and land mine. So I'm going to abandon mine and amend the commit in Ryan's revision (https://phabricator.services.mozilla.com/D77762). For now I've added my changes for this ticket to my revision (https://phabricator.services.mozilla.com/D83588) for visibility. I'll clean up my mess on Monday. Sorry for the confusion.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/67168dd1cfc4
add support for storing and fetching FxA ecosystem telemetry ids. r=rfkelly,markh

Comment 10

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/67168dd1cfc4