1656949 - Request for bigquery job to forward AET ID updates to pipeline from FxA logs

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, enhancement, P1)

Description

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

+++ This bug was initially created as a clone of Bug #1653654 +++

In https://github.com/mozilla/fxa/pull/6081, FxA has added an event to the auth-server logs to link AET anon_id values when they are changed.

The event name is account.updateEcosystemAnonId.complete and its format is:

{ previous: <old anon_id value>, next: <new anon_id value> }

These events need to be forwarded to the AET pipeline, to ensure continuity across password resets, and also to ensure we can recreate a historical list of IDs to fully delete user data in response to account deletion requests. The existing AET schema allows for historical values to be included, so I think it should work fine for this use case.

We need a bigquery-etl job to be setup to find this event in the fxa-auth-server logs and forward it to the AET endpoint using the AET event schema.

Comment 1

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Ah, I should point out that if a user's having their anon_id set for the first time, there will be an update event where the previous value will be some sort of nullish value. I'll have to double-check what that value will be.

Comment 2

[Jeff Klukas [:klukas] (UTC-4)]

Per https://bugzilla.mozilla.org/show_bug.cgi?id=1653654#c6 we have defined the integration point between FxA infra and the data pipeline is pubsub topic, where FxA will maintain a cloud function or similar that reads Stackdriver logs from Pub/Sub, transforms the data into the appropriate format for the data pipeline, and publishes to the destination pubsub topic that the data pipeline then picks up.

I believe that's the way to go for this need as well.

The FxA-side logic can transform these into documents for the existing firefox-accounts/account-ecosystem schema. The new anon_id value would go into the ecosystem_anon_id field in the JSON payload and the previous value would be presented as a single-element array named previous_ecosystem_anon_ids. If there's no previous value, the array should be missing or empty.

Comment 3

[Jeff Klukas [:klukas] (UTC-4)]

We previously have discussed a desire for better error handling around the logic of transforming log events and validating them against schemas, which would argue for pushing transformation logic to the data pipeline side. Perhaps as a future improvement we could consider defining support in the data pipeline for ingesting a Stackdriver Pub/Sub topic directly.

This would be somewhat similar to how we have special support in the pipeline for stub installer, which has a completely custom URI scheme. Early in the Decoder processing, we do special parsing of stub installer payloads in order to transform them into the structured format expected by the rest of the pipeline.

For now, I think having a cloud function FxA side that handles this specific use case will be significantly easier to implement. If you're interested in exploring this further, we can split off a bug for further investigation.

Comment 4

[Jeff Klukas [:klukas] (UTC-4)]

See https://bugzilla.mozilla.org/show_bug.cgi?id=1653654#c9 for more discussion of the possibility of passing Stackdriver messages to the data pipeline directly.

Comment 5

[Jeff Klukas [:klukas] (UTC-4)]

I have not been able to find any events matching this description in the FxA logs in BigQuery.

?ni jhirsch - Should I expect to see events of this type being produced? Can you give me some more identifying info to help track them down? Should I expect a value of "account.updateEcosystemAnonId.complete" in the jsonPayload.fields.event field?

Comment 6

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Hi :klukas, judging by https://bugzilla.mozilla.org/show_bug.cgi?id=1635659#c33, FxA might not be successfully updating the ecosystem_anon_id on password change/reset. If we can confirm this bug, it would explain why you aren't seeing any change events in the logs.

But yes, this sounds correct:

Should I expect a value of "account.updateEcosystemAnonId.complete" in the jsonPayload.fields.event field?

I can ping you in this bug when we are sure it's working.

Comment 7

[Jeff Klukas [:klukas] (UTC-4)]

It's my understanding this (https://github.com/mozilla/fxa/pull/6379) went out yesterday with the fxa prod deploy, but the deploy was rolled back. I would expect to see some events, but I have not been able to find any in the auth server logs:

SELECT
  COUNT(*)
FROM
  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20200914`
WHERE
  TO_JSON_STRING(jsonPayload) LIKE '%account.updateEcosystemAnonId.complete%'

-- Returns 0

Jared - Do you expect docker_fxa_auth would be where these events show up? Do we any have evidence further upstream that these events were being sent before the rollback?

Comment 8

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

The deploy should have gone out by now. I'm investigating some unrelated missing stuff in the auth-server logs. I can look for this, too.

Comment 9

[Jeff Klukas [:klukas] (UTC-4)]

This version of AET is no longer being pursued, so closing.