1594137 - Decommission pipeline of Firefox accounts events to Redshift

Status: RESOLVED INCOMPLETE

(Data Platform and Tools Graveyard :: Operations, task)

Description

[William Lachance (:wlach)]

We're currently tracking a list of data platform related AWS -> GCP migration tasks here:

https://docs.google.com/spreadsheets/d/1f6fVsEp8FaX6ri98benHexzsHWYU9HMEbNW1poJQd1w/edit#gid=0

One of the things we'd like to decommission is the pipeline of FxA events to a Redshift store in AWS. My understanding from talking to :klukas is that we've already substantially migrated this data into bigquery, using various bigquery-etl jobs (here's an example: https://github.com/mozilla/bigquery-etl/tree/master/sql/telemetry/fxa_all_events_v1).

However, based on discussions with :jbuck, it seems like the redshift store / pipeline is still running?

We should verify that my understanding of this situation is correct and, if so, decommission any jobs still writing to this store as well as the store itself.

Leif, could you help confirm/disconfirm the above? My understanding from Jeff is that you're closest to this problem space.

Comment 1

[Leif Oines [:loines]]

Sorry for the late response on this. This pipeline is still running, and in fact it is an entirely separate taxonomy of events from the ones that Jeff referred you to in the link above. They are the "original" set of FxA activity/flow metrics that includes performance metrics and other events that are not covered by the existing BQ etl (which only covers events sent to amplitude).

There is an FxA performance dashboard that the engineering team refers to that depends on these events. Otherwise, I am really the only regular consumer of these data. I still use it moderately, ~2-3 times a month to answer questions that the data in BQ can't answer.

In terms of the existing (already collected) data, they are just CSV files so copying them from s3 to GCP shouldn't be a big issue (I hope). Going forward for future data collection, here are the two long-term solutions that I see that don't involve losing data.

1. Like the amplitude events currently imported to BQ, these events are derived from the FxA server logs. So, in theory we could develop some jobs to pipe them to some BQ tables. I'm guessing that's largely data eng/ops work.

2. We could ditch this taxonomy of events entirely and port the events that don't have amplitude analogs (e.g. the performance measures) to an amplitude format. Then they would be taken care of by the existing BQ etl. That would require code changes to FxA, and thus would probably fall on that team. There are many such events, and I don't think this would be trivial work.

Overall, my perception is that (1) would involve the least amount of total work. However, (2) might be preferable as a longer term solution as the reason for these two parallel sets of events is historical and not strategic.

Is there a rush to decommission this? I'm not sure what a fast, light-lift short-term solution would be here. Jbuck mentioned that we have some options, but I honestly don't know what they are.

Comment 2

[Jeff Klukas [:klukas] (UTC-4)]

This pipeline is still running, and in fact it is an entirely separate taxonomy of events from the ones that Jeff referred you to in the link above. They are the "original" set of FxA activity/flow metrics that includes performance metrics and other events that are not covered by the existing BQ etl (which only covers events sent to amplitude).

It was my understanding that all these activity/flow metrics that are ingested to Redshift also now flow via Stackdriver into BQ in the moz-fx-fxa-prod-0712 project. We have implemented nightly queries to copy most of the amplitude events from moz-fx-fxa-prod-0712 into tables in the telemetry namespace.

If you had access to query relevant tables in moz-fx-fxa-prod-0712 directly, would that be able to serve the same needs that Redshift is currently needed for?

Is there a rush to decommission this? I'm not sure what a fast, light-lift short-term solution would be here. Jbuck mentioned that we have some options, but I honestly don't know what they are.

I don't know that there's a pressing need to decommission this, other than the general directive that we do eventually need to stop maintaining old AWS infra. It sounds like the use cases here are pretty slim at this point, so it does seem like a good candidate to focus some energy on sooner rather than later to get this completely off the books and off the AWS bill.

Comment 3

[Leif Oines [:loines]]

(In reply to Jeff Klukas [:klukas] (UTC-4) from comment #2)

If you had access to query relevant tables in moz-fx-fxa-prod-0712 directly, would that be able to serve the same needs that Redshift is currently needed for?

Are you talking about just querying the logs directly for the flow events? Yes, I can do that (and have done so) but it is painful not to mention expensive as a single day's worth of server logs is quite large. Also, I think the stackdriver stuff only sticks around for N months (3? or maybe that's changed?). Or are you saying we create some new tables within that project that just have the flow/activity events? That would be fine with me.

I don't know that there's a pressing need to decommission this, other than the general directive that we do eventually need to stop maintaining old AWS infra. It sounds like the use cases here are pretty slim at this point, so it does seem like a good candidate to focus some energy on sooner rather than later to get this completely off the books and off the AWS bill.

:thumbs:

Comment 4

[Jeff Klukas [:klukas] (UTC-4)]

Are you talking about just querying the logs directly for the flow events? Yes, I can do that (and have done so) but it is painful not to mention expensive as a single day's worth of server logs is quite large. Also, I think the stackdriver stuff only sticks around for N months (3? or maybe that's changed?). Or are you saying we create some new tables within that project that just have the flow/activity events? That would be fine with me.

I think we can handle this as we've handled previous FxA needs. If you can provide queries to pull the data that you need, it should be straightforward for data platform to get those scheduled to pull the data into telemetry datasets. As long as there's nothing new sensitivity-wise that we would need to hide from the general population with query access.

And yes, as far as I know the stackdriver pipeline has a 3 month retention policy, so we would need to use scheduled queries to keep data longer than that.

Comment 5

[Leif Oines [:loines]]

So I looked at the event volumes for the FxA Activity Metrics dataset for the last week and started a spreadsheet in [1]. For all the events with volume > 100k I noted if there are currently amplitude versions of those events, and if not, whether we should create an amplitude version. I think at a bare min., if we are going to decommission this dataset completely, that we will want to instrument those new amplitude events first.

I also ignored the performance events, that is an entirely separate question in my mind - I still think we want (something like) those events but it doesn't seem appropriate to send them to amplitude.

CC :adavis

[1] https://docs.google.com/spreadsheets/d/1eergxcV1PCp4oQYdMqMNfHrIgH5gnrnCh8_6k4liyrA/edit?usp=sharing

Comment 6

[Wil Clouser [:clouserw]]

Is there Amplitude budget for those new events? There are quite a few new events in that spreadsheet and I think we were already risking going over our limit.

Comment 7

[Leif Oines [:loines]]

I could estimate how much it would cause us to add. In some cases multiple could be collapsed into single events with different user properties. Another option is to log the events "like" an amplitude event so they show up in the logs/stackdriver as such, and make it into bigquery, but are not actually sent to amplitude.

I'm also not saying that we absolutely have to have all of the ones I indicated. For some, I was channeling Alex a bit because he had mentioned wanting them in the past. It might be useful for him to go through the list himself.

I would think of that spreadsheet as a first draft of a wishlist.

Comment 8

[Wesley Dawson [:whd]]

I'm reasonably confident that this bug is old enough that further work is obviated by time. Particularly it's my understanding that we no longer use Amplitude per https://github.com/mozilla/telemetry-airflow/blob/main/GRAVEYARD.md#2021-07-05-amplitude-export.