1579441 - Delete all Parquet files from S3

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, task, P1)

Description

[Jeff Klukas [:klukas] (UTC-4)]

This is part of the overall H2 effort towards retiring the AWS telemetry ingestion pipeline and data lake.

We will not be able to delete the data lake in AWS until the AWS pipeline is shut down and all relevant data is imported to GCP and available in BigQuery. At time of creating this bug, our target is late November.

Comment 1

[Jeff Klukas [:klukas] (UTC-4)]

Note that we are nearly done with the backfill of heka data to BQ, so we will be deleting heka files from AWS in the near future. After that's done, we'll turn our attention more seriously to deleting Parquet data. We may start altering BQ views next week to avoid using imported Parquet data, so I believe we're still on track to turn off Parquet datasets by end of November.

Comment 2

[Ryan Harter [:harter]]

Are we planning on moving parquet files to a GCP filestore or are parquet files that are not migrated to BQ going to be lost? For clarity, both of these options seem reasonable to me!

Thanks!

Comment 3

[Jeff Klukas [:klukas] (UTC-4)]

Generally, the Parquet datasets are now available as BQ tables. We are considering the BQ tables the migration path and we have no plan to store the Parquet files in GCS.

Do let us know if there is any specific dataset you're worried about.

Comment 4

[Ryan Harter [:harter]]

I'm currently working on an analysis that depends on telemetry_ip_privacy_parquet. I was hoping I could dispatch the analysis this week, but it looks like it's going to continue for a while. I just found this table so I haven't talked to anyone about preserving it. Is it something we can make available in GCP?

Comment 5

[Jeff Klukas [:klukas] (UTC-4)]

If we haven't already copied that over and made it available in BQ, it's on the list to be made available. More info will be coming in this bug.

Comment 6

[Jeff Klukas [:klukas] (UTC-4)]

Looks like it already exists as moz-fx-data-derived-datasets.telemetry_derived.telemetry_ip_privacy_parquet_v1 and should be accessible as telemetry.telemetry_ip_privacy_parquet_v1 in Redash, etc.

Comment 7

[Ryan Harter [:harter]]

Awesome! Thanks!

Comment 8

[Jeff Klukas [:klukas] (UTC-4)]

There appears to be some significant confusion about timeline for retiring Parquet data. I am going to send out a message to fx-data-dev and ask for feedback in this bug about particular use cases that would be severely impacted by that timeline.

Comment 9

[Jeff Klukas [:klukas] (UTC-4)]

There was no new feedback about critical Parquet use cases, though folks did appreciate having more clarity on the timeline. I checked in with :jason and :robotblake and sent out additional communication today to indicate we'll remove access to all Parquet data on 12/11 and then hard delete on 12/18.

Comment 10

[Jeff Klukas [:klukas] (UTC-4)]

?ni :robotblake Are you the one to shut off access permissions to telemetry-parquet? Can you clear the needsinfo when that's done?

Comment 11

[Blake Imsland [:robotblake]]

Yep, I can take care of that.

Comment 12

[Blake Imsland [:robotblake]]

So we're keeping taar/* and telemetry-ml/* prefixes temporarily because TAAR is still using them. Also socorro_crash/v2/* is still being written to by Airflow and is potentially still being used by Athena.

Any others?

Comment 13

[Blake Imsland [:robotblake]]

Turns out socorro_crash/v2/* isn't actually needed.

Access to all data has been removed with the exception of TAAR which still has access to the previously mentioned prefixes.

Comment 14

[Blake Imsland [:robotblake]]

All the data besides the previously mentioned prefixes have been removed!