1631849 - Decomission pioneer_study ping

Status: RESOLVED FIXED

(Data Platform and Tools :: General, task, P1)

Description

[Frank Bertsch [:frank]]

During the investigation of bug 1631530, it seemed to come out that we no longer need the pioneer_study ping to be ingested. Sarah Bird confirmed that we still need the existing data, however. This bug is for:

• Removing the pioneer_study ping from MPS
• Ending ingestion of said pings
• Retaining existing pioneer_study datasets for some time period

Requesting confirmation from rweiss and Sarah Bird for this. Is the above correct?

Comment 1

[Sarah Bird]

I believe this is correct, but I'd like to push the confirmation from my side up to :mlopatka.

Comment 2

[Wesley Dawson [:whd]]

(In reply to Frank Bertsch [:frank] from comment #0)

• Removing the pioneer_study ping from MPS

This can be probably be done safely since we explicitly ignore these in both MSG and ops infra. Only the decoder attempts to do anything pioneer-study related in GCP, and that's largely an oversight (pioneer v1 is unsupported on GCP).

• Ending ingestion of said pings

To be explicit, I think this means making existing pioneer study data emit UnwantedDataExceptions in the decoder. We'll want to be careful about this, depending on the amount of overlap the pathing has with fortchoming pioneer studies.

• Retaining existing pioneer_study datasets for some time period

From previous discussions with :sbird et al., these are currently retained indefinitely in AWS and we have no plans to delete this data.

Comment 3

[Frank Bertsch [:frank]]

This can be probably be done safely since we explicitly ignore these in both MSG and ops infra. Only the decoder attempts to do anything pioneer-study related in GCP, and that's largely an oversight (pioneer v1 is unsupported on GCP).

I wanted to be sure we're looking at the right thing, we are looking at the telemetry/pioneer-study doctype pings. Looks like those two places are ignoring the pioneer-study namespace.

Comment 4

[Wesley Dawson [:whd]]

I wanted to be sure we're looking at the right thing, we are looking at the telemetry/pioneer-study doctype pings. Looks like those two places are ignoring the pioneer-study namespace.

This is the envelope schema which we probably should have been ignoring as well, but weren't. Deleting this may be more complicated, depending on how we implement v2 (bug #1615086 and bug #1628539).

Comment 5

[Frank Bertsch [:frank]]

This is the envelope schema which we probably should have been ignoring as well, but weren't. Deleting this may be more complicated, depending on how we implement v2 (bug #1615086 and bug #1628539).

Anthony, can you chime in here about the v2 implementation? Is there something preventing us from removing the current pioneer_study schema and ending ingestion?

Comment 6

[Anthony Miyaguchi [:amiyaguchi]]

I'm not completely certain on the schema of the v2 envelope; it's possible that we reuse the schema for the envelope. It's also just as easy to create a new schema with a different name, so I'm not against removing it.

Comment 7

[[e-mail]]

I'm reconciling this work against some other Pioneer v1-related decommissioning. It is correct that there are no plans to ship future studies on v1, and it is also correct that those existing v1 datasets are still in use. Are the two open questions thus: 1) what needs to be decided to prevent complications with v2? 2) what are existing v1 resources that can be eliminated/spun down without destroying v1 datasets or preventing future access to v1 datasets?

Comment 8

[Frank Bertsch [:frank]]

1. what needs to be decided to prevent complications with v2?
   We should version v2 separately to avoid any complications. Anthony says there's nothing preventing us from removing v1 from the pipeline. Whether there could be downstream complications is something I doubt, but I'm also not completely aware of how those datasets are used.

2. what are existing v1 resources that can be eliminated/spun down without destroying v1 datasets or preventing future access to v1 datasets?
   My plan was to not touch the AWS resources, and simply remove pioneer v1 from GCP infra. Since we're not ingesting any of it it sounds like that shouldn't be a big deal. :whd, are we also good to remove the pioneer_study namespace? Will that complicate any access to v1 datasets on AWS?

Comment 9

[Wesley Dawson [:whd]]

We should version v2 separately to avoid any complications.

This is complicated by the fact that pioneer uses the telemetry transport and therefore is pinned to version 4. We previously had an internal identifier for versioning schemas in pioneer. :amiyaguchi is looking into this and in particular what the new wrapper schema should look like.

By my current understanding, assuming we re-use pioneer-study as the telemetry docType, v1 pings will be routed to the v2 infra and fail to decode in the new pioneer decoder. We wouldn't need to do anything else but we could also throw an UnwantedDataException if we wanted to. If we use a different docType for pioneer v2 we still need to figure out internal versioning, but we wouldn't have any complications WRT v1 (we would want to flag pioneer v1 data sent to telemetry as unwanted data however).

My plan was to not touch the AWS resources, and simply remove pioneer v1 from GCP infra. Since we're not ingesting any of it it sounds like that shouldn't be a big deal.

This is indeed not a big deal. Bug #1590211 represents the remaining AWS resources and we're leaving that as-is for now.

:whd, are we also good to remove the pioneer_study namespace? Will that complicate any access to v1 datasets on AWS?

We're good to remove the schemas under the pioneer-study directory (it's not a GCP-compliant namespace) and that won't complicate v1 access. We may re-use that directory as a proper namespace for v2.

Comment 10

[mlopatka]

Acknowledged. Everything in the bug description accurately aligns with our world view.
No blockers from my team to proceed with blocking pioneer_study pings from ingestion for outstanding v1 studies.

Comment 11

[Anthony Miyaguchi [:amiyaguchi]]

Having looked over telemetry/pioneer-study/pioneer-study.4.schema.json, I think we should re-use the schema. This would shift responsibility of handling the old, unwanted pings to the pioneer decoder. which would effectively reject all of the v1 pings. Once pioneer-study pings are routed to the separate pioneer decoder, the telemetry pipeline will no longer be populating the telemetry.pioneer_study tables.

Consider using a new schema e.g. pioneer-envelope for the v2 project. The telemetry decoder would continue to process pioneer-study pings before they are filtered out. The schema could be removed and the table removed, but we would also have an empty table for pioneer-envelope unless we bake in some logic to avoid creating the table. From the perspective of the pioneer decoder, using either schema doesn't matter.

Reusing the schema saves a bit of effort since the schema serves the needs of v2, aside from an additional schemaNamespace field necessary for routing. Old pings can be rejected in the pioneer-specific decoder by requiring this new field, perhaps wrapping the validation exception into an UnwantedDataException if it makes sense. Once the pings are rerouted, we can still mark the ping as unwanted data and delete the old table because the pioneer-study schema is a meta-schema that shouldn't have a decoded/live table.

Comment 12

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/532

Comment 13

[Frank Bertsch [:frank]]

Moving to a P1 since Anthony has started this work.

Comment 14

[Anthony Miyaguchi [:amiyaguchi]]

I've removed all of the old study schemas and update the telemetry.pioneer-study schema to include a new column that will start to reject v1 pings. We cannot remove the old tables until the pings are marked specifically as unwanted data or rerouted into the separate pioneer decoder, at least as far as I know.

Comment 15

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/gcp-ingestion/pull/1291