1664057 - Fix prio_processor dag in telemetry-airflow for v2 tag

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

The v2 tag of the prio-processor image uses the prio package that is published on pypi. This currently fails in production due to breaking changes in the container.

At the same time, I'd like to make modifications to the Airflow dag to support larger nodes for processing. Currently it's running on a n1-standard-1.

The current structure of Prio in Airflow contains three separate GCP projects: admin, A, and B. The admin project needs to write into buckets located in A and B by running a container. Containers are not run in the shared cluster because of service account permissions. Instead, each project spins up an ephemeral kubernetes cluster.

This will be changed so the cluster is always alive with a g1-small node, with a burstable node pool for processing multiple batches of jobs (e.g. backfill).

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: GitHub Pull Request

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Refactor authentication in bin and fix bootstrapping in Airflow #97