1664582 - Run prio-processor containers on persistent GKE clusters

Status: NEW

(Data Platform and Tools :: General, enhancement, P2)

Description

[Anthony Miyaguchi [:amiyaguchi]]

From 1664057:

The current structure of Prio in Airflow contains three separate GCP projects: admin, A, and B. The admin project needs to write into buckets located in A and B by running a container. Containers are not run in the shared cluster because of service account permissions. Instead, each project spins up an ephemeral kubernetes cluster.

This will be changed so the cluster is always alive with a g1-small node, with a burstable node pool for processing multiple batches of jobs (e.g. backfill). This should be roughly $5/mo to keep the cluster alive.

The Airflow PR will contain the new cluster configuration and the new pod operator definitions. The cluster definition will likely move into the cloudops infra repository to be managed programmatically for each project.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-airflow/pull/1144

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-airflow/pull/1157