1539919 - GCP-native job support for WTMO for writing artifacts to GCS

Status: RESOLVED WONTFIX

(Data Platform and Tools Graveyard :: Operations, enhancement, P1)

Description

[Wesley Dawson [:whd]]

We have a use case where we need to upload artifacts to GCS (currently https://github.com/fbertsch/mozilla-schema-generator). There is a notion that this should be done by a service like dataproc or similar GCP-native job. We presently run WTMO in GCP, but all jobs are run via AWS/EMR credentials. We should add support for doing things like launching dataproc jobs and whatever other methods meet our use cases.

I started setting this up but after discussion with :hwoo it seemed better to file a bug and have him hash out the details. On the shared infrastructure side, the only thing that needs to be done is to grant whatever service account that will be running the artifact generation job objectCreator access to the shared artifact bucket.

Comment 1

[Harold Woo]

No longer needed

Comment 2

[Wesley Dawson [:whd]]

To be clear, we needed GCP-native job support, which I believe we now have via GKEPodOperator and the provisioning of kubernetes clusters in GKE. The specific instance in the description was however unneeded hence the WONTFIX.