1476361 - New public-readable bucket writeable from databricks for federated learning study model results

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, enhancement)

Description

[bugzilla]

The federated learning streaming job is going to output a model in json every 30 minutes, which clients enrolled in the study will fetch periodically and use. Since the streaming job is going to be running on databricks, the bucket needs to be writeable from there as well as being world-readable.

Comment 1

[Arkadiusz Komarzewski [:akomar]]

Job also needs permission for reading from Kafka (currently we use `databricks-ec2` role for this).

Comment 2

[Jason Thomas [:jason]]

In Bug 1455725 we created a s3 bucket that is publicly accessible + CDN endpoint. Can we use that instead?

If so:

S3 bucket = net-mozaws-prod-us-west-2-data-public

CDN Endpoint = https://public-data.telemetry.mozilla.org

You need to make sure you set the object acl to 'public-read'.


# tested from shared_serverless cluster
curl https://public-data.telemetry.mozilla.org/jthomas/testacl1.txt
yay%

Comment 3

[Jason Thomas [:jason]]

https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/9388/command/9389 is the notebook I used to write the data if you need an example.

Comment 4

[bugzilla]

Fantastic, that'll work for us -- we'll put our stuff under the /awesomebar_study/ key