1580957 - Allow databricks-ec2 access to telemetry-spark-emr-2 bucket

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, task, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

Currently databricks does not have access to telemetry-spark-emr-2. This bucket contains credentials for various services that run on EMR. I am porting mozaggregator to run on Databricks, in an eventual port of the service to GCP. Currently the job fails with the following message:

ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied

This can be tested with the following code:

import boto3
import json

credentials_bucket = "telemetry-spark-emr-2"
credentials_prefix = "aggregator_dev_database_envvars.json"

s3 = boto3.resource("s3")
obj = s3.Object(credentials_bucket, credentials_prefix)
creds = json.loads(obj.get()["Body"].read().decode("utf-8"))

Comment 1

[Blake Imsland [:robotblake]]

This is working now.