1263636 - Store data generated by recurring releng tasks in S3

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Chris Cooper [:coop] (he/him)]

Tasks that produce releng data for consumption by other services should put that data in S3 instead of rsync-ing that data onto the releng web cluster.

We'll need a new S3 bucket(s?) that has write access from the new releng worker type (bug 1263626), and read access from the services that need to digest the information (treeherder, slave health, ...).

If this data needs to be served over HTTP(S), we can use CloudFront for that.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

worker types don't have or lack write access to S3.  The access can be gated in two ways: by adding a scope to the task for
  http://docs.taskcluster.net/auth/api-docs/#awsS3Credentials
which would require that the bucket be in the TaskCluster AWS account; or by storing an AWS access key and secret key in the secrets service, and adding a scope for that secret to the task.

I think all of these services are producing public data, right?  So read access is not restricted?

You definitely want to use https!  If you want a "clean" domain name, rather than https://<bucketname>.s3-us-west-2.amazonaws.com, then you'll need to use CloudFront.