Store data generated by recurring releng tasks in S3

RESOLVED INCOMPLETE

Status

RESOLVED INCOMPLETE
2 years ago
11 months ago

People

(Reporter: coop, Unassigned)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
Tasks that produce releng data for consumption by other services should put that data in S3 instead of rsync-ing that data onto the releng web cluster.

We'll need a new S3 bucket(s?) that has write access from the new releng worker type (bug 1263626), and read access from the services that need to digest the information (treeherder, slave health, ...).

If this data needs to be served over HTTP(S), we can use CloudFront for that.
worker types don't have or lack write access to S3.  The access can be gated in two ways: by adding a scope to the task for
  http://docs.taskcluster.net/auth/api-docs/#awsS3Credentials
which would require that the bucket be in the TaskCluster AWS account; or by storing an AWS access key and secret key in the secrets service, and adding a scope for that secret to the task.

I think all of these services are producing public data, right?  So read access is not restricted?

You definitely want to use https!  If you want a "clean" domain name, rather than https://<bucketname>.s3-us-west-2.amazonaws.com, then you'll need to use CloudFront.
(Assignee)

Updated

a year ago
Component: Tools → General
Product: Release Engineering → Release Engineering

Updated

11 months ago
Status: NEW → RESOLVED
Last Resolved: 11 months ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.