1316698 - auth: add S3 delegation end-point compatible with iam/security-credentials/role-name from EC2 metadata

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Jonas Finnemann Jensen (:jonasfj)]

We currently have the end-point:

> signature: auth.awsS3Credentials(level, bucket, prefix)
> scopes:    auth:aws-s3:<level>:<bucket>/<prefix>
>
> GET /v1/aws/s3/<level>/<bucket>/<prefix>
> {
>     
>   "credentials": {
>   "accessKeyId":     "...",
>   "secretAccessKey:  "...",
>   "sessionToken:     "..."
>   },
>   "expires": "2012-04-26T16:39:16Z"
> }

I propose that we add a new end-point just like it:
> signature: auth.awsS3IAMRole(level, bucket, prefix)
> scopes:    auth:aws-s3:<level>:<bucket>/<prefix>
>
> GET /v1/aws/s3-iam-role/<level>/<bucket>/<prefix>
> {
>   "Code":             "Success",
>   "LastUpdated":      "2012-04-26T16:39:16Z",
>   "Type":             "AWS-HMAC",
>   "AccessKeyId":      "AKIAIOSFODNN7EXAMPLE",
>   "SecretAccessKey":  "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
>   "Token":            "token",
>   "Expiration":       "2012-04-27T22:39:16Z"
> }

The goal is compatibility with how IAM-roles are assumed by EC2 instances.
See:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials

This someone can do:
  curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<role>
in EC2 and in taskcluster tasks do:
  curl http://taskcluster/auth/v1/aws/s3-iam-role/<level>/<bucket>/<prefix>

@dustin:
  I'm sourcing you for sanity check here. Good idea or bad idea?
  Should we mark awsS3Credentials as deprecated, or would we want to keep both?

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

It's just a different form of the same thing?  It might be better to make it the same method, and just return the different formats based on a query parameter.  That makes it clearer that this isn't actually a different operation.

Comment 2

[(not currently active) Ted Mielczarek]

I asked about this because in bug 1269355 I have to either transform the output of the current endpoint or write a custom parser for it in my S3 handling code in sccache2. I have code that can parse the IAM credentials output, it would be nice to be able to just point it at a different URL and have everything work properly. (The primary motivator here was jonas mentioning that the code should support refreshing the credentials, which the existing IAM code handles.)

Comment 3

[Jonas Finnemann Jensen (:jonasfj)]

bstack agrees with dustin, query parameter it is.

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

https://github.com/taskcluster/taskcluster-auth/pull/84

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/taskcluster/taskcluster-auth

https://github.com/taskcluster/taskcluster-auth/commit/6f8a84373fc33870cd144ca96865488a00650f6f
Bug 1316698 - EC2 metadata compat

https://github.com/taskcluster/taskcluster-auth/commit/a613065de514f21c67efe7dc326f1f0665432f61
Merge pull request #84 from taskcluster/bug-1316698

Bug 1316698 - EC2 metadata compat

Comment 6

[Jonas Finnemann Jensen (:jonasfj)]

Deployed...

Comment 7

[(not currently active) Ted Mielczarek]

Do I need to wait for anything to start using this, or is it available now that it has been deployed?

Comment 8

[Greg Arndt [:garndt]]

This should now be deployed.

Comment 9

[Greg Arndt [:garndt]]

Sorry, this should now be deployed AND available.

Comment 10

[(not currently active) Ted Mielczarek]

This doesn't seem to be working in my testing. I made a one-click loaner from my try job:
https://tools.taskcluster.net/task-inspector/#flTlghDxSuyDyxH5ECcngQ/

but the API is returning results in the existing format even with the query string:
[root@taskcluster-worker ~]# curl http://taskcluster/auth/v1/aws/s3/read-write/taskcluster-level-${MOZ_SCM_LEVEL}-sccache-${TASKCLUSTER_WORKER_GROUP%?}/?format=iam-role-compat
{
  "credentials": {
    "accessKeyId": "ASIAIBBHK2LWNN665CKA",
    "secretAccessKey": "QmBlV03kjGPr4WZKVnQd8DLeMuICiB7Ze84ZnX7L",
    "sessionToken": "FQoDYXdzEBQaDGi8F8WJE+gi5tBJHyKqA6DO0OfKrq+PNz+4NLYgmji51swEVWEAtyHSKefbZ5Ya1Ze3ReZnlf+C2Uri2VZQ9O9BHXMIQ7we00Yp8fhGbBKlaeG34cHNOQDvb0igUeO8Z7l6/ap/eiGA4VqKmop8DGrikRXtls25+JuTqzWe6k4
P08ejJuYJWo2WeKrcPJhKq0JHTAW9o+V2kclCQR7xnxrhuEnWVIupayIr52p7sKj1DkOrpHzhxR8I1WXJgbEAD1DANF8NS4UcRr5RbS8EiVTNdDx1BIHSGc3a3a3+hrqhGEL7Ue6IN1iz9w7IB6VIQK5o9qW/0OZcBzvdgfSelJjrBEsS2+l4FN+8KoiIFpTDQTKNPPRtEdn
wOyBT3I4GVMLoPJgpEJOe0+dkKnu0YA3iok383XRtQRtWZIpvZ8jAYpKUWlPK/JNQwVj3V0DTtXtaJieLMNx23SCqAZ7DYiSTuQq/UcuVcTfXF5FbvzcKFywcI/rZDRwS6VQEJ4pWzEhxZfmnJ8ysT/uARrzaoSc9ShrZLUnyGrEcLOJhKtcQD4o26s86/M33+nOAumVE1m3
1aj/EOAHs6yiHnLbBBQ=="
  },
  "expires": "2016-11-17T12:01:27.000Z"
}

am I doing something wrong here, or missing something?