1198391 - Implement separate scopes for viewing pending counts vs secrets

Selena Deckelmann :selenamarie :selena

Reporter

Description

•

10 years ago

We need separate scopes for the secrets UI from the pending counts.

John Ford [:jhford] CET/CEST Berlin Time

Comment 1

•

9 years ago

We do already have seperate scopes, the pending counts are not scope-protected and the listing of worker names is under an aws-provisioner:list-worker-types scope. I think right now, default credentials are missing all aws-provisioner:* scopes, intentionally. If we started to give them aws-provisioner:list-worker-types as a scope, we could theoretically show them the list of workers and how many pending tasks each has. If we also gave them aws-provisioner:aws-state, we could theoretically show them the entire UI aside from the defintion of the workerType, meaning no secrets. Before we put time into modifying the provisioner ui to work with degraded scope sets, do we know if this is a common use case? If we just want a list of worker types and pending counts, it might be easiest to have a different UI and keep the main UI easier to work on.

Pete Moore [:pmoore][:pete]

Comment 2

•

9 years ago

If there is not sensitive data under https://aws-provisioner.taskcluster.net/v1/aws-state I guess we could remove the `aws-provisioner:aws-state` scope requirement, and then point users (such as sheriffs) to this url until we have a new shiny UI for it? This would allow them to parse the json until we have a pretty UI that doesn't require taskcluster creds. Or we could grant aws-state to all/some clientids (or add via roles if already implemented) and then tell users to use a taskcluster client if they want to query the data (or could they use the existing UI?). If there is no sensitive data under this URL, I like the idea that you can just hit the url in your web browser and get the data, rather than needing to write code which interacts with one of the taskcluster client libraries... I had a look at the response body by writing a small wrapper app which uses the taskcluster client, the data looks like this: { "ami-test": { "running": [], "pending": [], "spotReq": [] }, "amiyaguchi-ff-desktop": { "running": [], "pending": [], "spotReq": [] }, "android-api-11": { "running": [ { "id": "i-2f456ef4", "srId": "sir-03812q34", "ami": "ami-fc02e3cf", "type": "m3.2xlarge", "region": "us-west-2", "zone": "us-west-2c", "state": "running", "launch": "2015-10-19T13:51:24.000Z" }, { "id": "i-4a73b88e", "srId": "sir-037x4cj7", "ami": "ami-fc02e3cf", "type": "m3.2xlarge", "region": "us-west-2", "zone": "us-west-2a", "state": "running", "launch": "2015-10-19T09:31:03.000Z" }, { "id": "i-4dadb88b", "srId": "sir-0383qn2v", "ami": "ami-fc02e3cf", "type": "m3.2xlarge", "region": "us-west-2", "zone": "us-west-2b", "state": "running", "launch": "2015-10-19T11:20:18.000Z" }, { "id": "i-1ce52fd8", "srId": "sir-037vk3b8", "ami": "ami-fc02e3cf", "type": "m3.2xlarge", "region": "us-west-2", "zone": "us-west-2a", "state": "running", "launch": "2015-10-19T13:22:51.000Z" }, { "id": "i-15fb31d1", "srId": "sir-0383glrv", "ami": "ami-fc02e3cf", "type": "m3.2xlarge", "region": "us-west-2", "zone": "us-west-2a", "state": "running", "launch": "2015-10-19T13:27:37.000Z" }, { "id": "i-28456ef3", "srId": "sir-037t790h", "ami": "ami-fc02e3cf", "type": "m3.2xlarge", "region": "us-west-2", "zone": "us-west-2c", "state": "running", "launch": "2015-10-19T13:51:24.000Z" }, { "id": "i-6efb31aa", "srId": "sir-037tr865", "ami": "ami-fc02e3cf", "type": "m3.2xlarge", "region": "us-west-2", "zone": "us-west-2a", "state": "running", "launch": "2015-10-19T13:28:00.000Z" }, { "id": "i-10fb31d4", "srId": "sir-037x5hx4", "ami": "ami-fc02e3cf", "type": "m3.2xlarge", "region": "us-west-2", "zone": "us-west-2a", "state": "running", "launch": "2015-10-19T13:27:40.000Z" } ], "pending": [], "spotReq": [] }, "b2g-desktop-debug": { "running": [ { "id": "i-3b4c67e0", "srId": "sir-0380t925", "ami": "ami-fc02e3cf", "type": "c3.2xlarge", "region": "us-west-2", "zone": "us-west-2c", "state": "running", "launch": "2015-10-19T13:31:48.000Z" }, { "id": "i-554a618e", "srId": "sir-0384gqr8", "ami": "ami-fc02e3cf", ..... ..... ..... I couldn't see anything in there that might be highly sensitive, so maybe removing `aws-provisioner:aws-state` scope requirement might really be a good option. I did notice in the docs http://docs.taskcluster.net/aws-provisioner/api-docs/#awsState it says "This method is a left over and will be removed as soon as the tools.tc.net UI is updated to use the per-worker state". I think in the case of the web app, it is efficient to be able to perform a single http request and get back e.g. pending count data for all worker types, to avoid needing to perform an individual http request per worker type (1 request vs ~60 requests). Might it be an option to keep this method, and not deprecate it, for efficiency gains? This is only valid if web pages will continue to show pending counts across all workers - if this is later e.g. paginated, or lazily loaded when scrolling, maybe being able to get pending/spot/running counts for a definable range of worker types in a single http request would be the answer. This is just an optimisation, much like doing a SELECT over a range in SQL rather than a single SELECT statement per row needed. Sorry if I am bikeshedding! :) Lastly, do we have a bug about updating aws provisioner UI to show pending counts even if you don't have permissions to view/manage worker types? I've seen talk about us doing it, but I couldn't find a bug under: * https://bugzilla.mozilla.org/buglist.cgi?quicksearch=product%3Ataskcluster%20component%3Atools * https://bugzilla.mozilla.org/buglist.cgi?quicksearch=product%3Ataskcluster%20component%3A%22aws-provisioner%22 so I wondered whether perhaps we'd forgotten to create a bug for it. Thanks John!

Flags: needinfo?(jhford)

Pete Moore [:pmoore][:pete]

Comment 3

•

9 years ago

Sorry John, rereading comment 1 I see you already answered the part about how to get pending counts across worker types without needing one request per worker type... by adding the pending counts to some endpoint that requires `aws-provisioner:list-worker-types`. So I agree it totally makes sense that http://docs.taskcluster.net/aws-provisioner/api-docs/#awsState endpoint is deprecated. Sorry for the noise on this! Question: Would we modify http://docs.taskcluster.net/aws-provisioner/api-docs/#listWorkerTypes to return more data per worker type (name, pending count, spot count, running count) or would we create a new endpoint for this? Left over: Do we need to create bugs for this stuff?

John Ford [:jhford] CET/CEST Berlin Time

Comment 4

•

9 years ago

We have the /state/:workerType endpoint. I definately don't want to put an endpoint in the provisioner which is basically redirecting the output from a Queue api for anything other than an unsupported convenience method. Having an endpoint that has this sort of metadata could be useful, but the only advantage to it would be allowing non-authed users to see status... is that our goal here?

Flags: needinfo?(jhford)

Pete Moore [:pmoore][:pete]

Comment 5

•

9 years ago

Yes, it was basically to solve the problem that sheriffs can't monitor pending counts. Maybe if they get taskcluster perma creds they could (or maybe via the new auth ldap integration) but atm the UI also shows them worker type definitions which could be sensitive. What I'm thinking is ideally anyone could view pending counts, even without creds, and that viewing the worker types would be locked down.

https://github.com/taskcluster/aws-provisioner/pull/67 9 years ago Dustin J. Mitchell [:dustin] (he/him) 54 bytes, text/x-github-pull-request	dustin : review+	Details \| Review
https://github.com/taskcluster/aws-provisioner/pull/72 9 years ago Dustin J. Mitchell [:dustin] (he/him) 54 bytes, text/x-github-pull-request	dustin : review+	Details \| Review