Closed Bug 1276963 Opened 8 years ago Closed 8 years ago

Insufficient scopes error while scheduling jobs in TC

Categories

(Taskcluster Graveyard :: Scheduler, defect)

Product:
Taskcluster Graveyard
Component:
Scheduler
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: martianwars, Unassigned)

References

Details

I am getting this error (https://pastebin.mozilla.org/8874014) when I try to schedule TC jobs using this script(https://github.com/armenzg/TC_developer_scheduling_experiments/blob/master/schedule_linux64_task.py). I get through the browser authentication and get this then. Also, that output has some more jobs where I didn't have sufficient scopes (length of snippet was too long)
Flags: needinfo?(garndt)
Flags: needinfo?(armenzg)
It appears that whatever taskcluster credentials you're using doesn't have the scopes for creating a task for that worker type based on the error you're getting. 

At a minimum you would need the scope "queue:create-task:aws-provisioner-v1/desktop-test"

Armen, did you end up creating a role for running this script that people can be a part of so they have scopes?
Flags: needinfo?(garndt)
Wrt to the SNI warning, you probably have an old Python version. Are you using Ubuntu? It only went away after I upgraded the OS (since it is uses /usr/bin/python). You can also install requests[security] to monkey patch this.

Hi gardnt, how should we go about this? I only want a set of credentials for Kalpesh start making progress for the "add new jobs" feature that pulse_actions will have.

Should we create a role? or a client?
I want Kalpesh to have access to such client until at least the end of the summer.
We can create a different one for production once we figure out the right set of scopes.
Flags: needinfo?(armenzg)
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
How was this resolved? Did we create a role?
I created a client [1]
I was trying to add these scopes to begin with [2], however, I can't add the scopes myself.

Can someone please help us figure out what scopes we need to schedule any task on any of the repositories?

[1] mozilla-ldap/armenzg@mozilla.com/martianwars
[2]
 queue:create-task:*
 queue:define-task:*
 scheduler:create-task-graph
 scheduler:extend-task-graph
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Selena Deckelmann :selenamarie :selena from comment #3)
> How was this resolved? Did we create a role?

martianwars logged in with okta rather than persona which gave him the appropriate scopes.
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #4)
> I created a client [1]
> I was trying to add these scopes to begin with [2], however, I can't add the
> scopes myself.
> 
> Can someone please help us figure out what scopes we need to schedule any
> task on any of the repositories?
> 
> [1] mozilla-ldap/armenzg@mozilla.com/martianwars
> [2]
>  queue:create-task:*
>  queue:define-task:*
>  scheduler:create-task-graph
>  scheduler:extend-task-graph

There is a rather long list of scopes that are necessary to schedule almost any task that is currently being scheduled.  Taskcluster clients are created with an authorizedScope of "assume:repo:<repo>" so that no repo can overstep the scopes its allowed.  This means that the permanent taskcluster credentials must at least possess all the scopes for those roles it will assume.

It sounds like mozci might need a permanent taskcluster credentials that has scopes similar to those mozilla-taskcluster has, which are:

    assume:repo:github.com/mozilla-b2g/gaia:*
    assume:repo:hg.mozilla.org/*
    assume:scheduler-id:task-graph-scheduler/*
    queue:cancel-task
    scheduler:create-task-graph

I'm going to ni? dustin on this since he worked on creating a lot of these roles and I want to see if we're on the right track.
Flags: needinfo?(dustin)
Yes, that seems reasonable, but note that that is a *very* powerful credential, essentially capable of publishing a compromised firefox, so please treat it carefully.

It will need to be a credential not under `mozilla-ldap/armenzg@mozilla.com/*`, otherwise it will be disabled.  So this should probably be a client under projects/ateam or the like.
Flags: needinfo?(dustin)
We will.

Thank you all!
Armen: I'm very concerned about the architecture of this service. Has an RRA been done?
Flags: needinfo?(armenzg)
Could you clarify what the see-also on bug 1273034 is for?

If it's "allow people to use their own scopes rather than needing a generic one for this tool" then it's bug 1273096 that's more relevant (which itself is dependent on bug 1273034).

Or is it re the RRA / some "avoid footgun" parts above (I'm struggling to follow above what should and shouldn't be done wrt scopes).
Flags: needinfo?(sdeckelmann)
From today's conversation.

To schedule any task on any repo all I need to do is "assume:repo:hg.mozilla.org/*"

Once we have mozilla-taskcluster re-written as planned and using action tasks I'm happy to move there.

For now, I would like at the RRA and see what we can have working now versus the future.

I'm leaving the NI in place until I file the RRA.

12:57 PM <selenamarie> armenzg_lunch: just ask kang :) its pretty easy.
12:57 PM <•selenamarie> armenzg_lunch: i'm worried about adding broad scopes to another service. if we could do it with scopes specific to a user, delegated based on a login, that would be better based on what i understand
12:58 PM <kang> armenzg_lunch: the super-tl-dr is https://bugzilla.mozilla.org/enter_bug.cgi?product=Enterprise%20Information%20Security&component=Rapid%20Risk%20Analysis with some info on the service to review, data it uses etc. the process is documented at https://wiki.mozilla.org/Security/Risk_management/Rapid_Risk_Assessment as wellg
1:45 PM <armenzg> thanks selenamarie kang
1:46 PM <armenzg> garndt: dustin - any of you could catch up with me wrt to what are the scopes we would need and the risks of each?
1:47 PM <dustin> armenzg: well, the scopes you would need are the scopes to run any task in any hg repository
1:48 PM <armenzg> dustin: what are those? I assume each different scopes has different risks
1:48 PM <armenzg> *scope
1:48 PM <dustin> right but we don't want to give a single list of scopes
1:48 PM <dustin> since every time we change teh scopes for tasks in-tree, that would go out of date
1:48 PM <armenzg> so, we're looking at an "assume"?
1:48 PM <dustin> yeah
1:48 PM <armenzg> OK; which one?
1:48 PM <dustin> assume:repo:hg.mozilla.org/*
1:48 PM <armenzg> so I can document it and read which scopes it has
1:49 PM <armenzg> great! it wasn't clear what assuming a repo implies
1:50 PM <dustin> if we wanted to limit it to just retriggering try jobs, you could use assume:repo:hg.mozilla.org/try:*
1:51 PM <armenzg> OK
1:51 PM <armenzg> let me start the RRA and see where we get
2:28 PM <dustin> I think the optimal solution here may be to go to "action tsaks"
2:28 PM <dustin> since mozilla-taskcluster can then give the action task the repo-appropriate scopes
2:58 PM <armenzg> dustin: what's the technichal difference or advantage between using mozilla-taskcluster or pulse_actions?
2:59 PM <armenzg> there's an agent that have the capacity to schedule a task which can schedule more tasks
2:59 PM <dustin> it's mostly about using the same codepaths that we already use for decision tasks
3:00 PM <armenzg> if we get pulse_actions to use action tasks, could we then port that to mozilla-taskcluster?
3:00 PM <armenzg> the last time we spoke about mozilla-taskcluster it seems that it was rather a mix of many things
3:00 PM <dustin> yeah, it is, I was using "mozilla-taskcluster" to include its successors
3:00 PM <armenzg> and perhaps needed to be rewritten
3:00 PM <dustin> so pulse_actions could trigger action tasks, too
3:00 PM <armenzg> is anyone scheduled to work on that?
3:01 PM <dustin> bits of it
3:01 PM <dustin> I'm working on replacing the hg poller with hooks
3:01 PM <armenzg> I would be happy to use that once it is ready for prime time
3:01 PM <armenzg> oh cool!
3:01 PM <armenzg> yeah, I saw that landing last month (the pulse exchange
3:01 PM <armenzg> )
3:01 PM <dustin> garndt is working on the treeherder integration
3:01 PM <dustin> anyway
3:01 PM <dustin> pulse_actions could create the action tasks, too
3:02 PM <armenzg> OK
3:02 PM <dustin> what i want to avoid is doing complex things (like, more complex than getting a pulse message, authenticating it, and creating a new task with scopes based on the repo in that pulse message) with elevated scopes
3:02 PM <armenzg> not our goal
3:04 PM <dustin> what's not?
3:10 PM <armenzg> to make anything more complicated that that
3:12 PM <dustin> so basically there should be 10-20 lines of code that use the assume:repo:hg.mozilla.org/* scope
Removing see also, pending comment 10. (I'm pretty such it should be a dependency rather than a see also, and on bug 1273096 not bug 1273034).
See Also: 1273034 → 
Filed the RRA and will discuss it with dustin.
Flags: needinfo?(armenzg)
So, we held the RRA today.  I think the key takeaway was:

assume we can ensure that only task graphs the user could have created (are at or below their SCM level) can be extended, so a level-1 person can only modify level-1 pushes (try)
  -> then this service provides no capabilities not already present by pushing to that tree

We'll accomplish this initially by only operating at level 1.  Then when treeherder's integration with taskcluster has improved (and thus we can determine the user's permissions and/or scm level reliably), we can improve that.

There were a few other issues mixed in on this bug, too.  Notably, the scopes assigned to users' temporary credentials currently do not include the treeherder routes for any specific tree.  I've punted the general fix off to bug 1278986.  However, I'm happy to give martianwars a clientId and accessToken that does have the necessary scopes.  I'll do that offline.
Flags: needinfo?(sdeckelmann)
I just scheduled three jobs https://treeherder.mozilla.org/#/jobs?repo=try&revision=50b7172b65652f6952298d40c550268834e6e24a successfully. Thank you dustin and armenzg! I guess I can carry on with the project now.
Excellent!
Thanks dustin!
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → FIXED
We needed to get this working for martianwars to keep making progress on bug 1254325.

Adding blocking.
Blocks: 1254325
Product: Taskcluster → Taskcluster Graveyard
You need to log in before you can comment on or make changes to this bug.