1486016 - please add generic-worker:os-group:Administrators to kmoir's tc scopes

Status: RESOLVED DUPLICATE of bug 1489268

(Taskcluster :: Operations and Service Requests, task)

Description

[Kim Moir [:kmoir] ET]

From 

https://wiki.mozilla.org/ReleaseEngineering/How_To/Self_Provision_a_TaskCluster_Windows_Instance


5) If you will require administrator privileges:

5.1) Ensure you have scope generic-worker:os-group:Administrators in https://tools.taskcluster.net/credentials/ 

I'm trying to get a windows machine to test some work in bug 1485228

Comment 1

[Kim Moir [:kmoir] ET]

bumping the priority because this is blocking my work

Comment 2

[Pete Moore [:pmoore][:pete]]

Hi Kim,

Sorry for the delay!

Which provisionerId / workerType is this for? Since you created this bug, the scopes now include the provisionerId and workerType so we can grant per worker type.

Comment 3

[Kim Moir [:kmoir] ET]

Provisioner	aws-provisioner-v1
WorkerType	gecko-3-b-win2012

and

Provisioner	scriptworker-prov-v1
WorkerType	depsigning

Comment 4

[Pete Moore [:pmoore][:pete]]

Hi Kim,

I've added generic-worker:os-group:aws-provisioner-v1/gecko-3-b-win2012/Administrators to role login-identity:mozilla-auth0/ad|Mozilla-LDAP|kmoir - see

  * https://tools.taskcluster.net/auth/roles/login-identity%3Amozilla-auth0%2Fad|Mozilla-LDAP|kmoir

If you log out of taskcluster tools site, and log back in again, then visit:

  * https://tools.taskcluster.net/credentials

then you should now see this scope in your list.

Note the generic-worker scopes don't apply to scriptworker-prov-v1/depsigning workers, since they are running scriptworker[1] rather than generic-worker[2]. Probably Aki can help you with those.

--

[1] https://github.com/mozilla-releng/scriptworker
[2] https://github.com/taskcluster/generic-worker

Comment 5

[Kim Moir [:kmoir] ET]

I tried to submit this task and got an error message that I was missing this scope.

generic-worker:allow-rdp:aws-provisioner-v1/gecko-3-b-win2012

Comment 6

[Pete Moore [:pmoore][:pete]]

Hey Kim,

Is it possible for you to use gecko-1-b-win2012? That would be a little safer security-wise, since you have Administrator on the machine, and RDP privileges, and the worker can potentially take production build jobs after you have finished with it (the workers continue to take jobs after the loan is completed).

We can look into implementing a feature that the worker self-terminates after loan if the task had Administrative privileges, but that currently isn't the case.

Another option is using the staging worker type:

gecko-1-b-win2012-beta

This worker type is for testing out worker changes (rather than testing out gecko changes) and you can run try jobs against it. The machine setup is handled by https://github.com/mozilla-releng/OpenCloudConfig and therefore you can test changes to the automated provisioning of the machines before committing those changes to actual level 1/2/3 workers. This could be a safer path that directly changing production workers.

If this works for you, I believe you should already have permission to RDP onto gecko-1-b-win2012-beta workers, but if not, please let me know.

The other problem with directly changing a gecko-3-b-win2012 worker, is that you can't influence which tasks it takes. It will claim available tasks for gecko-3-b-win2012 worker type, which can be real production jobs, but if you use the gecko-1-b-win2012-beta worker type, you can make changes, and then submit new tasks for it, and you know these tasks can only be picked up by machines in the same staging worker type pool.

If you like we can meet up to discuss this on vidyo etc, I realise this is a lot of information to dump in a bug that may be confusing or you may have questions about!

Thanks.

Comment 7

[Kim Moir [:kmoir] ET]

I tried your instructions and it is not working.  The task fails, here is the log

https://taskcluster-artifacts.net/WjJz4uS4TpCn-yxI2Jo3hQ/0/public/logs/live_backing.log

All I need is rdp acesss to a windows build machine to test out the commands for a set of patches I am writing + install some libraries as Administrator. 

I'm not changing the underlying worker type as part of these patches, just running some additional commands during the windows repackage phase. 

How do you access the gecko-1-b-win2012-beta workers?

I tried starting one but it just seems to be pending

https://tools.taskcluster.net/groups/cuMBwF2ZSlGIf87DmrW6vg/tasks/cuMBwF2ZSlGIf87DmrW6vg/runs/0

Comment 8

[Pete Moore [:pmoore][:pete]]

(In reply to Kim Moir [:kmoir] ET from comment #7)
> I tried your instructions and it is not working.  The task fails, here is
> the log
> 
> https://taskcluster-artifacts.net/WjJz4uS4TpCn-yxI2Jo3hQ/0/public/logs/
> live_backing.log
> 

Those errors are because a docker-worker payload has been provided to a generic-worker worker.

> All I need is rdp acesss to a windows build machine to test out the commands
> for a set of patches I am writing + install some libraries as Administrator. 
> 

ok.

> I'm not changing the underlying worker type as part of these patches, just
> running some additional commands during the windows repackage phase. 

ok.

> How do you access the gecko-1-b-win2012-beta workers?
> 
> I tried starting one but it just seems to be pending
> 
> https://tools.taskcluster.net/groups/cuMBwF2ZSlGIf87DmrW6vg/tasks/
> cuMBwF2ZSlGIf87DmrW6vg/runs/0

There is an issue in OpenCloudConfig that the first time a worker is spawned, it can take 20-30 minutes for a machine to be available and claiming jobs. That task has since been claimed and resolved by a worker, but was resolved as malformed-payload since it was a docker-worker payload for a generic-worker worker: https://tools.taskcluster.net/groups/cuMBwF2ZSlGIf87DmrW6vg/tasks/cuMBwF2ZSlGIf87DmrW6vg/details


See https://docs.taskcluster.net/docs/reference/workers/generic-worker/docs/payload for more information about what payload format generic-worker expects. Alternatively you should be able to edit an existing task for gecko-1-b-win2012, and change the worker type to gecko-1-b-win2012-beta before you submit it.

Note, the purpose of running against gecko-1-b-win2012-beta rather than any other worker type, is that you can then submit tasks against gecko-1-b-win2012-beta without impacting production traffic, which will allow you to test any changes you make on the worker.

The offer is still open to meet on vidyo if you want to walk through this in person.

Good luck!

Comment 9

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pmoore][:pete] from comment #8)

> There is an issue in OpenCloudConfig that the first time a worker is
> spawned, it can take 20-30 minutes for a machine to be available and
> claiming jobs.

See bug 1378383.

Comment 10

[Kim Moir [:kmoir] ET]

Okay thanks I was able to get a gecko-1-b-win2012-beta machine the way you suggested.  This page needed to be updated to reflect these instructions, it is not intuitive https://wiki.mozilla.org/ReleaseEngineering/How_To/Self_Provision_a_TaskCluster_Windows_Instance.

The current machine I'm using I don't have administrator privileges on because I didn't have scopes to assign them to myself.  

Could you give be Administrator scopes for gecko-1-b-win2012-beta?

Comment 11

[Kim Moir [:kmoir] ET]

Also, is there a way to make these machines more persistent - like last a week or so.  I spent several hours today setting up my instance to test my patches i.e. install new MozBuild, clone m-c, run local builds, and was getting ready to test my patches after supper only to discover my machine went away.  I don't have AWS privileges anymore so I can't set this up myself to be more persistent.

Comment 12

[Pete Moore [:pmoore][:pete]]

(In reply to Kim Moir [:kmoir] ET from comment #10)
> Okay thanks I was able to get a gecko-1-b-win2012-beta machine the way you
> suggested.  This page needed to be updated to reflect these instructions, it
> is not intuitive
> https://wiki.mozilla.org/ReleaseEngineering/How_To/Self_Provision_a_TaskCluster_Windows_Instance.

I don't see any modifications to that page - which changes need to be made? If you made any changes, it looks like they did not persist, as there is nothing in version history from you. If you find the page not intuitive, it would be helpful to provide information about what you consider would be more intuitive. Otherwise, your criticism is not constructive and does not serve to improve the situation.

It sounds like you are struggling to get set up, and I've offered several times to support you over vidyo. Please let's do that, rather than drawing this out any longer over bugmail.

(In reply to Kim Moir [:kmoir] ET from comment #10)
> Okay thanks I was able to get a gecko-1-b-win2012-beta machine the way you
> suggested.  This page needed to be updated to reflect these instructions, it
> is not intuitive
> https://wiki.mozilla.org/ReleaseEngineering/How_To/
> Self_Provision_a_TaskCluster_Windows_Instance.
> 
> The current machine I'm using I don't have administrator privileges on
> because I didn't have scopes to assign them to myself.  
> 
> Could you give be Administrator scopes for gecko-1-b-win2012-beta?

Done. See https://tools.taskcluster.net/auth/roles/login-identity%3Amozilla-auth0%2Fad|Mozilla-LDAP|kmoir

(In reply to Kim Moir [:kmoir] ET from comment #11)
> Also, is there a way to make these machines more persistent - like last a
> week or so.  I spent several hours today setting up my instance to test my
> patches i.e. install new MozBuild, clone m-c, run local builds, and was
> getting ready to test my patches after supper only to discover my machine
> went away.  I don't have AWS privileges anymore so I can't set this up
> myself to be more persistent.

Yes, apply any changes you want to make to https://github.com/mozilla-releng/OpenCloudConfig/blob/master/userdata/Manifest/gecko-1-b-win2012-beta.json and then your changes will persist. You can use an interactive session to work out what you want to install, but when you've decided, apply it to the manifest, and then the staging worker type will have your changes when you push your staging changes to the master branch of OpenCloudConfig, and the CI/CD has run.

But like I said above, I can talk you through this over vidyo, and we'll save a lot of time. If you feel the docs are not sufficient after this dicsussion, you are welcome to update the wiki or submit a PR for OpenCloudConfig or taskcluster docs site, or even generic-worker. Or if that is too much, you can also create a bug with details about what you feel is wrong, and somebody else can work on it. But then they need the details, not just "this is unintuitive".

Comment 13

[Kim Moir [:kmoir] ET]

I opened bug 1490309 to reflect the documentation changes.  I think bug 1489268 will serve my needs to get a reserved AWS instance that I use for testing.

Thanks for all your help.  I understand your frustration and I was frustrated too:-)  I did schedule a vidyo meeting with you yesterday for ET this morning to discuss this issue in person.  But when I got my instance working yesterday I cancelled it.  I think the only remaining issue is to have a reserved instance instead of a spot instance so I can continue to work without termination. I think the ciduty folks are the ones who can fix this because they have to setup the instance as reserved instead of spot.  I don't think I can change that since I don't have AWS permissions anymore.

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to Kim Moir [:kmoir] ET from comment #13)
> I think the only remaining issue is to
> have a reserved instance instead of a spot instance so I can continue to
> work without termination. I think the ciduty folks are the ones who can fix
> this because they have to setup the instance as reserved instead of spot.  I
> don't think I can change that since I don't have AWS permissions anymore.

Redirecting to CIDuty.