Closed Bug 1575845 Opened 6 years ago Closed 6 years ago

create worker-manager gcp providers

Categories

(Taskcluster :: Operations and Service Requests, task)

Product:
Taskcluster
Component:
Operations and Service Requests
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: grenade, Unassigned)

Details

Attachments

(2 files)

add gecko 1 & 3 gcp windows builders
47 bytes, text/x-phabricator-request
Details | Review
corrections to gcp windows gw config for gecko workers
47 bytes, text/x-phabricator-request
Details | Review

please create some new worker-manager gcp providers mapped to the following gcp projects (provider-name: gcp-project-name [gcp-folder]):

  • gecko-1: fx-workers-tier1-prod [firefox.gcp.mozilla.com/relops/applications/fx-workers-tier1]
  • gecko-3: fx-workers-tier3-prod [firefox.gcp.mozilla.com/relops/applications/fx-workers-tier3]
  • gecko-t: fx-workers-test-prod [firefox.gcp.mozilla.com/relops/applications/fx-workers-test]
  • sandbox-1: fx-workers-tier1-nonprod [firefox.gcp.mozilla.com/relops/applications/fx-workers-tier1]
  • sandbox-3: fx-workers-tier3-nonprod [firefox.gcp.mozilla.com/relops/applications/fx-workers-tier3]
  • sandbox-t: fx-workers-test-nonprod [firefox.gcp.mozilla.com/relops/applications/fx-workers-test]

the provider naming might still be up for debate. this is just my interpretation of the recent email dialogue on this subject.

service accounts created and roles applied (as per https://docs.taskcluster.net/docs/reference/core/worker-manager/google):

sender_fingerprint=0x000
recipient_fingerprint=0x111
worker_service_account_name=taskcluster-worker
worker_service_account_display_name="taskcluster worker"
manager_service_account_name=taskcluster-worker-manager
manager_service_account_display_name="taskcluster worker manager"
for project_name in fx-workers-tier1-prod fx-workers-tier3-prod fx-workers-test-prod fx-workers-tier1-nonprod fx-workers-tier3-nonprod fx-workers-test-nonprod; do
  gcloud iam service-accounts create ${worker_service_account_name} --display-name "${worker_service_account_display_name}" --project ${project_name}
  gcloud iam service-accounts create ${manager_service_account_name} --display-name "${service_account_display_name}" --project ${project_name}
  gcloud iam service-accounts keys create /tmp/${project_name}_${manager_service_account_name}.json --iam-account ${manager_service_account_name}@${project_name}.iam.gserviceaccount.com
  gpg2 -e -u ${sender_fingerprint} -r ${recipient_fingerprint} /tmp/${project_name}_${manager_service_account_name}.json
  rm -f /tmp/${project_name}_${manager_service_account_name}.json
  gcloud iam service-accounts add-iam-policy-binding ${worker_service_account_name}@${project_name}.iam.gserviceaccount.com --member serviceAccount:${manager_service_account_name}@${project_name}.iam.gserviceaccount.com --role roles/iam.serviceAccountUser --project ${project_name}
  for role in iam.serviceAccountUser compute.admin; do
    gcloud projects add-iam-policy-binding ${project_name} --member serviceAccount:${manager_service_account_name}@${project_name}.iam.gserviceaccount.com --role roles/${role}
  done
  for api in iam compute; do
    gcloud services enable ${api}.googleapis.com --project ${project_name}
  done
done

ok, PROVIDERS updated for Heroku app taskcluster-worker-manager.

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED

Rob, by the way, ci-admin and ci-configuration already support creating WorkerPool objects, so please use those to create the corresponding worker pools.

(In reply to Dustin J. Mitchell [:dustin] (he/him) from comment #3)

Rob, by the way, ci-admin and ci-configuration already support creating WorkerPool objects, so please use those to create the corresponding worker pools.

i've found worker-pools.yml after some poking around (there are dead links here & here). however, i will probably need to go through a fair bit of trial and error in gcp pool definitions and worker configuration before i have configurations that work as required. because of the time difference, that would mean a few weeks of review cycles just to experiment and discover what works. it will probably make sense for me to just modify the definitions in real time until we have something that works to commit to the ci-configuration repo.

having said that, both the ui and the api (using the tc cli) are returning server errors this morning so i probably won't be changing any worker pools until that's sorted.

You can use ci-admin and ci-configuration to do those modifications directly without reviewing each change - it's a way to reliably call the API that will be future-portable to later maintenance. I don't want to get to a place where we're trying to back-port manual modifications into ci-configuration, or end up with a situation where releng manages half of its workers with one tool and half with another.

I'll have a look at the API errors now. I think I just forgot that worker-manager hadn't been updated in a while so I've deployed it just now.

Ah

2019-08-23T12:36:24.960455+00:00 app[web.1]: Error: Identity and Access Management (IAM) API has not been used in project 235672186542 before or it is disabled. Enable it by visiting https://console.cloud.google.com/apis/api/iam.googleapis.com/overview?project=235672186542 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

I expect that's the case for all of the projects, and note that the docs give several APIs that must be enabled.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

(In reply to Dustin J. Mitchell [:dustin] (he/him) from comment #6)

I expect that's the case for all of the projects, and note that the docs give several APIs that must be enabled.

i have enabled iam and compute (as shown in my edited comment 1 script above). are there others that should be enabled?

That should be it..

Flags: needinfo?(dustin)

Setting PROVIDERS and restarting ⬢ taskcluster-worker-manager... done, v35

whoops:

2019-09-10T16:25:02.995621+00:00 app[web.1]: AssertionError [ERR_ASSERTION]: Must provide a workerServiceAccountId to google providers

Grenade, we now need an additional service account in each project for the workers to run as:
https://docs.taskcluster.net/docs/manual/deploying/workers#service-accounts
Note that there are now fewer permissions required for the existing worker-manager serviceAccounts (just compute admin and service account user).

Flags: needinfo?(dustin)
Flags: needinfo?(rthijssen)

thanks dustin! i've added a "taskcluster-worker" service account to each project to complement the "taskcluster-worker-manager" accounts created previously.

Flags: needinfo?(rthijssen)

Sorry, I'll need the numeric uniqueId for each of them :(

Great success! I see the six new providers in the pop-up in the UI now.

Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → FIXED

Ah, before I forget, can we ensure that we've reduced the permissions of the worker-manager service account to just the ones listed here now? The old way was much more powerful.

Flags: needinfo?(rthijssen)

Augh, I had the providers config in my clipboard today and failed copying a new value before I search for it in the gcp console. Who the hell knows what logs that ended up in within google. Not urgent since these aren't in use yet but could you rotate the creds for these 6 and pass them back to us? Sorry about that.

(In reply to Brian Stack [:bstack] from comment #15)

Ah, before I forget, can we ensure that we've reduced the permissions of the worker-manager service account to just the ones listed here now? The old way was much more powerful.

done. like so:

manager_service_account_name=taskcluster-worker-manager
for project_name in fx-workers-tier1-prod fx-workers-tier3-prod fx-workers-test-prod fx-workers-tier1-nonprod fx-workers-tier3-nonprod fx-workers-test-nonprod; do
  for role in iam.serviceAccountAdmin iam.roleAdmin resourcemanager.projectIamAdmin; do
    gcloud projects remove-iam-policy-binding ${project_name} --member serviceAccount:${manager_service_account_name}@${project_name}.iam.gserviceaccount.com --role roles/${role}
  done
  for role in iam.serviceAccountUser compute.admin; do
    gcloud projects add-iam-policy-binding ${project_name} --member serviceAccount:${manager_service_account_name}@${project_name}.iam.gserviceaccount.com --role roles/${role}
  done
done
Flags: needinfo?(rthijssen)

(In reply to Brian Stack [:bstack] from comment #16)

Augh, I had the providers config in my clipboard today and failed copying a new value before I search for it in the gcp console. Who the hell knows what logs that ended up in within google. Not urgent since these aren't in use yet but could you rotate the creds for these 6 and pass them back to us? Sorry about that.

keys rotated and sent to bstack

# key rotation script
sender_fingerprint=0x000
recipient_fingerprint=0x111
gpg2 --recv-keys ${recipient_fingerprint} --keyserver gpg.mozilla.org
manager_service_account_name=taskcluster-worker-manager
for project_name in fx-workers-tier1-prod fx-workers-tier3-prod fx-workers-test-prod fx-workers-tier1-nonprod fx-workers-tier3-nonprod fx-workers-test-nonprod; do
  # delete all old keys for the service account
  while read -r key_name; do
    echo deleting key id: "${key_name:(-40)}" for iam account ${manager_service_account_name}@${project_name}.iam.gserviceaccount.com
    gcloud iam service-accounts keys delete "${key_name:(-40)}" --iam-account=${manager_service_account_name}@${project_name}.iam.gserviceaccount.com --quiet
  done <<< "$(gcloud iam service-accounts keys list --iam-account ${manager_service_account_name}@${project_name}.iam.gserviceaccount.com --format json | jq -r '.[].name')"
  # create and encrypt a new key
  rm -f /tmp/${project_name}_${manager_service_account_name}.json /tmp/${project_name}_${manager_service_account_name}.json.gpg
  gcloud iam service-accounts keys create /tmp/${project_name}_${manager_service_account_name}.json --iam-account ${manager_service_account_name}@${project_name}.iam.gserviceaccount.com
  gpg2 -e -u ${sender_fingerprint} -r ${recipient_fingerprint} /tmp/${project_name}_${manager_service_account_name}.json
  rm -f /tmp/${project_name}_${manager_service_account_name}.json
done
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: