1552510 - `terraform apply` doesn't work

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Hassan Ali (:hassan)]

I don't seem to be able to run terraform apply successfully. I suspect it is because of https://github.com/taskcluster/taskcluster-mozilla-terraform/commit/3f2f01ab1d4de1b9512bf63b24883a7652b3e0a8 was merged. When I try to run taskcluster-staging-net deployment terraform apply throws:

Error: Error applying plan:

18 error(s) occurred:

* module.gke.google_project_iam_member.team_members_cluster_admin[1] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members_cluster_admin.1: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members[1] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members.1: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members[5] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members.5: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members_cluster_admin[5] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members_cluster_admin.5: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.taskcluster.google_project_iam_member.worker_manager_role_accounts: 1 error(s) occurred:

* google_project_iam_member.worker_manager_role_accounts: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members[3] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members.3: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members[6] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members.6: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.taskcluster.google_project_iam_member.worker_manager_role_policies: 1 error(s) occurred:

* google_project_iam_member.worker_manager_role_policies: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members_cluster_admin[6] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members_cluster_admin.6: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members_cluster_admin[4] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members_cluster_admin.4: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.taskcluster.google_project_iam_member.worker_manager_role_roles: 1 error(s) occurred:

* google_project_iam_member.worker_manager_role_roles: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members_cluster_admin[3] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members_cluster_admin.3: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members_cluster_admin[7] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members_cluster_admin.7: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members[2] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members.2: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.taskcluster.google_project_iam_member.worker_manager_role_instances: 1 error(s) occurred:

* google_project_iam_member.worker_manager_role_instances: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members[7] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members.7: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members[4] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members.4: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden
* module.gke.google_project_iam_member.team_members_cluster_admin[2] (destroy): 1 error(s) occurred:

* google_project_iam_member.team_members_cluster_admin.2: Error applying IAM policy for project "taskcluster-staging-net": Error setting IAM policy for project "taskcluster-staging-net": googleapi: Error 403: The caller does not have permission, forbidden

I tried doing what's in https://cloud.google.com/docs/authentication/getting-started but that didn't seem to work.

Comment 1

[Hassan Ali (:hassan)]

bstack, does terraform apply work for you by any chance?

Comment 2

[Brian Stack [:bstack]]

It has allowed me to apply. I made a couple of updates to it in the mean time. Can you try pulling master, updating the submodule and trying again?

Comment 3

[Hassan Ali (:hassan)]

Rebasing and updating the submodule fixed the issue. Thanks for the quick fix bstack!

Comment 4

[Hassan Ali (:hassan)]

Apply Complete!

Outputs:

cluster_ip = 35.229.108.89
cluster_url = https://taskcluster-staging.net
root_access_token = <sensitive>
websocktunnel_secret = <sensitive>

Navigating to 35.229.108.89 doesn't seem to show the UI. The connection times out. Is there anything else I need to do to have it show up?

Comment 5

[Brian Stack [:bstack]]

For future visitors of this bug, visiting the ip addr directly won't show anything due to how the ingress is configured.