Closed Bug 1490473 Opened 7 years ago Closed 7 years ago

test and deploy taskcluster-pulse monitor

Categories

(Taskcluster :: Services, enhancement)

Product:
Taskcluster
Component:
Services
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

References

Details

We're nervous that this is going to "accidentally" delete all pulse users; that would be pretty bad. So the initial deployment will be with the actual deletion mocked out and monitored.
Blocks: 1436456
}dustin@jemison ~ $ for svc in taskcluster-auth taskcluster-aws-provisioner-staging taskcluster-aws-provisioner taskcluster-github taskcluster-gitub-staging taskcluster-notify taskcluster-purge-cache taskcluster-queue taskcluster-scheduler taskcluster-treherder-staging taskcluster-treeherder taskcluster-events; do echo '{"expires": "3000-01-01T01:00:00.00Z"}' | taskcluster api pulse claimNamespace $svc; done helped :) With that finished, it wants to delete a bunch of entries from its Azure table (which makes sense and is harmless) and the following exchanges, all of which make sense: Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.683Z maintenance.handle-exchanges Deleting exchange/taskcluster-auth-kube-staging/v1/client-created because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-auth-kube-staging/v1/client-deleted because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-auth-kube-staging/v1/client-updated because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-auth-kube-staging/v1/role-created because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-auth-kube-staging/v1/role-deleted because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-auth-kube-staging/v1/role-updated because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-client-tester/taskcluster-client/really-simple-test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-client-tester/taskcluster-client/simple-test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-client-tester/taskcluster-client/test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-client-tester/taskcluster-client/test/really-simple-test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-client-tester/taskcluster-client/test/simple-test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-client-tester/taskcluster-client/test/test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-dustin-test/v1/test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-hooks-pr-74/hook-created because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.684Z maintenance.handle-exchanges Deleting exchange/taskcluster-hooks-pr-74/hook-deleted because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.685Z maintenance.handle-exchanges Deleting exchange/taskcluster-hooks-pr-74/hook-updated because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.685Z maintenance.handle-exchanges Deleting exchange/taskcluster-pulse-publisher-test/test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.685Z maintenance.handle-exchanges Deleting exchange/taskcluster-tests-CPIrhszKS56bzLoSSJc2xA/test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.685Z maintenance.handle-exchanges Deleting exchange/taskcluster-tests-QAJw8sMfQ5yCvKEEdTwV2A/test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.685Z maintenance.handle-exchanges Deleting exchange/taskcluster-tests-ShEe6-JkSCaybDEhbdGWVw/test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.685Z maintenance.handle-exchanges Deleting exchange/taskcluster-tests-Xrn7ASZ0Qs2rfH7Dw3OLpQ/test-exchange because associated namespace is expired! Sep 17 17:26:54 taskcluster-pulse app/monitorRabbit.1: 2018-09-17T17:26:54.685Z maintenance.handle-exchanges Deleting exchange/taskcluster-tests-cEXTZMRKQUqfg85luEb2Yw/test-exchange because associated namespace is expired! Notably, it's not touching anything not taskcluster-related!
https://sentry.prod.mozaws.net/operations/taskcluster-pulse/issues/4717572/ StatusCodeError: 401 - "{\"error\":\"not_authorised\",\"reason\":\"ACCESS_REFUSED - access to exchange 'exchange/taskcluster-auth-kube-staging/v1/client-created' in vhost '/' refused for user 'taskcluster'\"}"
Ah, it has permissions ^(queue/taskcluster-garbage-.*|exchange/taskcluster-garbage-.*) so apparently I've taken a belt-and-suspenders-with-a-belt approach here :)
ugh, that command line missed taskcluster-index. Fixed.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
I had left the monitor process disabled. I just re-enabled it, after adding namespaces for tc-hooks (added since the last comment) and taskcluster-task-analysis (not sure why this wasn't spotted..)
Component: Platform and Services → Services
You need to log in before you can comment on or make changes to this bug.