1593749 - Migrate code-review CI to community taskcluster deployment, code-review hooks to firefox deployment

Status: RESOLVED FIXED

(Taskcluster :: Operations and Service Requests, task)

Description

[Bastien Abadie [:bastien]]

The project uses Taskcluster for its CI needs, but also needs to access some parts of the Mozilla CI:

• public artifacts from analysis tasks
• the bot itself is triggered by a Pulse message sent from a task on Mozilla CI

Comment 1

[Marco Castelluccio [:marco]]

As we discussed on IRC, the CI for the project can be in the community taskcluster deployment, the hooks should be in the firefox deployment.

Comment 2

[Bastien Abadie [:bastien]]

I made a PR to test the new TC instance, simply updating the provisionerId & workerType: tasks run on the community instance.

Docker build tasks fail because they do not run on a privileged worker: sample failure task

Brian, could you help me out here ?

Comment 3

[Brian Stack [:bstack]]

I believe https://github.com/mozilla/community-tc-config/pull/91 will fix this issue. Thanks for finding it

Comment 4

[Bastien Abadie [:bastien]]

I merged the migration PR.

We still need a way to update firefox-ci hooks :)

Comment 5

[Bastien Abadie [:bastien]]

These hooks need to be migrated on the Firefox-CI instance before the 9th so that the code-review bot runs continously :

• project-relman/code-review-testing
• project-relman/code-review-production

They each need access to their respective secrets on the same instance:

• project/relman/code-review/runtime-testing
• project/relman/code-review/runtime-production

Most importantly they both need to be triggered by the following pulse binding: exchange/taskcluster-queue/v1/task-completed with the route route.project.relman.codereview.v1.try_ending

Comment 6

[Brian Stack [:bstack]]

I think this is all deployed now!

Comment 7

[Bastien Abadie [:bastien]]

The code-review hooks looks OK to me, the testing hook was triggered once but failed due to a deadline exception. I guess it's normal at this stage ?

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

drop-by comment: that deadline exception is because the task has the wrong workerType -- should be project-relman/releng-svc-memory, probably. That's the worker-type used in e.g., https://firefox-ci-tc.services.mozilla.com/hooks/project-relman/code-coverage-repo-testing

Comment 9

[Brian Stack [:bstack]]

Attachment: Bug 1593749 - Update relman workerpools to make sense

Comment 10

[Bastien Abadie [:bastien]]

The last code-review tasks on firefox-ci are still in the exception state (fire status is OK though) : https://firefox-ci-tc.services.mozilla.com/hooks/project-relman/code-review-production

Comment 11

[Marco Castelluccio [:marco]]

The hook has not been triggered since yesterday, which means there is something broken with the trigger code.

Indeed, the code-review-events app on Heroku is crashing.

Probably bug 1595464 is all we need (at least to fix the trigger issue).

Comment 12

[Bastien Abadie [:bastien]]

Could you create those 3 clients and send me their access tokens through email (here is my GPG public key)

• project/relman/code-review/backend-testing
• project/relman/code-review/events-production
• project/relman/code-review/events-testing

Comment 13

[Johan Lorenzo [:jlorenzo]]

Discussed with Bastien over Slack. This project, while developed on Github and thus using the Community TC instance, runs jobs on Firefox CI. The clients requested in comment 12 are needed in Firefox CI.

I then created them:

• https://firefox-ci-tc.services.mozilla.com/auth/clients/project%2Frelman%2Fcode-review%2Fbackend-testing
• https://firefox-ci-tc.services.mozilla.com/auth/clients/project%2Frelman%2Fcode-review%2Fevents-production
• https://firefox-ci-tc.services.mozilla.com/auth/clients/project%2Frelman%2Fcode-review%2Fevents-testing

These clients required these 2 roles, which I created:

• https://firefox-ci-tc.services.mozilla.com/auth/roles/project%3Arelman%3Acode-review%2Fruntime%2Ftesting
• https://firefox-ci-tc.services.mozilla.com/auth/roles/project%3Arelman%3Acode-review%2Fruntime%2Fproduction

Bastien confirmed these clients have been deployed and services are back working. Do you think we can close this ticket, Bastien?

Comment 14

[Bastien Abadie [:bastien]]

Not yet, the service is not back to normal (clients are fine though, thanks a lot !)

Comment 15

[Bastien Abadie [:bastien]]

Attachment: Bug 1593749 - Update code-review hook docker image to 1.0.6

Comment 16

[Bastien Abadie [:bastien]]

Attachment: Bug 1593749 - Update code-review hooks docker image to avoid Bug 1595838, r=callek

Comment 17

[Bastien Abadie [:bastien]]

Attachment: Bug 1593749 - Add scopes to code-review runtime roles, r=tomprince

Comment 18

[Bastien Abadie [:bastien]]

Code review bot now has all the necessary clients & scopes set in ci-configuration, things are running OK.

One last step is to restore the bugbug hook trigger (hosted on community instance) from code-review (hosted on firefox-ci instance).
I already made a patch to use a community Taskcluster client from the code-review code - it's currently running in production & testing.

We simply need a firefox-ci admin to update our secrets with the following payload (tokens to be sent directly through send.firefox.com):

1. Edit the events section in the testing secret, and add the following payload:

events:
  community_config:
    client_id: project/relman/bugbug/code-review-testing
    access_token: xxx
  ...

2. Edit the events section in the production secret, and add the following payload:

events:
  community_config:
    client_id: project/relman/bugbug/code-review-production
    access_token: yyy
  ...

Comment 19

[Justin Wood (:Callek)]

Deployed the updated secrets just now.

Comment 20

[Bastien Abadie [:bastien]]

I made a mistake, the required key is taskcluster_community (not community_config). Sorry about that :(

Comment 21

[Justin Wood (:Callek)]

(In reply to Bastien Abadie [:bastien] from comment #20)

I made a mistake, the required key is taskcluster_community (not community_config). Sorry about that :(

Fixed

Comment 22

[Bastien Abadie [:bastien]]

Code review bot has been running happily for the last 36 hours, thanks all !

Comment 23

[Johan Lorenzo [:jlorenzo]]

Clearing NI per comment 21 and comment 22.