1587554 - Migrate generic-worker, tc-lib-artifact-go CI to community deployment

Status: RESOLVED FIXED

(Taskcluster :: Operations and Service Requests, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This will need reconfiguration of a bunch of "manual" workers, and creation of some Windows workers in the community-tc deployment.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

It looks like https://github.com/taskcluster/taskcluster-lib-artifact-go/blob/master/.taskcluster.yml refers to a lot of the same workers, so please migrate that as well.

Comment 2

[Pete Moore [:pmoore][:pete]]

Attachment: GitHub Pull Request for generic-worker

In essence there are three parts to this PR:

1. Decision Task

The huge .taskcluster.yml is gone and replaced with a single decision task. I did this to reduce the risk of inconsistencies in CI between the different platforms (no more copy/paste between task definitions), and to pave the way for us to introduce a try commit syntax in a future PR.

2. No more hardcoded taskIds integration task dependencies

Rather than integration tests depending on pre-run tasks in the cluster, the integration tests now generate the test dependency on the fly if it doesn't exist. When the CI lived in taskcluster.net deployment, three different tasks were created, that tests depended on. These tasks had some artifacts that the integration tests could mount, for example. Migrating to the new cluster broke these tests.

Rather than just recreate them in the new cluster, I decided it was better to generate them on the fly if they don't exist. The tasks were documented already (those docs get deleted as part of this PR), but generating-on-the-fly seems better. So all the tasks that depended on these hardcoded taskIds in the past now don't have the hardcoded taskId, but instead call a method that generates the task if it doesn't exist, or just return the taskId if it does exist.

3. Migration to the new community-tc cluster

Obviously, everything needed to be ported to the new cluster. This PR includes all the changes for that to happen. See the worker overview page to see the new workers in action. At the same time I've extended test coverage to include Windows 10 on ARM, and Linux on ARM too (using a raspberry pi) and migrated the mac stadium worker to the new cluster too. We've lost Windows 7 support for now, but when we have a solution for Windows 7 (32 bit) we can add it back in by uncommenting a couple of lines in the new tasks.yml file. I've also manually set up a Windows 10 worker in Azure to give us Windows 10 coverage until we have an Azure provider that can spawn instances for us in Worker Manager.

Comment 3

[Chris Cooper [:coop] (he/him)]

Pete: is there anything left to do here?

Comment 4

[Pete Moore [:pmoore][:pete]]

All done, thanks!