1147206 - generic-worker: write a generic task cluster worker in go (golang)

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Comment 1

[Pete Moore [:pmoore][:pete]]

Working on this here: https://github.com/petemoore/generic-worker

Comment 2

[Pete Moore [:pmoore][:pete]]

Attachment: GitHub Pull Request of code to-date

Hi guys,

Would you be able to take a look at my efforts so far and give your (honest) feedback? It is the beginnings, still a lot to do, but it has taken basic form.

I wanted to get a prototype up quickly, and now I will start writing some tests - currently there are no tests - <gulp>. But I'll pause adding any features while I catch up writing tests for what is there so far.

If you also would like me to issue a Pull Request for the Task Cluster go client, let me know.

Many thanks in advance for your help and support! :)

Pete

Comment 3

[Jonas Finnemann Jensen (:jonasfj)]

Comment on attachment 8582766 [details]
GitHub Pull Request of code to-date

Looks good so far... Some retries and a few opportunities to handle errors
more robustly by constructing the signedDeleteUrl as soon as possible...

One could probably design some abstractions better... But it's a lot of refactor for an initial working version.

What I would probably do is create a PollingManager as follows:
pm = PollingManager:new(...)
taskRun = pm.poll()

Then have that polling manager hide all logic related to azure queue, and queue.claimTask, and starting the reclaim logic which probably lives at taskRun level and not inside the polling manager..

---
But for now just a working version, the refactor later :)

Comment 4

[Pete Moore [:pmoore][:pete]]

This is getting pretty close now. For example, see:
https://tools.taskcluster.net/task-inspector/#IluJCfPqTF6UJ6SYIRb32g/

Here I have submitted a task to build the generic worker itself on windows, and it has been picked up by my test worker in AWS running on Windows Server 2008 R2. The task has been claimed, it has been run, and the status has been correctly reported. The log file and the binary artifact have been published as artifacts.

There are still some parts to be done:
1) integration with the aws provisioner
2) running the generic worker as a service

There will be improvements and new features added in Q3.

Comment 5

[Pete Moore [:pmoore][:pete]]

This is done now!

Released version 1.0.11 of the generic worker: https://github.com/taskcluster/generic-worker/releases

I've created the following worker type:
https://tools.taskcluster.net/aws-provisioner/#win2008r2-v1/

Example production task:
https://tools.taskcluster.net/task-inspector/#ZwNzGHJmTZGtWY_g4oG48g/0

Some docs:
http://docs.taskcluster.net/workers/generic-worker/