Closed Bug 1147206 Opened 8 years ago Closed 7 years ago

generic-worker: write a generic task cluster worker in go (golang)


(Taskcluster :: Workers, defect)

Not set


(Not tracked)



(Reporter: pmoore, Assigned: pmoore)




(1 file)

No description provided.
Depends on: 1147209
Hi guys,

Would you be able to take a look at my efforts so far and give your (honest) feedback? It is the beginnings, still a lot to do, but it has taken basic form.

I wanted to get a prototype up quickly, and now I will start writing some tests - currently there are no tests - <gulp>. But I'll pause adding any features while I catch up writing tests for what is there so far.

If you also would like me to issue a Pull Request for the Task Cluster go client, let me know.

Many thanks in advance for your help and support! :)

Assignee: nobody → pmoore
Attachment #8582766 - Flags: feedback?(jopsen)
Attachment #8582766 - Flags: feedback?(garndt)
Comment on attachment 8582766 [details]
GitHub Pull Request of code to-date

Looks good so far... Some retries and a few opportunities to handle errors
more robustly by constructing the signedDeleteUrl as soon as possible...

One could probably design some abstractions better... But it's a lot of refactor for an initial working version.

What I would probably do is create a PollingManager as follows:
pm = PollingManager:new(...)
taskRun = pm.poll()

Then have that polling manager hide all logic related to azure queue, and queue.claimTask, and starting the reclaim logic which probably lives at taskRun level and not inside the polling manager..

But for now just a working version, the refactor later :)
Attachment #8582766 - Flags: feedback?(jopsen) → feedback+
Depends on: 1149325
Blocks: 1156304
Attachment #8582766 - Flags: feedback?(garndt)
This is getting pretty close now. For example, see:

Here I have submitted a task to build the generic worker itself on windows, and it has been picked up by my test worker in AWS running on Windows Server 2008 R2. The task has been claimed, it has been run, and the status has been correctly reported. The log file and the binary artifact have been published as artifacts.

There are still some parts to be done:
1) integration with the aws provisioner
2) running the generic worker as a service

There will be improvements and new features added in Q3.
This is done now!

Released version 1.0.11 of the generic worker:

I've created the following worker type:

Example production task:

Some docs:
Closed: 7 years ago
Resolution: --- → FIXED
Duplicate of this bug: 1119546
Component: TaskCluster → Generic-Worker
Product: Testing → Taskcluster
Component: Generic-Worker → Workers
You need to log in before you can comment on or make changes to this bug.