generic-worker: write a generic task cluster worker in go (golang)

RESOLVED FIXED

Status

Taskcluster
Generic-Worker
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: pmoore, Assigned: pmoore)

Tracking

Details

Attachments

(1 attachment)

Comment hidden (empty)
(Assignee)

Updated

2 years ago
Depends on: 1147209
(Assignee)

Comment 1

2 years ago
Working on this here: https://github.com/petemoore/generic-worker
(Assignee)

Comment 2

2 years ago
Created attachment 8582766 [details]
GitHub Pull Request of code to-date

Hi guys,

Would you be able to take a look at my efforts so far and give your (honest) feedback? It is the beginnings, still a lot to do, but it has taken basic form.

I wanted to get a prototype up quickly, and now I will start writing some tests - currently there are no tests - <gulp>. But I'll pause adding any features while I catch up writing tests for what is there so far.

If you also would like me to issue a Pull Request for the Task Cluster go client, let me know.

Many thanks in advance for your help and support! :)

Pete
Assignee: nobody → pmoore
Status: NEW → ASSIGNED
Attachment #8582766 - Flags: feedback?(jopsen)
Attachment #8582766 - Flags: feedback?(garndt)
Comment on attachment 8582766 [details]
GitHub Pull Request of code to-date

Looks good so far... Some retries and a few opportunities to handle errors
more robustly by constructing the signedDeleteUrl as soon as possible...

One could probably design some abstractions better... But it's a lot of refactor for an initial working version.

What I would probably do is create a PollingManager as follows:
pm = PollingManager:new(...)
taskRun = pm.poll()

Then have that polling manager hide all logic related to azure queue, and queue.claimTask, and starting the reclaim logic which probably lives at taskRun level and not inside the polling manager..

---
But for now just a working version, the refactor later :)
Attachment #8582766 - Flags: feedback?(jopsen) → feedback+
(Assignee)

Updated

2 years ago
Depends on: 1149325
(Assignee)

Updated

2 years ago
Blocks: 1156304
(Assignee)

Updated

2 years ago
Attachment #8582766 - Flags: feedback?(garndt)
(Assignee)

Comment 4

2 years ago
This is getting pretty close now. For example, see:
https://tools.taskcluster.net/task-inspector/#IluJCfPqTF6UJ6SYIRb32g/

Here I have submitted a task to build the generic worker itself on windows, and it has been picked up by my test worker in AWS running on Windows Server 2008 R2. The task has been claimed, it has been run, and the status has been correctly reported. The log file and the binary artifact have been published as artifacts.

There are still some parts to be done:
1) integration with the aws provisioner
2) running the generic worker as a service

There will be improvements and new features added in Q3.
(Assignee)

Comment 5

2 years ago
This is done now!

Released version 1.0.11 of the generic worker: https://github.com/taskcluster/generic-worker/releases

I've created the following worker type:
https://tools.taskcluster.net/aws-provisioner/#win2008r2-v1/

Example production task:
https://tools.taskcluster.net/task-inspector/#ZwNzGHJmTZGtWY_g4oG48g/0

Some docs:
http://docs.taskcluster.net/workers/generic-worker/
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Updated

2 years ago
Duplicate of this bug: 1119546

Updated

2 years ago
Blocks: 1180775
(Assignee)

Updated

2 years ago
Component: TaskCluster → Generic-Worker
Product: Testing → Taskcluster
You need to log in before you can comment on or make changes to this bug.