Last Comment Bug 1147206 - generic-worker: write a generic task cluster worker in go (golang)
: generic-worker: write a generic task cluster worker in go (golang)
Status: RESOLVED FIXED
:
Product: Taskcluster
Classification: Other
Component: Generic-Worker (show other bugs)
: unspecified
: x86 Mac OS X
-- normal
: ---
Assigned To: Pete Moore [:pmoore][:pete]
:
:
Mentors:
: 1119546 (view as bug list)
Depends on: 1147209 1149325
Blocks: 1119546 1156304 1180775
  Show dependency treegraph
 
Reported: 2015-03-24 16:42 PDT by Pete Moore [:pmoore][:pete]
Modified: 2015-07-14 01:32 PDT (History)
4 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
GitHub Pull Request of code to-date (56 bytes, text/plain)
2015-03-24 16:57 PDT, Pete Moore [:pmoore][:pete]
jopsen: feedback+
Details

Description User image Pete Moore [:pmoore][:pete] 2015-03-24 16:42:44 PDT

    
Comment 1 User image Pete Moore [:pmoore][:pete] 2015-03-24 16:49:22 PDT
Working on this here: https://github.com/petemoore/generic-worker
Comment 2 User image Pete Moore [:pmoore][:pete] 2015-03-24 16:57:36 PDT
Created attachment 8582766 [details]
GitHub Pull Request of code to-date

Hi guys,

Would you be able to take a look at my efforts so far and give your (honest) feedback? It is the beginnings, still a lot to do, but it has taken basic form.

I wanted to get a prototype up quickly, and now I will start writing some tests - currently there are no tests - <gulp>. But I'll pause adding any features while I catch up writing tests for what is there so far.

If you also would like me to issue a Pull Request for the Task Cluster go client, let me know.

Many thanks in advance for your help and support! :)

Pete
Comment 3 User image Jonas Finnemann Jensen (:jonasfj) 2015-03-25 11:47:15 PDT
Comment on attachment 8582766 [details]
GitHub Pull Request of code to-date

Looks good so far... Some retries and a few opportunities to handle errors
more robustly by constructing the signedDeleteUrl as soon as possible...

One could probably design some abstractions better... But it's a lot of refactor for an initial working version.

What I would probably do is create a PollingManager as follows:
pm = PollingManager:new(...)
taskRun = pm.poll()

Then have that polling manager hide all logic related to azure queue, and queue.claimTask, and starting the reclaim logic which probably lives at taskRun level and not inside the polling manager..

---
But for now just a working version, the refactor later :)
Comment 4 User image Pete Moore [:pmoore][:pete] 2015-06-09 13:28:43 PDT
This is getting pretty close now. For example, see:
https://tools.taskcluster.net/task-inspector/#IluJCfPqTF6UJ6SYIRb32g/

Here I have submitted a task to build the generic worker itself on windows, and it has been picked up by my test worker in AWS running on Windows Server 2008 R2. The task has been claimed, it has been run, and the status has been correctly reported. The log file and the binary artifact have been published as artifacts.

There are still some parts to be done:
1) integration with the aws provisioner
2) running the generic worker as a service

There will be improvements and new features added in Q3.
Comment 5 User image Pete Moore [:pmoore][:pete] 2015-06-25 12:16:18 PDT
This is done now!

Released version 1.0.11 of the generic worker: https://github.com/taskcluster/generic-worker/releases

I've created the following worker type:
https://tools.taskcluster.net/aws-provisioner/#win2008r2-v1/

Example production task:
https://tools.taskcluster.net/task-inspector/#ZwNzGHJmTZGtWY_g4oG48g/0

Some docs:
http://docs.taskcluster.net/workers/generic-worker/
Comment 6 User image Pete Moore [:pmoore][:pete] 2015-06-25 12:17:53 PDT
*** Bug 1119546 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.