1363414 - Update taskcluster-worker config to use watchdog, etc.

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

from bug 1359209:

As for config see:
https://docs.taskcluster.net/reference/workers/taskcluster-worker/docs/configur
ation

plugins:
  watchdog: {} # this is sane (if any step takes more than 30 min we shutdown;
               # except the running step which is limited by maxruntime)
  maxruntime:
    maxRunTime: '3 hours' # tasks can never take more than 3 hours
    perTaskLimit: allow   # tasks may specify 'task.payload.maxRunTime' less
than 3 hours, if not it defaults to 3 hours
  logprefix: # new plugin just landed on master, will prefix task log with
taskId: ..., runId: ...
    hostname: {env: 'HOSTNAME'} # will prefix all tasks logs with: "hostname:
$HOSTNAME"
    machineType: 'mac-mini-g4'  # this is just free-form metadata to be
inserted at the beginning of task logs
    # you can add more here, maybe there is something neat from puppet like
machine name, etc that would be nice to have

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

logprefix is cool, but watchdog doesn't work yet - https://github.com/taskcluster/taskcluster-worker/issues/271

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1363414: add log prefixing;

Review commit: https://reviewboard.mozilla.org/r/137486/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137486/

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I'm figuring you're holding off on review here until watchdog is fixed -- which it is now?

If so, let me know what version it's fixed in, and I'll make a new patch to update to that version :)

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

Comment on attachment 8865900 [details]
Bug 1363414: add log prefixing;

https://reviewboard.mozilla.org/r/137486/#review145664

::: modules/taskcluster_worker/templates/taskcluster-worker.yml.erb:32
(Diff revision 1)
>      maxruntime:
>        maxRunTime: '96 hours'
>        perTaskLimit: allow
> +    #  if any step takes more than 30 min we shutdown, except the running step
> +    #  which is limited by maxruntime (doesn't work - https://github.com/taskcluster/taskcluster-worker/issues/271)
> +    # watchdog: {}

Fixed, feel free to create a new release :)

::: modules/taskcluster_worker/templates/taskcluster-worker.yml.erb:35
(Diff revision 1)
> +    #  if any step takes more than 30 min we shutdown, except the running step
> +    #  which is limited by maxruntime (doesn't work - https://github.com/taskcluster/taskcluster-worker/issues/271)
> +    # watchdog: {}
> +    logprefix:
> +      hostname: <%= scope.lookupvar('::fqdn') %>
> +      workerType: <%= @taskcluster_worker_type %>

Are you sure you don't want workerId instead?

Comment 5

[Jonas Finnemann Jensen (:jonasfj)]

@dustin, super sorry, I got logged out of bugzilla and didn't notice the request...
It all looks fine :)

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8865900 [details]
Bug 1363414: add log prefixing;

https://reviewboard.mozilla.org/r/137486/#review146142

::: modules/taskcluster_worker/templates/taskcluster-worker.yml.erb:35
(Diff revision 1)
> +    #  if any step takes more than 30 min we shutdown, except the running step
> +    #  which is limited by maxruntime (doesn't work - https://github.com/taskcluster/taskcluster-worker/issues/271)
> +    # watchdog: {}
> +    logprefix:
> +      hostname: <%= scope.lookupvar('::fqdn') %>
> +      workerType: <%= @taskcluster_worker_type %>

The workerId is a substring of the hostname in this case.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I bumped the version to 0.1.2 and un-outcommented the watchdog config, and landed it.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

https://hg.mozilla.org/build/puppet/rev/008027f47da7314d25b3ab8369f49457a03b2d5d