Closed Bug 1017473 Opened 11 years ago Closed 7 years ago

[docker-worker] Operational logs per-task

Categories

(Taskcluster :: Workers, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: jlal, Unassigned)

Details

(Whiteboard: [docker-worker])

Having some kind of log per task which gives insight into the running state of the task would be both cool and useful. I am not sure right not if this is just logs (like X thing happened) or both logs and stats (which would be easier to read as a human but maybe less useful to machines). Things we could place here (we can't dump all of information for security reasons probably) - task claimed - heartbeats / timeouts - memory consumption - cpu usage? - other useful crap from docker top? - metrics related to artifact uploads, etc.. There are a bunch of other issues around this: - how long do we keep them - who consumes them - how to place items in here without security issues This does _not_ replace the usecase for something like graphite (which we should still use) but that will report data on a host/worker level.
Why not attach these to the task. Currently tasks only have "terminal.log", why not add one called "docker-worker.log" and only report things that are specific to the task. Like cpu and ram used by the task at every 5min interval. Or artifacts uploaded (or attempted uploaded), and upload time. This would also be a good place to put the "docker pull"-log which doesn't really belong in "terminal.log". I prefer "terminal.log" being exactly what you get when you run the task locally. Logs outside of loggly (syslog) for host/worker-level doesn't seem useful to me. It would be better to add graphite or some other magic then.
>Why not attach these to the task. Currently tasks only have "terminal.log", why not add one called "docker-worker.log" and only report things that are specific to the task. Like cpu and ram used by the task at every 5min interval. Or artifacts uploaded (or attempted uploaded), and upload time. This is exactly what I am proposing. Log in the sense of our task "logs" not in the sense of syslogs which are useless unless reported somewhere (we don't need to talk about this). >This would also be a good place to put the "docker pull"-log which doesn't really belong in "terminal.log". Are you sure about this? What happens when the user does not use a real image or some other error that is caused by task configuration... Maybe it could be less verbose but it is not all that verbose now relative to the other stuff that gets spammed via mozharness.
> This is exactly what I am proposing. Sweet, so it'll be another entry in logs.json, and it'll be a log per task. > Are you sure about this? What happens when the user does not use a real image or some other > error that is caused by task configuration... Why not separate the two: - "terminal.log" for data reported from the actual task that ran in a terminal in a docker container. - "docker-worker.log" for data report from docker-worker. "docker-worker.log" should also include task configuration errors and warnings. If a task is so misconfigured that it never starts, then "terminal.log" won't exists and it won't be declared in logs.json, so all people will see is "docker-worker.log" with the errors. Both logs, if in logs.json, will be visible in the task-inspector, and both logs can be reported to treeherder (I think they support multiple logs).
Summary: Operational logs per-task → [docker-worker] Operational logs per-task
Component: TaskCluster → Docker-Worker
Product: Testing → Taskcluster
Whiteboard: [docker-worker]
Component: Docker-Worker → Worker
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
Component: Worker → Workers
You need to log in before you can comment on or make changes to this bug.