1164175 - docker-worker: Don't include 'docker pull' progress output in logs

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

> 706766fe1019 - Downloading [>                                                  ] 527.2 kB/65.76 MB 21s
> 706766fe1019 - Downloading [>                                                  ] 1.054 MB/65.76 MB 14s
> 706766fe1019 - Downloading [=>                                                 ] 1.593 MB/65.76 MB 11s

etc. etc. etc.

it'd be nice to silence this, without silencing the actual *useful* output from docker pull.

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

This should be done: https://github.com/taskcluster/dockerode-process
We can do whatever we want, it's mostly just a matter of decide what is useful and what isn't useful.

If when viewing logs we respect backspace characters we could print this in a way where we delete previous status pages... Or something, I'm not sure what works best.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Do you mean that you've already done something to address this?  Because I don't see a commit to that repo since 2014..

I had been hoping docker pull had a --no-progress option, but that does not appear to be the case.  The --log-level seems not to affect it, either.

A generic \r -> delete-to-last-newline translation would do the trick, and likely be useful for other purposes, so I'll see if that would work.

Comment 3

[Jonas Finnemann Jensen (:jonasfj)]

No, we've done anything... But it is our code the makes the ugly things :)
We used to have a bug for cleaning it up, but it might have drowned or nobody cared to solve it.

The code that prints this ugly stuff is right here:
https://github.com/taskcluster/dockerode-process/blob/300bbc93ed56e1046b63a7fb9911ec91b7b89967/utils.js#L14-L43

We maintain this repository, if anyone feels like refactoring to print something nicer it straightforward.
Silencing it is just making _transform: function() {return done();}
We would should probably print something useful, like errors, layers, speed and time used downloading.
(But maybe we don't need the progress updates, or maybe we should delete them with backspace).

Note, docker-worker doesn't wrap the docker cli tool, we interface the docker unix domain socket and communicate with the docker daemon using the remote HTTP interface. So anything we print to log from pulling is purely our fault.

Comment 4

[Edgar Chen]

https://github.com/taskcluster/dockerode-process/pull/7

Comment 5

[Edgar Chen]

Attachment: PR to fix the bug

Comment 6

[Greg Arndt [:garndt]]

Comment on attachment 8634364 [details] [review]
PR to fix the bug

Looks good. Individual layer progress is now a thing of the past and I see we just log the time each layer took. Still some noise in the logs but should be much less now.