Upload artifacts in parallel

NEW
Unassigned

Status

Taskcluster
Worker
a year ago
a year ago

People

(Reporter: dustin, Unassigned)

Tracking

Details

(Reporter)

Description

a year ago
We just noticed that, in docker-worker, we upload each artifact *declaration* in parallel, but most tasks just have one (for public/artifacts).  Within that declaration, we get a tarball from docker, and upload each artifact as it comes out of the tarball.  We could probably parallelize that and save a few minutes, especially for multiple large artifacts.
Bug 1270656 is the buildbot/mozharness equivalent of this bug.
Some concurrency patterns that may aid tackling this bug:

  * https://talks.golang.org/2012/concurrency.slide#1
  * https://blog.golang.org/pipelines
You need to log in before you can comment on or make changes to this bug.