1270656 - Upload files in parallel

Status: RESOLVED WONTFIX

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Gregory Szorc [:gps]]

The upload-files mozharness step appears to process file uploads [to taskcluster] serially. We could make file upload faster if we employed a thread pool to upload multiple files at a time.

Comment 1

[Gregory Szorc [:gps]]

So, I'd *really* like to use concurrent.futures and its ThreadPoolExecutor for this. However, that would require making concurrent.futures available to mozharness.

We already have python/futures in mozilla-central. However, there is a bootstrap problem. We create a mozharness archive from the contents of testing/mozharness and extract that to the machine to run mozharness.

jlund: do we have the ability to throw additional files from mozilla-central into the mozharness zip archive used by archiver? I could probably figure out how to do if if I knew where the code lived. I'm guessing the mozharness zip generation is somewhere in buildbot land?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Gregory, you can find the archiver here:
https://hg.mozilla.org/build/tools/file/default/buildfarm/utils/archiver_client.py

Comment 3

[Jordan Lund (:jlund)]

> jlund: do we have the ability to throw additional files from mozilla-central
> into the mozharness zip archive used by archiver? I could probably figure
> out how to do if if I knew where the code lived. I'm guessing the mozharness
> zip generation is somewhere in buildbot land?

as is, we can't really add things into the mozharness archive. archiver simply downloads a subdir archive from a GET to hgmo, caches it in s3, and serves that to anything that requests it. basically just a proxy.

(In reply to Henrik Skupin (:whimboo) from comment #2)
> Gregory, you can find the archiver here:
> https://hg.mozilla.org/build/tools/file/default/buildfarm/utils/
> archiver_client.py

yep, this is the client. we would have to teach the client to know about more than just the mozharness path so that it also tells archiver to archive `python/future/` and download that on slaves: https://hg.mozilla.org/build/tools/file/default/buildfarm/utils/archiver_client.py#l34

the backend, a relengapi endpoint, should just work®  https://api.pub.build.mozilla.org/docs/usage/archiver/


though, archiver is meant to just be a stop gap until we can scale hgmo subdir archive requests..

Comment 4

[Gregory Szorc [:gps]]

I'm not actively working on this.