Closed Bug 1270656 Opened 6 years ago Closed 5 years ago

Upload files in parallel

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: gps, Unassigned)

References

(Blocks 1 open bug)

Details

The upload-files mozharness step appears to process file uploads [to taskcluster] serially. We could make file upload faster if we employed a thread pool to upload multiple files at a time.
So, I'd *really* like to use concurrent.futures and its ThreadPoolExecutor for this. However, that would require making concurrent.futures available to mozharness.

We already have python/futures in mozilla-central. However, there is a bootstrap problem. We create a mozharness archive from the contents of testing/mozharness and extract that to the machine to run mozharness.

jlund: do we have the ability to throw additional files from mozilla-central into the mozharness zip archive used by archiver? I could probably figure out how to do if if I knew where the code lived. I'm guessing the mozharness zip generation is somewhere in buildbot land?
Flags: needinfo?(jlund)
> jlund: do we have the ability to throw additional files from mozilla-central
> into the mozharness zip archive used by archiver? I could probably figure
> out how to do if if I knew where the code lived. I'm guessing the mozharness
> zip generation is somewhere in buildbot land?

as is, we can't really add things into the mozharness archive. archiver simply downloads a subdir archive from a GET to hgmo, caches it in s3, and serves that to anything that requests it. basically just a proxy.

(In reply to Henrik Skupin (:whimboo) from comment #2)
> Gregory, you can find the archiver here:
> https://hg.mozilla.org/build/tools/file/default/buildfarm/utils/
> archiver_client.py

yep, this is the client. we would have to teach the client to know about more than just the mozharness path so that it also tells archiver to archive `python/future/` and download that on slaves: https://hg.mozilla.org/build/tools/file/default/buildfarm/utils/archiver_client.py#l34

the backend, a relengapi endpoint, should just work®  https://api.pub.build.mozilla.org/docs/usage/archiver/


though, archiver is meant to just be a stop gap until we can scale hgmo subdir archive requests..
Flags: needinfo?(jlund)
I'm not actively working on this.
Assignee: gps → nobody
Blocks: fastci
Status: ASSIGNED → NEW
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.