Closed Bug 1270656 Opened 6 years ago Closed 5 years ago
Upload files in parallel
The upload-files mozharness step appears to process file uploads [to taskcluster] serially. We could make file upload faster if we employed a thread pool to upload multiple files at a time.
So, I'd *really* like to use concurrent.futures and its ThreadPoolExecutor for this. However, that would require making concurrent.futures available to mozharness. We already have python/futures in mozilla-central. However, there is a bootstrap problem. We create a mozharness archive from the contents of testing/mozharness and extract that to the machine to run mozharness. jlund: do we have the ability to throw additional files from mozilla-central into the mozharness zip archive used by archiver? I could probably figure out how to do if if I knew where the code lived. I'm guessing the mozharness zip generation is somewhere in buildbot land?
Gregory, you can find the archiver here: https://hg.mozilla.org/build/tools/file/default/buildfarm/utils/archiver_client.py
> jlund: do we have the ability to throw additional files from mozilla-central > into the mozharness zip archive used by archiver? I could probably figure > out how to do if if I knew where the code lived. I'm guessing the mozharness > zip generation is somewhere in buildbot land? as is, we can't really add things into the mozharness archive. archiver simply downloads a subdir archive from a GET to hgmo, caches it in s3, and serves that to anything that requests it. basically just a proxy. (In reply to Henrik Skupin (:whimboo) from comment #2) > Gregory, you can find the archiver here: > https://hg.mozilla.org/build/tools/file/default/buildfarm/utils/ > archiver_client.py yep, this is the client. we would have to teach the client to know about more than just the mozharness path so that it also tells archiver to archive `python/future/` and download that on slaves: https://hg.mozilla.org/build/tools/file/default/buildfarm/utils/archiver_client.py#l34 the backend, a relengapi endpoint, should just work® https://api.pub.build.mozilla.org/docs/usage/archiver/ though, archiver is meant to just be a stop gap until we can scale hgmo subdir archive requests..
I'm not actively working on this.
Assignee: gps → nobody
Status: ASSIGNED → NEW
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.