1317875 - Use multiple HTTP connections to upload files

Status: NEW

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

Currently in mozharness's buildbase.py, upload_files() and _taskcluster_upload() upload files sequentially, using a single I/O thread.

From my work optimizing the decision graph, I know that throwing multiple I/O threads/connections at the problem will make things significantly faster.

We should plug in concurrent.futures and a requests.Session and aggressively use multiple I/O threads to perform uploads. See the code in taskcluster/taskgraph/create.py for inspiration.

Perfherder data says the upload step takes ~3 minutes on average on Linux. I think we can reduce it to less than 60s. Since upload is in the critical path for running tests, this should speed up end to end times by whatever we reduce the wall time execution by.