1608446 - Download and extract test packages in parallel

Status: NEW

(Testing :: General, enhancement, P3)

Description

[Marco Castelluccio [:marco]]

We are currently downloading and extracting them one after the other.
While parallel extraction by itself probably wouldn't buy us much, in this case we can parallelize downloads and extractions (so while we extract a package, we can start downloading the next).

Comment 1

[Chris AtLee [:catlee]]

I had an old patch that I dusted off to give this a shot. The results are here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c67d3f85b376cb68b87fb2ac06461e0a9d1604d1

It looks like most of the time is spent in extraction. zstd will help a bit there.

Comment 2

[Chris AtLee [:catlee]]

If we shell out to tar, we can drastically improve our extraction times. See here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=bc1f33c8c8329d708e99470f27f0bb9bf958cdfa

Comment 3

[Chris AtLee [:catlee]]

(In reply to Chris AtLee [:catlee] from comment #2)

If we shell out to tar, we can drastically improve our extraction times. See here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=bc1f33c8c8329d708e99470f27f0bb9bf958cdfa

e.g. unpacking target.common.tests.tar.gz takes 5s with tar instead of 50s with python's tarfile module.

Comment 4

[Tom Prince [:tomprince]]

Attachment: Bug 1608446: Use `workspace/` for tests.

Comment 5

[Tom Prince [:tomprince]]

Attachment: Bug 1608446: Use fetches in tests.

Comment 6

[Tom Prince [:tomprince]]

Attachment: Bug 1608446: explicit common test tar r?jmaher

When I was looking at the patches for this bug, it was a bit difficult to see
what was included in the common test package (when looking at the extract lists
in mozharness). At this point, it'd probably be reasonable to make the things
included explicit, rather than implicit via excludes.

Though Bug 1330421 makes that difficult in other ways; I was wondering it might
make sense to move the generated/compiled bits to their own package, in
preperation for Bug 1628981. (Though a bunch of that stuff could in toolchain
packages probably, instead)

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months.
:ahal, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 8

[Andrew Halberstadt [:ahal]]

This looks like it's still an issue, though unsure why the attached patches never landed. It's unclear if moving to fetches (like bug 1657606 and revision in here accomplishes) is enough to fix the problem completely. If so we should likely dupe this to bug 1657606.