tests tarballs should be .tar.gz or .tar.zstd

RESOLVED FIXED

Status

RESOLVED FIXED
7 years ago
3 months ago

People

(Reporter: catlee, Assigned: catlee)

Tracking

Trunk

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments)

(Assignee)

Description

7 years ago
zip files are pretty inefficient for archiving test data. switching to .tar.xz would be a huge win for disk usage, disk io, and network bandwidth.

58099173 firefox-13.0a1.en-US.linux-i686.tests.zip
35245341 firefox-13.0a1.en-US.linux-i686.tests.tar.bz2
25022044 firefox-13.0a1.en-US.linux-i686.tests.tar.xz

the build system should support creating tests.tar.xz files if some flag is passed to make package-tests
(Assignee)

Updated

7 years ago
Blocks: 733532

Comment 1

7 years ago
What is xz compression? What tools tools are used to compress/decompress these on our major platforms? Or is this request *nix-only? If we're using a nonstandard compression anyway, how does this compare with 7zip?
(Assignee)

Comment 2

7 years ago
xz is a compression format based on LZMA2. (see also http://tukaani.org/xz/ and http://en.wikipedia.org/wiki/Xz). I like it because it supports the same interfaces as gzip/gunzip, bzip2/bunzip2 and is simply a compression format vs. a compression and archival format.

Recent versions of 7zip and tar support xz.

The same tests archive compressed with 7z comes in at 23647819 bytes (1,374,225 bytes smaller).
(Assignee)

Comment 3

7 years ago
I meant to add that I have no strong preference for xz vs 7zip for this. Either way is a huge win.

Updated

8 months ago
Product: Core → Firefox Build System
(Assignee)

Comment 4

4 months ago
zstd is probably a better choice long-term, but may be hard to get working on all platforms.

gz is better than zip files, and works everywhere right now.
Summary: tests tarballs should be .tar.xz → tests tarballs should be .tar.gz or .tar.zstd
(Assignee)

Comment 5

4 months ago
By using standard gzip, we can shrink the test archive sizes from ~217MB on win64 to ~153MB (30% less) without noticeable impact to test packaging time, or test extraction time.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c3b24e1de2c7b81bf07b98bc25879dc6849ee500

Comment 6

4 months ago
We should ultimately standardize on zstandard within CI: it will be smaller and faster than gzip and there are no good reasons why we wouldn't want CI running as optimally as possible. There are cases where we may want to produce non-zstandard artifacts so end-users can more easily consume them. But for those special cases, we can produce multiple versions of an artifact [until zstandard becomes ubiquitous].

That being said, we don't yet support zstandard everywhere within CI. We have support on the Debian-based Docker images. But I'm pretty sure support is missing on the testing Docker images, random Docker images, macOS, and Windows. There's a bit of follow-up work here. I'm scratching the itch when it blocks me. But it isn't something I or anyone is actively working on AFAIK.

So I think switching from zip to tar.gz is the appropriate action at this time. It has clear wins and should "just work."

Also, I remember measuring the time/space of various compression levels for these archives. We settled on level 5 because level 6 (the default) was spinning substantially more CPU cycles without much to show for it. Fortunately, test_archive.py hardcodes the level to 5 for both zip and tar.gz. So no further measurement should be needed.

I'm itching to review this change!
Created attachment 8986601 [details]
Bug 733530: Use .tar.gz for test archives r=gps
Comment on attachment 8986601 [details]
Bug 733530: Use .tar.gz for test archives r=gps

Gregory Szorc [:gps] has approved the revision.

https://phabricator.services.mozilla.com/D1743
Attachment #8986601 - Flags: review+
Created attachment 8987955 [details]
Bug 733530: Support tar.gz test archives for artifact builds r=gps
Comment on attachment 8987955 [details]
Bug 733530: Support tar.gz test archives for artifact builds r=gps

Gregory Szorc [:gps] has approved the revision.

https://phabricator.services.mozilla.com/D1841
Attachment #8987955 - Flags: review+

Comment 11

4 months ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b1342f2deeec
Support tar.gz test archives for artifact builds r=gps

Updated

4 months ago
Assignee: nobody → catlee
Status: NEW → ASSIGNED
Keywords: leave-open

Updated

4 months ago
Duplicate of this bug: 733528

Comment 14

4 months ago
ciduty were triaging and looks like bug 733528 is related to this effort. Marking that as a dupe of this bug. Please re-open if 733528 is still relevant or has scope outside of this bug.

Comment 15

4 months ago
Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9c75cab2e322
Use .tar.gz for test archives r=gps
Created attachment 8989708 [details]
Bug 733530 Use tar instead of unzip in periodic updates r=catlee

Replace 'unzip -q' with 'tar -xvf' for tests .tar.gz

Comment 18

4 months ago
backoutbugherder
Backed out changeset 9c75cab2e322 (bug 733530) for breaking artifact builds
https://hg.mozilla.org/mozilla-central/rev/a9dc5dc8e2b8
Comment on attachment 8989708 [details]
Bug 733530 Use tar instead of unzip in periodic updates r=catlee

Chris AtLee [:catlee] has approved the revision.

https://phabricator.services.mozilla.com/D1939
Attachment #8989708 - Flags: review+
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/11788 for changes under testing/web-platform/tests
(Assignee)

Comment 21

4 months ago
(In reply to Bogdan Tara[:bogdan_tara] from comment #18)
> Backed out changeset 9c75cab2e322 (bug 733530) for breaking artifact builds
> https://hg.mozilla.org/mozilla-central/rev/a9dc5dc8e2b8

I think this is failing here:
https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#214

It's trying to treat the tests archive as a jar/zip file. I don't really understand what this code is doing...why are we re-compressing things into .processed.jar files?
Upstream PR merged
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/11826 for changes under testing/web-platform/tests
Upstream PR merged
Upstream PR merged
Upstream PR merged

Comment 27

3 months ago
Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6f2020891542
Use .tar.gz for test archives r=gps
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/12259 for changes under testing/web-platform/tests
Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.
Upstream PR merged

Comment 32

3 months ago
Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/60d153b3f3c8
Use tar instead of unzip in periodic updates r=catlee
(Assignee)

Updated

3 months ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 months ago
Resolution: --- → FIXED
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.