Closed Bug 733530 Opened 12 years ago Closed 6 years ago

tests tarballs should be .tar.gz or .tar.zstd

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: catlee, Assigned: catlee)

References

(Blocks 1 open bug)

Details

Attachments

(3 files)

Bug 733530: Use .tar.gz for test archives r=gps
46 bytes, text/x-phabricator-request
gps
: review+
Details | Review
Bug 733530: Support tar.gz test archives for artifact builds r=gps
46 bytes, text/x-phabricator-request
gps
: review+
Details | Review
Bug 733530 Use tar instead of unzip in periodic updates r=catlee
46 bytes, text/x-phabricator-request
catlee
: review+
Details | Review 
zip files are pretty inefficient for archiving test data. switching to .tar.xz would be a huge win for disk usage, disk io, and network bandwidth.

58099173 firefox-13.0a1.en-US.linux-i686.tests.zip
35245341 firefox-13.0a1.en-US.linux-i686.tests.tar.bz2
25022044 firefox-13.0a1.en-US.linux-i686.tests.tar.xz

the build system should support creating tests.tar.xz files if some flag is passed to make package-tests
Blocks: 733532
What is xz compression? What tools tools are used to compress/decompress these on our major platforms? Or is this request *nix-only? If we're using a nonstandard compression anyway, how does this compare with 7zip?
xz is a compression format based on LZMA2. (see also http://tukaani.org/xz/ and http://en.wikipedia.org/wiki/Xz). I like it because it supports the same interfaces as gzip/gunzip, bzip2/bunzip2 and is simply a compression format vs. a compression and archival format.

Recent versions of 7zip and tar support xz.

The same tests archive compressed with 7z comes in at 23647819 bytes (1,374,225 bytes smaller).
I meant to add that I have no strong preference for xz vs 7zip for this. Either way is a huge win.
Product: Core → Firefox Build System
zstd is probably a better choice long-term, but may be hard to get working on all platforms.

gz is better than zip files, and works everywhere right now.
Summary: tests tarballs should be .tar.xz → tests tarballs should be .tar.gz or .tar.zstd
By using standard gzip, we can shrink the test archive sizes from ~217MB on win64 to ~153MB (30% less) without noticeable impact to test packaging time, or test extraction time.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c3b24e1de2c7b81bf07b98bc25879dc6849ee500
We should ultimately standardize on zstandard within CI: it will be smaller and faster than gzip and there are no good reasons why we wouldn't want CI running as optimally as possible. There are cases where we may want to produce non-zstandard artifacts so end-users can more easily consume them. But for those special cases, we can produce multiple versions of an artifact [until zstandard becomes ubiquitous].

That being said, we don't yet support zstandard everywhere within CI. We have support on the Debian-based Docker images. But I'm pretty sure support is missing on the testing Docker images, random Docker images, macOS, and Windows. There's a bit of follow-up work here. I'm scratching the itch when it blocks me. But it isn't something I or anyone is actively working on AFAIK.

So I think switching from zip to tar.gz is the appropriate action at this time. It has clear wins and should "just work."

Also, I remember measuring the time/space of various compression levels for these archives. We settled on level 5 because level 6 (the default) was spinning substantially more CPU cycles without much to show for it. Fortunately, test_archive.py hardcodes the level to 5 for both zip and tar.gz. So no further measurement should be needed.

I'm itching to review this change!
Comment on attachment 8986601 [details]
Bug 733530: Use .tar.gz for test archives r=gps

Gregory Szorc [:gps] has approved the revision.

https://phabricator.services.mozilla.com/D1743
Attachment #8986601 - Flags: review+
Comment on attachment 8987955 [details]
Bug 733530: Support tar.gz test archives for artifact builds r=gps

Gregory Szorc [:gps] has approved the revision.

https://phabricator.services.mozilla.com/D1841
Attachment #8987955 - Flags: review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b1342f2deeec
Support tar.gz test archives for artifact builds r=gps
Assignee: nobody → catlee
Status: NEW → ASSIGNED
Keywords: leave-open
ciduty were triaging and looks like bug 733528 is related to this effort. Marking that as a dupe of this bug. Please re-open if 733528 is still relevant or has scope outside of this bug.
Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9c75cab2e322
Use .tar.gz for test archives r=gps
Replace 'unzip -q' with 'tar -xvf' for tests .tar.gz
Backed out changeset 9c75cab2e322 (bug 733530) for breaking artifact builds
https://hg.mozilla.org/mozilla-central/rev/a9dc5dc8e2b8
Comment on attachment 8989708 [details]
Bug 733530 Use tar instead of unzip in periodic updates r=catlee

Chris AtLee [:catlee] has approved the revision.

https://phabricator.services.mozilla.com/D1939
Attachment #8989708 - Flags: review+
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/11788 for changes under testing/web-platform/tests
(In reply to Bogdan Tara[:bogdan_tara] from comment #18)
> Backed out changeset 9c75cab2e322 (bug 733530) for breaking artifact builds
> https://hg.mozilla.org/mozilla-central/rev/a9dc5dc8e2b8

I think this is failing here:
https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#214

It's trying to treat the tests archive as a jar/zip file. I don't really understand what this code is doing...why are we re-compressing things into .processed.jar files?
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/11826 for changes under testing/web-platform/tests
Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6f2020891542
Use .tar.gz for test archives r=gps
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/12259 for changes under testing/web-platform/tests
Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.
Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/60d153b3f3c8
Use tar instead of unzip in periodic updates r=catlee
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Blocks: 1507094
See Also: → 1579224
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: