Closed Bug 733528 Opened 11 years ago Closed 5 years ago

Support creating and extracting .tar.xz files

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 733530

People

(Reporter: catlee, Unassigned)

References

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2092] )

test archives are quite large, and are currently compressed using zip. as of this writing:

58099173 firefox-13.0a1.en-US.linux-i686.tests.zip
35245341 firefox-13.0a1.en-US.linux-i686.tests.tar.bz2
25022044 firefox-13.0a1.en-US.linux-i686.tests.tar.xz

Using .xz files would save more than 50% on disk space, disk io and network bandwidth, three resources we're always regularly hitting upper limits on.

We need all of our build and test machines to support creating and extracting .tar.xz files.
Blocks: 733532
Assignee: nobody → coop
Priority: -- → P3
Assignee: coop → cbook
seems linux should be very easy via the integrated updater. 

For mac there is http://sourceforge.net/projects/macpkg/.

Windows might be a little more work, i found:

 If you have xz-5.x.x.tar.gz in C:\devel, you should be able to build
    it now with the following commands:

        cd /c/devel
        tar xzf xz-5.x.x.tar.gz
        cd xz-5.x.x
        bash windows/build.bash

    If you used some other directory than C:\devel\tools for the build
    tools, edit the variables near the beginning of build.bash first.

    If you want to build manually, read the buildit() function in
    build.bash. Look especially at the latter configure invocation.
Assignee: cbook → nobody
Product: mozilla.org → Release Engineering
When I did some performance measurements on lzma archives for my previous employer a while back, they consumed less disk space but took longer to extract.  I haven't checked this again recently, but balancing download time of test packages vs. extraction time could be as important since some systems may have other limitations.
All of our linux platforms (including ec2 and spot) have xz installed by default now.

$ xz --version
xz (XZ Utils) 4.999.9beta
liblzma 4.999.9beta

Need someone for build or find packages for Mac and Windows.
Blocks: 807637
ping. this is getting more and more pressing as each firefox release grows by several megabytes.

the only difference between creating .bz2 and .xz is to use tar cJf instead of tar cjf (upper case J).
to make it even more effective create uncompressed .tar via tar cf and then compress it by invoking xz utility with extreme setting, i.e. xz -e -9.
(In reply to maillist-mozilla from comment #5)
> ping. this is getting more and more pressing as each firefox release grows
> by several megabytes.
> 
> the only difference between creating .bz2 and .xz is to use tar cJf instead
> of tar cjf (upper case J).
> to make it even more effective create uncompressed .tar via tar cf and then
> compress it by invoking xz utility with extreme setting, i.e. xz -e -9.

Unfortunately it's not quite that easy. We support many different platforms in our build/test infrastructure, including older platforms like Windows XP and OSX 10.7. We need to make sure xz archives are supported there as well.
That being said, it may be possible do this on Linux only to start with.
oh, i thought this discussion is mainly about the source code tarballs.
aren't the windows/mac binaries shipped in the native format (sfx for windows, dmg for mac) anyways ?
release-specific tasks like source tarballs are being tracked in bug 807637. this bug is about general support for .xz in our infra.
Please note that the increase in extraction time might make CI systems, and consequently the feedback loop to the developer, take longer due to greater extraction times with lzma.
(In reply to Andreas Tolfsen (:ato) from comment #10)
> Please note that the increase in extraction time might make CI systems, and
> consequently the feedback loop to the developer, take longer due to greater
> extraction times with lzma.

i think you misunderstood something. xz/lzma decompression is 4x *faster* than bzip2 decompression.
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2084]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2084] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2092]
Just a tiny update for macOS: xz compressed tarballs are supported in current versions, even in the GUI. xz compressed files can be uncompressed using the CLI.

Meanwhile some other projects like the kernel and Debian (dpkg) switched to xz.
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → DUPLICATE
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.