733528 - Support creating and extracting .tar.xz files

Status: RESOLVED DUPLICATE of bug 733530

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Chris AtLee [:catlee]]

test archives are quite large, and are currently compressed using zip. as of this writing:

58099173 firefox-13.0a1.en-US.linux-i686.tests.zip
35245341 firefox-13.0a1.en-US.linux-i686.tests.tar.bz2
25022044 firefox-13.0a1.en-US.linux-i686.tests.tar.xz

Using .xz files would save more than 50% on disk space, disk io and network bandwidth, three resources we're always regularly hitting upper limits on.

We need all of our build and test machines to support creating and extracting .tar.xz files.

Comment 2

[Carsten Book [:Tomcat]]

seems linux should be very easy via the integrated updater. 

For mac there is http://sourceforge.net/projects/macpkg/.

Windows might be a little more work, i found:

 If you have xz-5.x.x.tar.gz in C:\devel, you should be able to build
    it now with the following commands:

        cd /c/devel
        tar xzf xz-5.x.x.tar.gz
        cd xz-5.x.x
        bash windows/build.bash

    If you used some other directory than C:\devel\tools for the build
    tools, edit the variables near the beginning of build.bash first.

    If you want to build manually, read the buildit() function in
    build.bash. Look especially at the latter configure invocation.

Comment 3

[Andreas Tolfsen ❲:ato❳]

When I did some performance measurements on lzma archives for my previous employer a while back, they consumed less disk space but took longer to extract.  I haven't checked this again recently, but balancing download time of test packages vs. extraction time could be as important since some systems may have other limitations.

Comment 4

[Chris Cooper [:coop] (he/him)]

All of our linux platforms (including ec2 and spot) have xz installed by default now.

$ xz --version
xz (XZ Utils) 4.999.9beta
liblzma 4.999.9beta

Need someone for build or find packages for Mac and Windows.

Comment 5

[maillist-mozilla]

ping. this is getting more and more pressing as each firefox release grows by several megabytes.

the only difference between creating .bz2 and .xz is to use tar cJf instead of tar cjf (upper case J).
to make it even more effective create uncompressed .tar via tar cf and then compress it by invoking xz utility with extreme setting, i.e. xz -e -9.

Comment 6

[Chris AtLee [:catlee]]

(In reply to maillist-mozilla from comment #5)
> ping. this is getting more and more pressing as each firefox release grows
> by several megabytes.
> 
> the only difference between creating .bz2 and .xz is to use tar cJf instead
> of tar cjf (upper case J).
> to make it even more effective create uncompressed .tar via tar cf and then
> compress it by invoking xz utility with extreme setting, i.e. xz -e -9.

Unfortunately it's not quite that easy. We support many different platforms in our build/test infrastructure, including older platforms like Windows XP and OSX 10.7. We need to make sure xz archives are supported there as well.

Comment 7

[Chris AtLee [:catlee]]

That being said, it may be possible do this on Linux only to start with.

Comment 8

[maillist-mozilla]

oh, i thought this discussion is mainly about the source code tarballs.
aren't the windows/mac binaries shipped in the native format (sfx for windows, dmg for mac) anyways ?

Comment 9

[Chris AtLee [:catlee]]

release-specific tasks like source tarballs are being tracked in bug 807637. this bug is about general support for .xz in our infra.

Comment 10

[Andreas Tolfsen ❲:ato❳]

Please note that the increase in extraction time might make CI systems, and consequently the feedback loop to the developer, take longer due to greater extraction times with lzma.

Comment 11

[maillist-mozilla]

(In reply to Andreas Tolfsen (:ato) from comment #10)
> Please note that the increase in extraction time might make CI systems, and
> consequently the feedback loop to the developer, take longer due to greater
> extraction times with lzma.

i think you misunderstood something. xz/lzma decompression is 4x *faster* than bzip2 decompression.

Comment 12

[sjw]

Just a tiny update for macOS: xz compressed tarballs are supported in current versions, even in the GUI. xz compressed files can be uncompressed using the CLI.

Meanwhile some other projects like the kernel and Debian (dpkg) switched to xz.