678371 - Investigate better extraction methods for tests on Windows

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[William Lachance (:wlach)]

We current use unzip (with no arguments) to unpack the tests after downloading them from a central server. After doing a bit of testing on my workstation, I'm pretty sure this is suboptimal (it currently takes over 2 minutes to exact on my Win7 VM running on a fastish machine). Other options include:

1. Using 'unzip -qq' (reduce console spew)
2. Using 7zip
3. Using tar.bz2 archives with 7zip
4. Using tar.bz2 archives with stock tar

All of these seem a little faster to me (by a small margin with -qq, by potentially as much as a minute with .tar.bz2 + 7zip). I'm going to write up a script to measure the difference between all these methods.

At the same time, it may be worth investigating extracting into a ramdisk, as disk i/o seems to account for almost all of the extraction time.

Comment 1

[William Lachance (:wlach)]

Ok, I wrote up some scripts and did some tests on Windows. Unfortunately the results weren't quite as spectacular as I'd hoped. My guess is that yesterday my results were being influenced by writing the zip extraction to my console, which wasn't happening in my test (and I hope we're not doing in production)

Procedure for running these:
* Install the 64-bit version of 7-Zip on a windows machine (default path)
* Run the 'test-unzip.sh' (or the 'test-unzip-reftest.sh' variant) on a mozilla-build command line as follows:

   for MODE in ZIP_NO_QQ ZIP_QQ TAR 7ZIP_ZIP 7ZIP_TAR; do for I in 1 2 3 4 5; do
test-unzip-reftest.sh $MODE; done; done > test-unzip-results.log

* Extract out a CSV file from the results:

   grep ZIPRESULT results.log | sed -e 's/ZIPRESULT: //' > test-unzip-results.csv

* Run the attached python script to get averages:

   python parse-zip-results.py test-unzip-results.csv

Following this procedure, I got the following results on the ateam test talos slave.

Unpack everything:

7ZIP_TAR,41.1538000584
7ZIP_ZIP,42.4886000156
TAR,52.2455999851
ZIP_QQ,55.6842000485
ZIP_NO_QQ,62.3021999836

Unpack just reftest (this is probably a more realistic simulation of what actually happens):

7ZIP_ZIP,14.8788000584
7ZIP_TAR,19.8104000092
TAR,23.0297998905
ZIP_NO_QQ,23.6271999836
ZIP_QQ,26.3085999966

Observations:

* 7-Zip is fastest
* ZIP is noticeably faster than TAR (at least when using 7-Zip) when extracting just part of the archive
* Just passing -qq to unzip results in a small speed boost when processing a lot of files (mysteriously it seems to on average be slower in the reftest example above: I'd put that down to something random happening while the test was running)

I don't know enough about the specifics of our build environment to say how much time we might save by using 7-Zip on Windows. One other variable to consider is that tar.bz2 archives are considerably smaller than zip archives (29M vs. 59M) which may result in a small reduction in file transfer times.

Comment 2

[William Lachance (:wlach)]

Attachment: Script that allows testing unzipping/untarring of firefox test archives in a number of different ways

Comment 3

[William Lachance (:wlach)]

Attachment: Script that allows testing unzipping/untarring of just the reftest part of the firefox test archives in a number of different ways

Comment 4

[William Lachance (:wlach)]

Attachment: Python script to handle parsing the unzip log results

Comment 5

[William Lachance (:wlach)]

Just did some quick testing with imdisk (a RAMdisk driver for windows: http://www.ltr-data.se/opencode.html/#ImDisk) and found that this seems to reduce extraction times to ~5 seconds for just reftest on my VM using 7-Zip, which is blazingly faster (needless to say). I'll experiment with it on our talos machine and see if I don't get similar results there.

Comment 6

[William Lachance (:wlach)]

Ok, that was interesting. It turns out that imdisk has several options. One is to create a new partition disk backed by a file, another backed by memory. I decided to experiment with both (with the idea that memory might be faster disk i/o wise, but might restrict the amount of memory available for actually running tests).

Here's the results for a complete unpacking with a file-backed virtual disk:

7ZIP_ZIP,24.5458000183
7ZIP_TAR,24.9534000397
ZIP_NO_QQ,36.7146000385
ZIP_QQ,34.5679999352
TAR,29.405600071

Here's some selected results for a complete unpacking with a memory-backed disk:

7ZIP_TAR,16.2843999863
7ZIP_ZIP,15.8316000462

(not sure if the ~10 second savings is worth the brittleness or loss of memory that this entails)

In addition to the savings on unpack, deleting all the files recursively on windows takes about 20 seconds. With this approach, we can simply format the virtual disk, which takes around a second (I haven't measured but that seems to be about how long it takes). So that's a total savings of about a minute on Windows with a file-backed virtual disk, assuming we do an average of one full unpack/delete over the course of a build run.

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to William Lachance from comment #6)
> Ok, that was interesting. It turns out that imdisk has several options. One
> is to create a new partition disk backed by a file, another backed by
> memory. I decided to experiment with both (with the idea that memory might
> be faster disk i/o wise, but might restrict the amount of memory available
> for actually running tests).
> 
> Here's the results for a complete unpacking with a file-backed virtual disk:
> 
> 7ZIP_ZIP,24.5458000183
> 7ZIP_TAR,24.9534000397
> ZIP_NO_QQ,36.7146000385
> ZIP_QQ,34.5679999352
> TAR,29.405600071
> 
> Here's some selected results for a complete unpacking with a memory-backed
> disk:
> 
> 7ZIP_TAR,16.2843999863
> 7ZIP_ZIP,15.8316000462
> 
> (not sure if the ~10 second savings is worth the brittleness or loss of
> memory that this entails)
> 
> In addition to the savings on unpack, deleting all the files recursively on
> windows takes about 20 seconds. With this approach, we can simply format the
> virtual disk, which takes around a second (I haven't measured but that seems
> to be about how long it takes). So that's a total savings of about a minute
> on Windows with a file-backed virtual disk, assuming we do an average of one
> full unpack/delete over the course of a build run.

Using a virutal disk does sound slightly brittle, it also means that we are no longer running a configuration that is reasonable to expect in a consumer system.

One optimization that we make that I don't see accounted for in your tests is selectively unpacking the zip files.  This is to specify wildcards of the files we want.  An example would be "unzip ../firefox-8.0a1.en-US.win32.tests.zip bin* certs* mochitest*"

I found that doing this caused the extraction times to drop from 3:30 down to 0:30 for mochitests.  It also reduced the time to clobber the tests directories from 0:32 to 0:07.

Do you think you'd have a chance to test again using the selective unpacking?

Also, thanks for your investigation and interest!

Comment 8

[William Lachance (:wlach)]

Hi John, 

We're already doing the selective unpack, AFAIK. Some of my earlier tests take that into account (see comment #1). It's important to keep in mind though that if the full suite of tests are run, then you still have to deal with an overall full unpack/clobber time spread over many machines, so tests that unpack/clobber the whole thing are still relevant. I'd expect the results with virtual disks and selective unpacking to be proportional to those done without.

When I said "brittle" I was referring mostly to the in-memory virtual disk. You are right that even the file-backed virtual disk is a bit different from what you'd expect to see on a consumer system, I guess the question is whether there are particular features/conditions of testing on a real disk that are important to keep for the purposes of non-performance-based tests. I don't know of any, but if there are, yet another option might be to create a seperate "testing" partition on our Windows machines which can be wiped out/formatted at will.

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to William Lachance from comment #8)
> Hi John, 
> 
> We're already doing the selective unpack, AFAIK. 

Yes, we are.  Apologies, I didn't see that you ran the script with selectively unpacking.

> Some of my earlier tests
> take that into account (see comment #1). It's important to keep in mind
> though that if the full suite of tests are run, then you still have to deal
> with an overall full unpack/clobber time spread over many machines, so tests
> that unpack/clobber the whole thing are still relevant. I'd expect the
> results with virtual disks and selective unpacking to be proportional to
> those done without.
> 
> When I said "brittle" I was referring mostly to the in-memory virtual disk.
> You are right that even the file-backed virtual disk is a bit different from
> what you'd expect to see on a consumer system, I guess the question is
> whether there are particular features/conditions of testing on a real disk
> that are important to keep for the purposes of non-performance-based tests.
> I don't know of any, but if there are, yet another option might be to create
> a seperate "testing" partition on our Windows machines which can be wiped
> out/formatted at will.

It is possible that there are effects from using these tools beyond performance characteristics.

We have done reformatting on test machines in the past and found that the advantages were outweighed by the complexity of automating the unmount+format+mount.

We also need to consider the added complexity of setting up and managing these virtual disks.  Staying with zip files also has the advantage of not requiring any tooling changes for automation or build system.

Comment 10

[William Lachance (:wlach)]

(In reply to John Ford [:jhford] from comment #9)

> It is possible that there are effects from using these tools beyond
> performance characteristics.
> 
> We have done reformatting on test machines in the past and found that the
> advantages were outweighed by the complexity of automating the
> unmount+format+mount.
> 
> We also need to consider the added complexity of setting up and managing
> these virtual disks.  Staying with zip files also has the advantage of not
> requiring any tooling changes for automation or build system.

Yup, I guess we'd have to weigh any performance benefits of using virtual disks or seperate partitions over the time needed to make the change as well as the risk of unwanted side affects. Maybe we can talk about it during our GoFaster meeting this week.

Switching to 7-Zip looks like a pretty obvious win though, no?

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to William Lachance from comment #10)
> Yup, I guess we'd have to weigh any performance benefits of using virtual
> disks or seperate partitions over the time needed to make the change as well
> as the risk of unwanted side affects. Maybe we can talk about it during our
> GoFaster meeting this week.
> 
> Switching to 7-Zip looks like a pretty obvious win though, no?

It is faster, we just need to balance the ongoing performance wins with the time/effort to deploy a new application on windows.

To move this forward, I think what we need to do is put 7zip on a windows XP, windows 7 and windows 7 x64 testing slave.  We'd then need to patch the buildbotcustom.process.factory.TalosFactory and buildbotcustom.process.factory.MozillaTestFactory (names by memory, might be slightly different) to call "7za e -y" instead of "unzip..." and run that in staging to compare times.  If this still proved to be useful, we would need to write an OPSI package to deploy 7za.  Likely, we'd only want to install the 7za binary on the slaves, as the GUI doesn't help us.

Another option for deployment, which could help devs is to include 7za in MozillaBuild.

William, do you have interest/time to work on this?

Comment 12

[William Lachance (:wlach)]

So I did a bit more in(In reply to John Ford [:jhford] from comment #11)
> (In reply to William Lachance from comment #10)
> > Yup, I guess we'd have to weigh any performance benefits of using virtual
> > disks or seperate partitions over the time needed to make the change as well
> > as the risk of unwanted side affects. Maybe we can talk about it during our
> > GoFaster meeting this week.
> > 
> > Switching to 7-Zip looks like a pretty obvious win though, no?
> 
> It is faster, we just need to balance the ongoing performance wins with the
> time/effort to deploy a new application on windows.
> 
> To move this forward, I think what we need to do is put 7zip on a windows
> XP, windows 7 and windows 7 x64 testing slave.  We'd then need to patch the
> buildbotcustom.process.factory.TalosFactory and
> buildbotcustom.process.factory.MozillaTestFactory (names by memory, might be
> slightly different) to call "7za e -y" instead of "unzip..." and run that in
> staging to compare times.  If this still proved to be useful, we would need
> to write an OPSI package to deploy 7za.  Likely, we'd only want to install
> the 7za binary on the slaves, as the GUI doesn't help us.
> 
> Another option for deployment, which could help devs is to include 7za in
> MozillaBuild.
> 
> William, do you have interest/time to work on this?

I did a bit more investigation and it looks like we already include the 32-bit of 7-Zip in mozilla-build, however it's actually a bit *slower* than unzip. Apparently it's only the 64-bit version that's faster for some reason, so I guess the benefits would be isolated there.

Not really sure where that leaves us. If it is worth the time and hassle to install 7-Zip on our 64-bit slaves (or even switch to virtual disks, though it seems like that may not be a good idea), I could definitely help do that.

A further option would be to pass -qq to unzip, which should result in a small bump in performance, is trivial to implement, and has almost no risk.

Comment 13

[William Lachance (:wlach)]

After talking with catlee at the meeting, the consensus was that the ideas in this bug don't compute from a cost/benefit perspective. 

* 64-bit 7-Zip only saves 10 seconds or so in the best case, complicated to implement.
* unzip -qq has only a marginal benefit, and having verbose output of what tests are being extracted in some cases.