1245553 - Investigate UNZIP on windows alternitives

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Comment 1

[Q]

We are currently using a binary (5.5.2) from 2005 in C:\mozilla-build\info-zip. This version has some potential race condition issues and has a 2gb files size limit. 

Looking at using version  6.0  of the tool from 2009 as a stop gap. However, we should consider changing the test harness to use a modern utility such as 7zip etc.

Comment 2

[Q]

Based on info in http://gregoryszorc.com/blog/2015/10/22/append-i/o-performance-on-windows/

It may be beneficial to write a simple unzip replacement in python for windows. 

GPS how hard do you think something like this would be ?

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So I started to work on a patch for bug 1237706 which actually let mozharness use the Python ZipFile/TarFile classes in case no unzip executable is available. Maybe this change would be beneficial for your problem?

Comment 4

[Gregory Szorc [:gps]]

Attachment: PoC Python code

Here is some PoC Python code. It needs some more intelligence, such as not starting a thread pool if not running on Windows and when the number of files that need written is small (there is an overhead to starting and stopping the thread pool).

Use it like so:

with BackgroundFileCloser() as bfc:
    for item in archive:
        fh = open(item.path, 'wb')
        fh.write(item.data)
        bfc.close(fh)

This will be faster than 7-zip on Windows. The CPU critical section of archive extraction is decompression. And that is implemented in C in Python. Python will introduce minimal overhead being a higher-level language. The thread pool asynchronously closing file handles will more than cancel out the Python overhead.

Comment 5

[Q]

Custom script in python for windows 7 cloud. Even that will be going away with task cluster. Unzipping will be handled differently there