1404887 - mozfile.extract_zip reads entire zip entries into memory

Status: RESOLVED FIXED

(Testing :: Mozbase, enhancement)

Description

[(not currently active) Ted Mielczarek]

bkelly linked a WPT test run that failed trying to symbolicate an assertion stack:
https://treeherder.mozilla.org/logviewer.html#?job_id=133188470&repo=try&lineNumber=36760

It looks like Python OOMed trying to extract the symbols zip in mozfile.extract_zip:
https://dxr.mozilla.org/mozilla-central/rev/41286177c59c74ec37961b1edea34cc90d6f6dc5/testing/mozbase/mozfile/mozfile/mozfile.py#73

That function reads each zip entry into memory to extract it to disk, which is unnecessary. This function was probably written prior to us requiring Python 2.6, but now that we require 2.7 we can either use ZipFile.extract:
https://docs.python.org/2/library/zipfile.html#zipfile.ZipFile.extract

or rewrite this whole function to use ZipFile.extractall:
https://docs.python.org/2/library/zipfile.html#zipfile.ZipFile.extractall

Comment 1

[Nathan Froyd [:froydnj]]

Attachment: use ZipFile.extract in mozfile

Reading the whole zip entry into memory is inefficient and can cause
OOMs if the entry is large enough.  Let the ZipFile object choose the
most efficient extraction strategy instead.

We'll wait until Ted unsets the not-accepting-reviews flag on his profile. :)

Comment 2

[(not currently active) Ted Mielczarek]

Comment on attachment 8914286 [details] [diff] [review]
use ZipFile.extract in mozfile

Review of attachment 8914286 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!

Comment 3

[Nathan Froyd [:froydnj]]

Attachment: use ZipFile.extract in mozfile

Turns out ZipFile.extract does all the logic we were doing to makedirs, etc.,
and we needed to pass in the directory to extract to, not the filename to write
into.  This is a nicer patch in some ways!

Comment 4

[(not currently active) Ted Mielczarek]

Comment on attachment 8921202 [details] [diff] [review]
use ZipFile.extract in mozfile

Review of attachment 8921202 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6a828043c22d
use ZipFile.extract in mozfile; r=ted.mielczarek

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/6a828043c22d