Closed Bug 991507 Opened 6 years ago Closed 6 years ago

[mozversion] BadZipfile raised when run against tarako device

Categories

(Testing :: Mozbase, defect, P1)

ARM
Gonk (Firefox OS)
defect

Tracking

(Not tracked)

RESOLVED FIXED
mozilla31

People

(Reporter: davehunt, Assigned: davehunt)

References

Details

(Keywords: ateam-b2g-device-task)

Attachments

(1 file)

It appears that the application.zip on tarako devices behaves differently. The following is thrown when running mozversion against a tarako device:

Traceback (most recent call last):
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/mozversion_temp/bin/mozversion", line 9, in <module>
    load_entry_point('mozversion==0.2', 'console_scripts', 'mozversion')()
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/mozversion_temp/local/lib/python2.7/site-packages/mozversion/mozversion.py", line 216, in cli
    dm_type=dm_type, host=host)
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/mozversion_temp/local/lib/python2.7/site-packages/mozversion/mozversion.py", line 198, in get_version
    version = RemoteB2GVersion(sources=sources, dm_type=dm_type, host=host)
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/mozversion_temp/local/lib/python2.7/site-packages/mozversion/mozversion.py", line 162, in __init__
    zip_file = zipfile.ZipFile(StringIO(dm.pullFile(path)))
  File "/usr/lib/python2.7/zipfile.py", line 714, in __init__
    self._GetContents()
  File "/usr/lib/python2.7/zipfile.py", line 748, in _GetContents
    self._RealGetContents()
  File "/usr/lib/python2.7/zipfile.py", line 788, in _RealGetContents
    raise BadZipfile, "Bad magic number for central directory"
zipfile.BadZipfile: Bad magic number for central directory
Blocks: 964588
This happens because the application.zip files produced for the Tarako builds aren't standard zipfiles; none of them can be opened with a simple unzip.  I'm not quite sure what they are...Fabrice, do you have any idea?
Flags: needinfo?(fabrice)
They are "optimized jars", like what we do for the omni.ja on most platforms. Why do we need to open them?
Flags: needinfo?(fabrice)
We extract the gaia commit from a file in the settings app's application.zip.  We can't rely on sources.xml, since a version of gaia may have been flashed onto the device since sources.xml was written.

Is there a better way to get the gaia commit?
Flags: needinfo?(fabrice)
Ok, linux unzip can unzip this, even though Python's unzip can't.  We should update mozversion to use linux unzip as a fallback here, I think.
Flags: needinfo?(fabrice)
Priority: -- → P1
Do you mean bundle Linux's unzip as a resource, or fall back to using the unzip on the system? This would restrict mozversion to only working on compatible platforms. Is there a way to convert this zip file into one we can unzip with Python?

I also just tried unzipping with busybox. While this works with an application.zip from a hamachi build, it fails with one from the tarako build with: "unzip: invalid zip magic 00085F0A"

Note that on Mac, the unzip is successful but a warning is printed:

$ unzip /Users/dhunt/Downloads/gaia/profile/webapps/settings.gaiamobile.org/application.zip resources/gaia_commit.txt
Archive:  /Users/dhunt/Downloads/gaia/profile/webapps/settings.gaiamobile.org/application.zip
warning [/Users/dhunt/Downloads/gaia/profile/webapps/settings.gaiamobile.org/application.zip]:  2636546 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [/Users/dhunt/Downloads/gaia/profile/webapps/settings.gaiamobile.org/application.zip]:  reported length of central directory is
  -2636546 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
  zipfile?).  Compensating...
  inflating: resources/gaia_commit.txt
Gregory might have some input for us here?
Flags: needinfo?(gps)
This patch falls back to the system's unzip command if Python is unable to read the file. I'd still prefer to explore if we can get this information from outside of the zip file, or somehow convert the zip file so we can avoid this path. This is blocking bug 964588 though, so we may have to incur some technical debt in order to get something running.
Assignee: nobody → dave.hunt
Status: NEW → ASSIGNED
Attachment #8403204 - Flags: review?(jgriffin)
Ostensibly the python script here can unpack/repack omni.ja as needed. I suspect it's rather slow though: http://sourceforge.net/p/sevenzip/discussion/45797/thread/dab0da38/
Flags: needinfo?(gps)
Comment on attachment 8403204 [details] [diff] [review]
Use system unzip command if Python cannot read the zip file. v1.0

Review of attachment 8403204 [details] [diff] [review]:
-----------------------------------------------------------------

Let's land this for now, since timing is important.  Longer-term, as discussed, we'll file a bug to make a better way to extract the gaia build revision from a device, and if that gets no traction, we can look at importing the omnijar unzipper that wlach mentioned.

::: testing/mozbase/mozversion/mozversion/mozversion.py
@@ +106,5 @@
> +                self.info('Unable to unzip application.zip, falling back to '
> +                          'system unzip')
> +                from subprocess import call
> +                call(['unzip', '-j', app_zip.name, 'resources/gaia_commit.txt',
> +                      '-d', tempdir])

Can we use subprocess.check_output here instead of call, and then log the output?  This will prevent us from having silent failures which are hard to debug.
Attachment #8403204 - Flags: review?(jgriffin) → review+
(In reply to Jonathan Griffin (:jgriffin) from comment #9)
> Comment on attachment 8403204 [details] [diff] [review]
> Use system unzip command if Python cannot read the zip file. v1.0
> 
> Review of attachment 8403204 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Let's land this for now, since timing is important.  Longer-term, as
> discussed, we'll file a bug to make a better way to extract the gaia build
> revision from a device, and if that gets no traction, we can look at
> importing the omnijar unzipper that wlach mentioned.

I've raised bug 993697 for obtaining the gaia changeset and date another way.

> ::: testing/mozbase/mozversion/mozversion/mozversion.py
> @@ +106,5 @@
> > +                self.info('Unable to unzip application.zip, falling back to '
> > +                          'system unzip')
> > +                from subprocess import call
> > +                call(['unzip', '-j', app_zip.name, 'resources/gaia_commit.txt',
> > +                      '-d', tempdir])
> 
> Can we use subprocess.check_output here instead of call, and then log the
> output?  This will prevent us from having silent failures which are hard to
> debug.

Unfortunately this throws subprocess.CalledProcessError because of the unusual zip file. As discussed on IRC, we're going to land this using call for now.
Keywords: checkin-needed
Blocks: 993700
https://hg.mozilla.org/mozilla-central/rev/edcce2158c32
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.