Closed
Bug 991507
Opened 10 years ago
Closed 10 years ago
[mozversion] BadZipfile raised when run against tarako device
Categories
(Testing :: Mozbase, defect, P1)
Tracking
(Not tracked)
RESOLVED
FIXED
mozilla31
People
(Reporter: davehunt, Assigned: davehunt)
References
Details
(Keywords: ateam-b2g-device-task)
Attachments
(1 file)
3.95 KB,
patch
|
jgriffin
:
review+
|
Details | Diff | Splinter Review |
It appears that the application.zip on tarako devices behaves differently. The following is thrown when running mozversion against a tarako device: Traceback (most recent call last): File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/mozversion_temp/bin/mozversion", line 9, in <module> load_entry_point('mozversion==0.2', 'console_scripts', 'mozversion')() File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/mozversion_temp/local/lib/python2.7/site-packages/mozversion/mozversion.py", line 216, in cli dm_type=dm_type, host=host) File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/mozversion_temp/local/lib/python2.7/site-packages/mozversion/mozversion.py", line 198, in get_version version = RemoteB2GVersion(sources=sources, dm_type=dm_type, host=host) File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/mozversion_temp/local/lib/python2.7/site-packages/mozversion/mozversion.py", line 162, in __init__ zip_file = zipfile.ZipFile(StringIO(dm.pullFile(path))) File "/usr/lib/python2.7/zipfile.py", line 714, in __init__ self._GetContents() File "/usr/lib/python2.7/zipfile.py", line 748, in _GetContents self._RealGetContents() File "/usr/lib/python2.7/zipfile.py", line 788, in _RealGetContents raise BadZipfile, "Bad magic number for central directory" zipfile.BadZipfile: Bad magic number for central directory
Comment 1•10 years ago
|
||
This happens because the application.zip files produced for the Tarako builds aren't standard zipfiles; none of them can be opened with a simple unzip. I'm not quite sure what they are...Fabrice, do you have any idea?
Flags: needinfo?(fabrice)
Comment 2•10 years ago
|
||
They are "optimized jars", like what we do for the omni.ja on most platforms. Why do we need to open them?
Flags: needinfo?(fabrice)
Comment 3•10 years ago
|
||
We extract the gaia commit from a file in the settings app's application.zip. We can't rely on sources.xml, since a version of gaia may have been flashed onto the device since sources.xml was written. Is there a better way to get the gaia commit?
Flags: needinfo?(fabrice)
Comment 4•10 years ago
|
||
Ok, linux unzip can unzip this, even though Python's unzip can't. We should update mozversion to use linux unzip as a fallback here, I think.
Flags: needinfo?(fabrice)
Updated•10 years ago
|
Keywords: ateam-b2g-device-task
Priority: -- → P1
Assignee | ||
Comment 5•10 years ago
|
||
Do you mean bundle Linux's unzip as a resource, or fall back to using the unzip on the system? This would restrict mozversion to only working on compatible platforms. Is there a way to convert this zip file into one we can unzip with Python? I also just tried unzipping with busybox. While this works with an application.zip from a hamachi build, it fails with one from the tarako build with: "unzip: invalid zip magic 00085F0A" Note that on Mac, the unzip is successful but a warning is printed: $ unzip /Users/dhunt/Downloads/gaia/profile/webapps/settings.gaiamobile.org/application.zip resources/gaia_commit.txt Archive: /Users/dhunt/Downloads/gaia/profile/webapps/settings.gaiamobile.org/application.zip warning [/Users/dhunt/Downloads/gaia/profile/webapps/settings.gaiamobile.org/application.zip]: 2636546 extra bytes at beginning or within zipfile (attempting to process anyway) error [/Users/dhunt/Downloads/gaia/profile/webapps/settings.gaiamobile.org/application.zip]: reported length of central directory is -2636546 bytes too long (Atari STZip zipfile? J.H.Holm ZIPSPLIT 1.1 zipfile?). Compensating... inflating: resources/gaia_commit.txt
Assignee | ||
Comment 7•10 years ago
|
||
This patch falls back to the system's unzip command if Python is unable to read the file. I'd still prefer to explore if we can get this information from outside of the zip file, or somehow convert the zip file so we can avoid this path. This is blocking bug 964588 though, so we may have to incur some technical debt in order to get something running.
Comment 8•10 years ago
|
||
Ostensibly the python script here can unpack/repack omni.ja as needed. I suspect it's rather slow though: http://sourceforge.net/p/sevenzip/discussion/45797/thread/dab0da38/
Flags: needinfo?(gps)
Comment 9•10 years ago
|
||
Comment on attachment 8403204 [details] [diff] [review] Use system unzip command if Python cannot read the zip file. v1.0 Review of attachment 8403204 [details] [diff] [review]: ----------------------------------------------------------------- Let's land this for now, since timing is important. Longer-term, as discussed, we'll file a bug to make a better way to extract the gaia build revision from a device, and if that gets no traction, we can look at importing the omnijar unzipper that wlach mentioned. ::: testing/mozbase/mozversion/mozversion/mozversion.py @@ +106,5 @@ > + self.info('Unable to unzip application.zip, falling back to ' > + 'system unzip') > + from subprocess import call > + call(['unzip', '-j', app_zip.name, 'resources/gaia_commit.txt', > + '-d', tempdir]) Can we use subprocess.check_output here instead of call, and then log the output? This will prevent us from having silent failures which are hard to debug.
Attachment #8403204 -
Flags: review?(jgriffin) → review+
Assignee | ||
Comment 10•10 years ago
|
||
(In reply to Jonathan Griffin (:jgriffin) from comment #9) > Comment on attachment 8403204 [details] [diff] [review] > Use system unzip command if Python cannot read the zip file. v1.0 > > Review of attachment 8403204 [details] [diff] [review]: > ----------------------------------------------------------------- > > Let's land this for now, since timing is important. Longer-term, as > discussed, we'll file a bug to make a better way to extract the gaia build > revision from a device, and if that gets no traction, we can look at > importing the omnijar unzipper that wlach mentioned. I've raised bug 993697 for obtaining the gaia changeset and date another way. > ::: testing/mozbase/mozversion/mozversion/mozversion.py > @@ +106,5 @@ > > + self.info('Unable to unzip application.zip, falling back to ' > > + 'system unzip') > > + from subprocess import call > > + call(['unzip', '-j', app_zip.name, 'resources/gaia_commit.txt', > > + '-d', tempdir]) > > Can we use subprocess.check_output here instead of call, and then log the > output? This will prevent us from having silent failures which are hard to > debug. Unfortunately this throws subprocess.CalledProcessError because of the unusual zip file. As discussed on IRC, we're going to land this using call for now.
Keywords: checkin-needed
Comment 11•10 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/edcce2158c32
Keywords: checkin-needed
Target Milestone: --- → mozilla31
Comment 12•10 years ago
|
||
https://hg.mozilla.org/mozilla-central/rev/edcce2158c32
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•