1242592 - JarWriter under Windows creates archives which cause no file permissions if extracted under Linux or Mac

Status: NEW

(Firefox Build System :: General, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

mozfile.extract_zip() fails in setting the correct file permissions for archives which seem to be have created under Windows. You can test this on your Linux machine with the following archive:

http://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/firefox-46.0a1.en-US.win64.common.tests.zip

Just download and execute the following:

> import mozfile
> mozfile.extract_zip(%archive_path%, %target%)

In this case the external_attr and internal_attr property of each file is 0, which also results in a mode value of 0:

17:30:47     INFO - Using ZipFile to extract tests_win32.zip to b
bin/plugins/gmp-fakeopenh264/1.0/fakeopenh264.dll - 0 (0)
bin/plugins/gmp-fakeopenh264/1.0/fakeopenh264.info - 0 (0)
bin/plugins/gmp-fakeopenh264/1.0/fakeopenh264.voucher - 0 (0)

Compared to Linux:

17:30:47     INFO - Using ZipFile to extract tests_linux.zip to b
bin/plugins/gmp-fakeopenh264/1.0/fakeopenh264.info - 420 (2175008768)
bin/plugins/gmp-fakeopenh264/1.0/fakeopenh264.voucher - 420 (2175008768)
bin/plugins/gmp-fakeopenh264/1.0/libfakeopenh264.so - 493 (2179792896)

Especially under Linux and Mac the extracted files are NOT usable and cause our Firefox UI tests to fail, e.g.

> $ pip install -r config/firefox_ui_requirements.txt
> Could not open requirements file: [Errno 13] Permission denied: 'config/firefox_ui_requirements.txt'

Keep in mind that all is working fine for the test archive created under Linux and Mac. So I'm unsure if that is a bug in how we create the archives, which makes it invalid, or if we also have to adjust mozfile to correctly extract the files so it can cope with partial invalid archives.

This is something which you can see if you are trapped by bug 1239808.

My question is if we only set the permissions if the checked external_attr is not 0. This fixes the problem for me whatever platform zip I'm using.

Comment 1

[Andrew Halberstadt [:ahal]]

Sorry, I don't really know anything about this. Sounds like you have a patch that fixes it though?

Comment 2

[Gregory Szorc [:gps]]

This feels like a problem with archive generation, which occurs as part of the build system.

We're using a Python-based archiving tool. It should be normalizing the permissions so things are consistent. It sounds like there is a bug there.

Comment 3

[Gregory Szorc [:gps]]

Here is the call stack for dealing with file mode:

https://hg.mozilla.org/mozilla-central/file/bd8bb6298d90/python/mozbuild/mozbuild/action/test_archive.py#l369
https://hg.mozilla.org/mozilla-central/file/bd8bb6298d90/python/mozbuild/mozpack/mozjar.py#l612

We appear to be grabbing and setting the file mode when archiving. I'm not sure why there is an inconsistency between Windows and Linux. Perhaps we should be normalizing the mode more. Let me debug this on my Windows machine to see if I can find anything interesting...

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

FYI when I move a file inside the archive to another folder, its permissions will be corrected and a following extraction will apply the correct permissions.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hopefully bug 1239808 will be fixed soon, so it should not be a blocker for us anymore.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We are no longer blocked on this issue given that bug 1239808 added platform dependent *.test_packages.json files.