JarWriter under Windows creates archives which cause no file permissions if extracted under Linux or Mac

NEW
Unassigned

Status

3 years ago
8 months ago

People

(Reporter: whimboo, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
mozfile.extract_zip() fails in setting the correct file permissions for archives which seem to be have created under Windows. You can test this on your Linux machine with the following archive:

http://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/firefox-46.0a1.en-US.win64.common.tests.zip

Just download and execute the following:

> import mozfile
> mozfile.extract_zip(%archive_path%, %target%)

In this case the external_attr and internal_attr property of each file is 0, which also results in a mode value of 0:

17:30:47     INFO - Using ZipFile to extract tests_win32.zip to b
bin/plugins/gmp-fakeopenh264/1.0/fakeopenh264.dll - 0 (0)
bin/plugins/gmp-fakeopenh264/1.0/fakeopenh264.info - 0 (0)
bin/plugins/gmp-fakeopenh264/1.0/fakeopenh264.voucher - 0 (0)

Compared to Linux:

17:30:47     INFO - Using ZipFile to extract tests_linux.zip to b
bin/plugins/gmp-fakeopenh264/1.0/fakeopenh264.info - 420 (2175008768)
bin/plugins/gmp-fakeopenh264/1.0/fakeopenh264.voucher - 420 (2175008768)
bin/plugins/gmp-fakeopenh264/1.0/libfakeopenh264.so - 493 (2179792896)

Especially under Linux and Mac the extracted files are NOT usable and cause our Firefox UI tests to fail, e.g.

> $ pip install -r config/firefox_ui_requirements.txt
> Could not open requirements file: [Errno 13] Permission denied: 'config/firefox_ui_requirements.txt'

Keep in mind that all is working fine for the test archive created under Linux and Mac. So I'm unsure if that is a bug in how we create the archives, which makes it invalid, or if we also have to adjust mozfile to correctly extract the files so it can cope with partial invalid archives.

This is something which you can see if you are trapped by bug 1239808.

My question is if we only set the permissions if the checked external_attr is not 0. This fixes the problem for me whatever platform zip I'm using.
Flags: needinfo?(ahalberstadt)
Sorry, I don't really know anything about this. Sounds like you have a patch that fixes it though?
Flags: needinfo?(ahalberstadt)

Comment 2

3 years ago
This feels like a problem with archive generation, which occurs as part of the build system.

We're using a Python-based archiving tool. It should be normalizing the permissions so things are consistent. It sounds like there is a bug there.
Component: Mozbase → Build Config
Product: Testing → Core

Comment 3

3 years ago
Here is the call stack for dealing with file mode:

https://hg.mozilla.org/mozilla-central/file/bd8bb6298d90/python/mozbuild/mozbuild/action/test_archive.py#l369
https://hg.mozilla.org/mozilla-central/file/bd8bb6298d90/python/mozbuild/mozpack/mozjar.py#l612

We appear to be grabbing and setting the file mode when archiving. I'm not sure why there is an inconsistency between Windows and Linux. Perhaps we should be normalizing the mode more. Let me debug this on my Windows machine to see if I can find anything interesting...
(Reporter)

Comment 4

3 years ago
FYI when I move a file inside the archive to another folder, its permissions will be corrected and a following extraction will apply the correct permissions.
(Reporter)

Updated

3 years ago
Summary: mozfile.extract_zip() does not always set the correct file permissions (e.g. archives created under Windows) → JarWriter under Windows creates archives which cause no file permissions if extracted under Linux or Mac
(Reporter)

Comment 5

3 years ago
Hopefully bug 1239808 will be fixed soon, so it should not be a blocker for us anymore.
Whiteboard: [qa-automation-blocked]
(Reporter)

Comment 6

3 years ago
We are no longer blocked on this issue given that bug 1239808 added platform dependent *.test_packages.json files.
Whiteboard: [qa-automation-blocked]

Updated

8 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.