Improve full installer file extraction performance on AArch64
Categories
(Firefox :: Installer, enhancement, P1)
Tracking
()
Tracking | Status | |
---|---|---|
firefox66 | --- | fixed |
People
(Reporter: molly, Assigned: molly)
References
(Blocks 2 open bugs)
Details
Attachments
(8 files)
328.00 KB,
application/octet-stream
|
Details | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review |
On the HP Envy x2 (with a Snapdragon 835), the file extraction process in the 7-zip SFX module takes about 13 seconds for a release build. I'm not sure that's slow enough to cause stub installer timeouts (the rest of the installation is as fast as it is on anything), but we should certainly improve that if we can. This probably means building a native AArch64 copy of the SFX module, so we don't leave the x86 emulator trying to handle such a CPU-intensive piece of code.
Assignee | ||
Comment 1•5 years ago
|
||
This binary runs about twice as fast as the emulated x86 one on the machine I have. I'd say that proves the concept. It wasn't as hard to make this build as I thought it would be; I had Visual Studio convert the VC++6 workspace into a VC2017 solution, added the ARM64 platform, switched to the statically linked C runtime, and then turned on a couple of things like link-time code generation to get to a reasonable file size. Before I try to land anything, I want to polish up those automatically converted project files so they can be checked in, have some documentation in order, and patch the installer packaging code so that it automatically decides whether to use this or the x86 binary. I might also rebuild the x86 binary using the same method, so we can get rid of its arcane build procedure and dependency on VC6.
Comment 2•5 years ago
|
||
\o/
Comment 3•5 years ago
|
||
(In reply to Matt Howell (he/him) [:mhowell] (out until Jan 7) from comment #1) > I might also rebuild the x86 > binary using the same method, so we can get rid of its arcane build > procedure and dependency on VC6. But the installer will no longer run on downlevel systems (and display a custom error message). So officially shipped x86 binaries should continue to use VC6. But it will help people that make their own builds without VC6. This is not a problem for aarch64, of course.
Assignee | ||
Comment 4•5 years ago
|
||
(In reply to Masatoshi Kimura [:emk] from comment #3) > But the installer will no longer run on downlevel systems (and display a > custom error message). So officially shipped x86 binaries should continue to > use VC6. But it will help people that make their own builds without VC6. > This is not a problem for aarch64, of course. That's correct, and it's the main reason I haven't made any change like this before. But we should be able to get as far back as WinXP SP3 using the special toolchain designed for that purpose, which I think is likely to be far enough downlevel.
Assignee | ||
Comment 5•5 years ago
|
||
(In reply to Matt Howell (he/him) [:mhowell] from comment #4)
(In reply to Masatoshi Kimura [:emk] from comment #3)
But the installer will no longer run on downlevel systems (and display a
custom error message). So officially shipped x86 binaries should continue to
use VC6. But it will help people that make their own builds without VC6.
This is not a problem for aarch64, of course.That's correct, and it's the main reason I haven't made any change like this
before. But we should be able to get as far back as WinXP SP3 using the
special toolchain designed for that purpose, which I think is likely to be
far enough downlevel.
I'm going to consider this change in a separate bug, rather than trying to roll it into this bug which isn't really related.
Assignee | ||
Comment 6•5 years ago
|
||
Assignee | ||
Comment 7•5 years ago
|
||
Depends on D15978
Assignee | ||
Comment 8•5 years ago
|
||
Depends on D15979
Assignee | ||
Comment 9•5 years ago
|
||
Depends on D15980
Assignee | ||
Comment 10•5 years ago
|
||
Depends on D15982
Assignee | ||
Comment 11•5 years ago
|
||
Depends on D15983
Comment 12•5 years ago
|
||
Pushed by mhowell@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/303b19a8b923 Part 1 - Rename the existing x86 7-zip SFX files to indicate what platform they're for. r=agashlin https://hg.mozilla.org/integration/autoland/rev/6a738993de29 Part 2 - Add build files necessary for compiling a 7-zip SFX stub for AArch64. r=agashlin https://hg.mozilla.org/integration/autoland/rev/eb09e108a439 Part 3 - Add the new AArch64 7-zip SFX stub binary. r=agashlin https://hg.mozilla.org/integration/autoland/rev/3b95c5d8466d Part 4 - Update the LZMA SDK package documentation to explain the new files. r=agashlin https://hg.mozilla.org/integration/autoland/rev/803d20ba3f80 Part 5 - Select the appropriate 7-zip SFX stub binary for the build target. r=firefox-build-system-reviewers,mshal#firefox-build-system-reviewers https://hg.mozilla.org/integration/autoland/rev/63debb1d02d3 Part 6 - Disable running UPX in the installer package for targets it doesn't support. r=firefox-build-system-reviewers,mshal#firefox-build-system-reviewers
Comment 13•5 years ago
|
||
Backed out for causing windows 2012 opt repack bustages.
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=222006435&repo=autoland&lineNumber=731
Backout link: https://hg.mozilla.org/integration/autoland/rev/113f3db802c4ed0408d8cca513c595959d1be8a7
17:08:39 INFO - Copy/paste: c:\mozilla-build\python\python.exe mach --log-no-times repackage installer --package-name firefox --tag browser/installer/windows/app.tag --sfx-stub other-licenses/7zstub/firefox/7zSD.sfx --setupexe Z:\task_1547569741\fetches\setup.exe --package Z:\task_1547569741\fetches\target.zip --output z:\task_1547569741\build\outputs\target.installer.exe
17:09:26 INFO - 7-Zip 18.05 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-04-30
17:09:26 INFO - Scanning the drive:
17:09:26 INFO - 10 folders, 106 files, 183896527 bytes (176 MiB)
17:09:26 INFO - Creating archive: c:/users/task_1547569741/appdata/local/temp/tmpsl4dx4/app.7z
17:09:26 INFO - Add new data to archive: 10 folders, 106 files, 183896527 bytes (176 MiB)
17:09:26 INFO - Files read from disk: 103
17:09:26 INFO - Archive size: 44899772 bytes (43 MiB)
17:09:26 INFO - Everything is Ok
17:09:26 INFO - Error running mach:
17:09:26 INFO - ['--log-no-times', 'repackage', 'installer', '--package-name', 'firefox', '--tag', 'browser/installer/windows/app.tag', '--sfx-stub', 'other-licenses/7zstub/firefox/7zSD.sfx', '--setupexe', 'Z:\task_1547569741\fetches\setup.exe', '--package', 'Z:\task_1547569741\fetches\target.zip', '--output', 'z:\task_1547569741\build\outputs\target.installer.exe']
17:09:26 INFO - The error occurred in code that was called by the mach command. This is either
17:09:26 INFO - a bug in the called code itself or in the way that mach is calling it.
17:09:26 INFO - You should consider filing a bug for this issue.
17:09:26 INFO - If filing a bug, please include the full output of mach, including this error
17:09:26 INFO - message.
17:09:26 INFO - The details of the failure are as follows:
17:09:26 INFO - IOError: [Errno 2] No such file or directory: 'z:/task_1547569741/build/src/other-licenses/7zstub/firefox/7zSD.sfx'
17:09:26 INFO - File "z:\task_1547569741\build\src\python/mozbuild/mozbuild/mach_commands.py", line 3069, in repackage_installer
17:09:26 INFO - use_upx=use_upx,
17:09:26 INFO - File "z:\task_1547569741\build\src\python/mozbuild\mozbuild\repackaging\installer.py", line 45, in repackage_installer
17:09:26 INFO - archive_exe(package_name, tag, sfx_package, output, use_upx)
17:09:26 INFO - File "z:\task_1547569741\build\src\python/mozbuild\mozbuild\action\exe_7z_archive.py", line 37, in archive_exe
17:09:26 INFO - shutil.copyfileobj(open(i, 'rb'), o)
17:09:26 ERROR - Return code: 1
17:09:26 ERROR - 1 not in success codes: [0]
17:09:26 WARNING - setting return code to 2
17:09:26 FATAL - Halting on failure while running ['c:\mozilla-build\python\python.exe', 'mach', '--log-no-times', 'repackage', u'installer', u'--package-name', u'firefox', u'--tag', u'browser/installer/windows/app.tag', u'--sfx-stub', u'other-licenses/7zstub/firefox/7zSD.sfx', '--setupexe', u'Z:\task_1547569741\fetches\setup.exe', '--package', u'Z:\task_1547569741\fetches\target.zip', '--output', u'z:\task_1547569741\build\outputs\target.installer.exe']
17:09:26 FATAL - Running post_fatal callback...
17:09:26 FATAL - Exiting 2
Assignee | ||
Comment 14•5 years ago
|
||
I've pushed new versions of parts 5 and 6; these seem to have fixed the failing jobs, which I verified on try. The changes are significant enough to need re-review, so I've also reopened the requests for those parts.
Comment 15•5 years ago
|
||
Pushed by mhowell@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/1a0c0bcbc31f Part 1 - Rename the existing x86 7-zip SFX files to indicate what platform they're for. r=agashlin https://hg.mozilla.org/integration/autoland/rev/2a0392761736 Part 2 - Add build files necessary for compiling a 7-zip SFX stub for AArch64. r=agashlin https://hg.mozilla.org/integration/autoland/rev/4337d1cf1a99 Part 3 - Add the new AArch64 7-zip SFX stub binary. r=agashlin https://hg.mozilla.org/integration/autoland/rev/f66cc702c86d Part 4 - Update the LZMA SDK package documentation to explain the new files. r=agashlin https://hg.mozilla.org/integration/autoland/rev/ef7f2df97310 Part 5 - Select the appropriate 7-zip SFX stub binary for the build target. r=firefox-build-system-reviewers,mshal#firefox-build-system-reviewers https://hg.mozilla.org/integration/autoland/rev/11a6b0fbd289 Part 6 - Disable running UPX in the installer package for targets it doesn't support. r=firefox-build-system-reviewers,mshal#firefox-build-system-reviewers
Comment 16•5 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/1a0c0bcbc31f
https://hg.mozilla.org/mozilla-central/rev/2a0392761736
https://hg.mozilla.org/mozilla-central/rev/4337d1cf1a99
https://hg.mozilla.org/mozilla-central/rev/f66cc702c86d
https://hg.mozilla.org/mozilla-central/rev/ef7f2df97310
https://hg.mozilla.org/mozilla-central/rev/11a6b0fbd289
Updated•5 years ago
|
Comment 17•5 years ago
|
||
Updated•5 years ago
|
Comment 18•5 years ago
|
||
Pushed by mozilla@hocat.ca: https://hg.mozilla.org/integration/autoland/rev/ad292914ca7e Point partner repack repackage tasks at partner repackage configs. r=aki
Comment 19•5 years ago
|
||
bugherder |
Description
•