Closed Bug 1514806 Opened 10 months ago Closed 9 months ago

Improve full installer file extraction performance on AArch64

Categories

(Firefox :: Installer, enhancement, P1)

ARM64
Windows
enhancement

Tracking

()

RESOLVED FIXED
Firefox 66
Tracking Status
firefox66 --- fixed

People

(Reporter: mhowell, Assigned: mhowell)

References

(Blocks 2 open bugs)

Details

Attachments

(8 files)

328.00 KB, application/octet-stream
Details
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
On the HP Envy x2 (with a Snapdragon 835), the file extraction process in the 7-zip SFX module takes about 13 seconds for a release build. I'm not sure that's slow enough to cause stub installer timeouts (the rest of the installation is as fast as it is on anything), but we should certainly improve that if we can. This probably means building a native AArch64 copy of the SFX module, so we don't leave the x86 emulator trying to handle such a CPU-intensive piece of code.
This binary runs about twice as fast as the emulated x86 one on the machine I have. I'd say that proves the concept.

It wasn't as hard to make this build as I thought it would be; I had Visual Studio convert the VC++6 workspace into a VC2017 solution, added the ARM64 platform, switched to the statically linked C runtime, and then turned on a couple of things like link-time code generation to get to a reasonable file size.

Before I try to land anything, I want to polish up those automatically converted project files so they can be checked in, have some documentation in order, and patch the installer packaging code so that it automatically decides whether to use this or the x86 binary. I might also rebuild the x86 binary using the same method, so we can get rid of its arcane build procedure and dependency on VC6.
(In reply to Matt Howell (he/him) [:mhowell] (out until Jan 7) from comment #1)
> I might also rebuild the x86
> binary using the same method, so we can get rid of its arcane build
> procedure and dependency on VC6.

But the installer will no longer run on downlevel systems (and display a custom error message). So officially shipped x86 binaries should continue to use VC6. But it will help people that make their own builds without VC6.
This is not a problem for aarch64, of course.
(In reply to Masatoshi Kimura [:emk] from comment #3)
> But the installer will no longer run on downlevel systems (and display a
> custom error message). So officially shipped x86 binaries should continue to
> use VC6. But it will help people that make their own builds without VC6.
> This is not a problem for aarch64, of course.

That's correct, and it's the main reason I haven't made any change like this before. But we should be able to get as far back as WinXP SP3 using the special toolchain designed for that purpose, which I think is likely to be far enough downlevel.

(In reply to Matt Howell (he/him) [:mhowell] from comment #4)

(In reply to Masatoshi Kimura [:emk] from comment #3)

But the installer will no longer run on downlevel systems (and display a
custom error message). So officially shipped x86 binaries should continue to
use VC6. But it will help people that make their own builds without VC6.
This is not a problem for aarch64, of course.

That's correct, and it's the main reason I haven't made any change like this
before. But we should be able to get as far back as WinXP SP3 using the
special toolchain designed for that purpose, which I think is likely to be
far enough downlevel.

I'm going to consider this change in a separate bug, rather than trying to roll it into this bug which isn't really related.

Pushed by mhowell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/303b19a8b923
Part 1 - Rename the existing x86 7-zip SFX files to indicate what platform they're for. r=agashlin
https://hg.mozilla.org/integration/autoland/rev/6a738993de29
Part 2 - Add build files necessary for compiling a 7-zip SFX stub for AArch64. r=agashlin
https://hg.mozilla.org/integration/autoland/rev/eb09e108a439
Part 3 - Add the new AArch64 7-zip SFX stub binary. r=agashlin
https://hg.mozilla.org/integration/autoland/rev/3b95c5d8466d
Part 4 - Update the LZMA SDK package documentation to explain the new files. r=agashlin
https://hg.mozilla.org/integration/autoland/rev/803d20ba3f80
Part 5 - Select the appropriate 7-zip SFX stub binary for the build target. r=firefox-build-system-reviewers,mshal#firefox-build-system-reviewers
https://hg.mozilla.org/integration/autoland/rev/63debb1d02d3
Part 6 - Disable running UPX in the installer package for targets it doesn't support. r=firefox-build-system-reviewers,mshal#firefox-build-system-reviewers

Backed out for causing windows 2012 opt repack bustages.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=busted%2Csuccess%2Crunning%2Crunnable&searchStr=windows%2C2012%2Cx64%2Copt%2Crepackage-win64%2Fopt%2C%28rpk%29&fromchange=63debb1d02d3ca285def538c88c1e47295923613&tochange=113f3db802c4ed0408d8cca513c595959d1be8a7&selectedJob=222006435

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=222006435&repo=autoland&lineNumber=731

Backout link: https://hg.mozilla.org/integration/autoland/rev/113f3db802c4ed0408d8cca513c595959d1be8a7

17:08:39 INFO - Copy/paste: c:\mozilla-build\python\python.exe mach --log-no-times repackage installer --package-name firefox --tag browser/installer/windows/app.tag --sfx-stub other-licenses/7zstub/firefox/7zSD.sfx --setupexe Z:\task_1547569741\fetches\setup.exe --package Z:\task_1547569741\fetches\target.zip --output z:\task_1547569741\build\outputs\target.installer.exe
17:09:26 INFO - 7-Zip 18.05 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-04-30
17:09:26 INFO - Scanning the drive:
17:09:26 INFO - 10 folders, 106 files, 183896527 bytes (176 MiB)
17:09:26 INFO - Creating archive: c:/users/task_1547569741/appdata/local/temp/tmpsl4dx4/app.7z
17:09:26 INFO - Add new data to archive: 10 folders, 106 files, 183896527 bytes (176 MiB)
17:09:26 INFO - Files read from disk: 103
17:09:26 INFO - Archive size: 44899772 bytes (43 MiB)
17:09:26 INFO - Everything is Ok
17:09:26 INFO - Error running mach:
17:09:26 INFO - ['--log-no-times', 'repackage', 'installer', '--package-name', 'firefox', '--tag', 'browser/installer/windows/app.tag', '--sfx-stub', 'other-licenses/7zstub/firefox/7zSD.sfx', '--setupexe', 'Z:\task_1547569741\fetches\setup.exe', '--package', 'Z:\task_1547569741\fetches\target.zip', '--output', 'z:\task_1547569741\build\outputs\target.installer.exe']
17:09:26 INFO - The error occurred in code that was called by the mach command. This is either
17:09:26 INFO - a bug in the called code itself or in the way that mach is calling it.
17:09:26 INFO - You should consider filing a bug for this issue.
17:09:26 INFO - If filing a bug, please include the full output of mach, including this error
17:09:26 INFO - message.
17:09:26 INFO - The details of the failure are as follows:
17:09:26 INFO - IOError: [Errno 2] No such file or directory: 'z:/task_1547569741/build/src/other-licenses/7zstub/firefox/7zSD.sfx'
17:09:26 INFO - File "z:\task_1547569741\build\src\python/mozbuild/mozbuild/mach_commands.py", line 3069, in repackage_installer
17:09:26 INFO - use_upx=use_upx,
17:09:26 INFO - File "z:\task_1547569741\build\src\python/mozbuild\mozbuild\repackaging\installer.py", line 45, in repackage_installer
17:09:26 INFO - archive_exe(package_name, tag, sfx_package, output, use_upx)
17:09:26 INFO - File "z:\task_1547569741\build\src\python/mozbuild\mozbuild\action\exe_7z_archive.py", line 37, in archive_exe
17:09:26 INFO - shutil.copyfileobj(open(i, 'rb'), o)
17:09:26 ERROR - Return code: 1
17:09:26 ERROR - 1 not in success codes: [0]
17:09:26 WARNING - setting return code to 2
17:09:26 FATAL - Halting on failure while running ['c:\mozilla-build\python\python.exe', 'mach', '--log-no-times', 'repackage', u'installer', u'--package-name', u'firefox', u'--tag', u'browser/installer/windows/app.tag', u'--sfx-stub', u'other-licenses/7zstub/firefox/7zSD.sfx', '--setupexe', u'Z:\task_1547569741\fetches\setup.exe', '--package', u'Z:\task_1547569741\fetches\target.zip', '--output', u'z:\task_1547569741\build\outputs\target.installer.exe']
17:09:26 FATAL - Running post_fatal callback...
17:09:26 FATAL - Exiting 2

Flags: needinfo?(mhowell)

I've pushed new versions of parts 5 and 6; these seem to have fixed the failing jobs, which I verified on try. The changes are significant enough to need re-review, so I've also reopened the requests for those parts.

Flags: needinfo?(mhowell)
Pushed by mhowell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1a0c0bcbc31f
Part 1 - Rename the existing x86 7-zip SFX files to indicate what platform they're for. r=agashlin
https://hg.mozilla.org/integration/autoland/rev/2a0392761736
Part 2 - Add build files necessary for compiling a 7-zip SFX stub for AArch64. r=agashlin
https://hg.mozilla.org/integration/autoland/rev/4337d1cf1a99
Part 3 - Add the new AArch64 7-zip SFX stub binary. r=agashlin
https://hg.mozilla.org/integration/autoland/rev/f66cc702c86d
Part 4 - Update the LZMA SDK package documentation to explain the new files. r=agashlin
https://hg.mozilla.org/integration/autoland/rev/ef7f2df97310
Part 5 - Select the appropriate 7-zip SFX stub binary for the build target. r=firefox-build-system-reviewers,mshal#firefox-build-system-reviewers
https://hg.mozilla.org/integration/autoland/rev/11a6b0fbd289
Part 6 - Disable running UPX in the installer package for targets it doesn't support. r=firefox-build-system-reviewers,mshal#firefox-build-system-reviewers
Assignee: nobody → mhowell
Blocks: 1522928
Attachment #9039283 - Attachment description: Bug 1514806: Point partner repack repackage tasks at repackage configs. → Bug 1514806: Point partner repack repackage tasks at partner repackage configs.
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/ad292914ca7e
Point partner repack repackage tasks at partner repackage configs. r=aki
You need to log in before you can comment on or make changes to this bug.