Open Bug 1321922 Opened 5 years ago Updated 8 months ago
Support symlinks on Windows
Symlinks on NTFS exist. But they have historically required administrative privileges to create and therefore are effectively unusable by the build system. This appears to be changing! https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/ documents the new hotness. Hopefully a few months from now we can detect whether userland symlinks are supported on Windows and use them automatically. This should make a number of development related tasks on Windows significantly faster, as we'll no longer need to copy and keep synchronized thousands of files into the objdir.
See also bug 969647.
They definitely work on the Taskcluster builders, we're doing all of our Windows TC builds from inside a symlinked directory there: https://dxr.mozilla.org/mozilla-central/rev/6dbc6e9f62a705d5f523cc750811bd01c8275ec6/taskcluster/taskgraph/transforms/job/mozharness.py#224 I believe the Task user was explicitly granted symlink permission, however.
The last time we tried this we got hung up on the msys tools we were using for some parts of the build not supporting symlinks properly. I think we were using `tar` for copying files around or something like that and it blew up and corrupted the build. A lot of that code has changed lately, so it would be worthwhile to try again.
> A lot of that code has changed lately, so it would be worthwhile to try > again. Please do :) Thanks.
If someone wanted to take a crack at fixing this, the most useful place to start would probably be the mozpack code, since that's what we use to install files from install manifests, which covers things like `EXPORTS` and `FINAL_TARGET_FILES`. Currently the only way we turn on symlink usage is checking `hasattr(os, 'symlink')` here: https://dxr.mozilla.org/mozilla-central/rev/0528a414c2a86dad0623779abde5301d37337934/python/mozbuild/mozpack/copier.py#302 We'd need to make this a slightly more complicated check. If the script is running on Windows, and developer mode is enabled (there's a registry key you can check: https://stackoverflow.com/a/41232108/69326 ). Then we'd need to actually implement `AbsoluteSymlinkFile.copy` for Windows: https://dxr.mozilla.org/mozilla-central/rev/0528a414c2a86dad0623779abde5301d37337934/python/mozbuild/mozpack/files.py#321 Presumably you'd want to do that by calling `CreateSymbolicLinkW` directly, something like: http://blog.bfitz.us/?p=2035 (note that the blog post in comment 0 mentions that you need to pass SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE (0x2) in the flags!)
I was thinking about this again today and tried out some sample code. This Python code works to create symlinks on Windows 10: ``` SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE = 0x2 def symlink(path, target): flags = SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE path_buf = ctypes.create_unicode_buffer(path) target_buf = ctypes.create_unicode_buffer(target) return ctypes.windll.kernel32.CreateSymbolicLinkW(path_buf, target_buf, flags) != 0 ``` Here's a Python snippet to check if developer mode is enabled: ``` def developer_mode_enabled(): import _winreg try: key = _winreg.OpenKey( _winreg.HKEY_LOCAL_MACHINE, r'SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock') val, ty = _winreg.QueryValueEx(key, 'AllowDevelopmentWithoutDevLicense') if ty == _winreg.REG_DWORD: return val != 0 except WindowsError: pass return False ```
You are my hero, Ted!
I have a ton of patches in bug 1382697 that basically implement this. Many of them can probably land with minimal refactoring. IIRC that bug suffered from classical "biting off more than you can chew." The work should definitely be split up to land the basic refactoring and support for symlinks before any of the larger, more controversial and dangerous changes.
You need to log in before you can comment on or make changes to this bug.