Closed Bug 1767299 Opened 3 years ago Closed 3 years ago

`./mach build` fails with an "Unable to create process" error with MSStore-python + SSH

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: saschanaz, Unassigned)

References

(Blocks 1 open bug)

Details

When running ./mach build via SSH session in VSCode terminal:

> ./mach build                                                                                                                                                               0:01.80 Clobber not needed.
 0:02.19 Adding make options from C:\Users\sasch\Documents\GitHub\gecko-dev\mozconfig
    AUTOCLOBBER=1
    MOZ_OBJDIR=C:/Users/sasch/Documents/GitHub/gecko-dev/obj-x86_64-pc-mingw32
    OBJDIR=C:/Users/sasch/Documents/GitHub/gecko-dev/obj-x86_64-pc-mingw32
    FOUND_MOZCONFIG=C:/Users/sasch/Documents/GitHub/gecko-dev/mozconfig
    export FOUND_MOZCONFIG
  Parallelism determined by memory: using 64 jobs for 64 cores based on 63.9 GiB RAM and estimated job size of 1.0 GiB
 0:02.19 C:/Users/sasch/.mozbuild/mozmake/mozmake.exe -f client.mk -j64 -s
 0:05.20 Elapsed: 0.00s; From dist/private: Kept 0 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:05.20 Elapsed: 0.00s; From dist/public: Kept 0 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:05.33 Elapsed: 0.09s; From dist/xpi-stage: Kept 0 existing; Added/updated 94; Removed 0 files and 0 directories.
 0:06.25 Elapsed: 1.00s; From _tests: Kept 0 existing; Added/updated 1253; Removed 0 files and 0 directories.
 0:06.97 Elapsed: 1.72s; From dist/bin: Kept 0 existing; Added/updated 2706; Removed 0 files and 0 directories.
 0:07.95 Elapsed: 2.73s; From dist/include: Kept 0 existing; Added/updated 6790; Removed 0 files and 0 directories.
 0:08.13 ./buildid.h.stub
 0:08.14 ./source-repo.h.stub
 0:08.92 ./application.ini.stub
 0:08.94 ./IsCombiningDiacritic.h.stub
 0:09.34 ./application.ini.h.stub
 0:10.17 toolkit/library/rust/force-cargo-library-build
 0:10.17 testing/geckodriver/force-cargo-program-build
 0:10.30     Blocking waiting for file lock on package cache
 0:10.31 browser/app
 0:15.41     Blocking waiting for file lock on package cache
 0:15.77 netwerk/test/http3server/force-cargo-program-build
 0:15.80 browser/app/firefox.exe
 0:15.88     Blocking waiting for file lock on package cache
 0:19.30     Blocking waiting for file lock on package cache
 0:19.31     Blocking waiting for file lock on package cache
 0:22.84     Blocking waiting for file lock on package cache
 0:23.59     Finished release [unoptimized] target(s) in 7.81s
 0:23.63     Finished release [unoptimized] target(s) in 13.44s
 0:25.25     Finished release [unoptimized] target(s) in 15.07s
 0:25.45 security/manager/ssl/ipcclientcerts/force-cargo-library-build
 0:29.25     Finished release [unoptimized] target(s) in 3.79s
 0:29.39 security/manager/ssl/osclientcerts/force-cargo-library-build
 0:33.16     Finished release [unoptimized] target(s) in 3.75s
 0:33.30 toolkit/crashreporter/mozwer-rust/force-cargo-library-build
 0:37.06     Finished release [unoptimized] target(s) in 3.76s
 0:39.48 toolkit/mozapps/defaultagent/rust/force-cargo-library-build
 0:43.31     Finished release [unoptimized] target(s) in 3.82s
 0:43.91 Unable to create process using 'C:\Users\sasch\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\python.exe -m mozbuild.action.jar_maker -q -d ../../dist/bin/browser -t C:/Users/sasch/Documents/GitHub/gecko-dev -f flat --relativesrcdir=devtools/shared -c C:/Users/sasch/Documents/GitHub/gecko-dev/devtools/shared/en-US -DNDEBUG=1 -DTRIMMED=1 -DA11Y_LOG=1 -DACCESSIBILITY=1 -DBROWSER_CHROME_URL=chrome://browser/content/browser.xhtml -DBROWSER_CHROME_URL_QUOTED=\"chrome://browser/content/browser.xhtml\" -DBUILD_CTYPES=1 -DCROSS_COMPILE= -DEARLY_BETA_OR_EARLIER=1 -DENABLE_SHARED_MEMORY=1 -DENABLE_SPIDERMONKEY_TELEMETRY=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DENABLE_TESTS=1 -DENABLE_WASM_AVX=1 -DENABLE_WASM_EXCEPTIONS=1 -DENABLE_WASM_EXTENDED_CONST=1 -DENABLE_WASM_FUNCTION_REFERENCES=1 -DENABLE_WASM_GC=1 -DENABLE_WASM_MEMORY64=1 -DENABLE_WASM_MOZ_INTGEMM=1 -DENABLE_WASM_RELAXED_SIMD=1 -DENABLE_WASM_SIMD=1 -DENABLE_WASM_SIMD_WORMHOLE=1 -DENABLE_WASM_TYPE_REFLECTIONS=1 -DENABLE_WEBDR
 0:43.91 mozmake[4]: *** [C:/Users/sasch/Documents/GitHub/gecko-dev/config/rules.mk:851: misc] Error 101
 0:43.91 mozmake[3]: *** [C:/Users/sasch/Documents/GitHub/gecko-dev/config/recurse.mk:99: devtools/shared/misc] Error 2
 0:43.91 mozmake[3]: *** Waiting for unfinished jobs....
 0:46.36 mozmake[2]: *** [C:/Users/sasch/Documents/GitHub/gecko-dev/config/recurse.mk:34: misc] Error 2
 0:46.38 mozmake[1]: *** [C:/Users/sasch/Documents/GitHub/gecko-dev/config/rules.mk:352: default] Error 2
 0:46.38 mozmake: *** [client.mk:63: build] Error 2
 0:46.41 571 compiler warnings present.

This somehow does not happen without SSH. Not sure how it got non-virtualenv python path when calling jar_maker.

Hey Kagami,

Could you provide some more details about your use case? I'm not too familiar with VSCode SSH sessions, but I assume you're probably using it to connect to/from a VM or WSL?

Flags: needinfo?(krosylight)

Ah sorry, by VSCode SSH session I mean this: https://code.visualstudio.com/docs/remote/ssh

I'm using it to code and build on my remote machine in the office while being at home. No VM nor WSL here.

Flags: needinfo?(krosylight)

BTW, I wonder whether this also happens on raw SSH without VSCode. Will check tomorrow.

Flags: needinfo?(krosylight)

Same happens with pure SSH without VSCode (the Windows default provided version):

> ssh -V
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2

Could be an issue with SSH itself, though.

Flags: needinfo?(krosylight)

Hmm, I'm not able to reproduce this.

1.) Download and installed OpenSSH-x64-v8.9.1.0.msi on 2 of my Windows 10 machines.

C:\Users\Alex Hochheiden>ssh -V
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2

Version matches yours on both machines.
2.) Downloaded and installed Python3.10 from the Microsoft store on both Machines
3.) Open cmd, SSH from Windows Machine A into Windows Machine B

ssh -l <username> <local ip>

4.) Navigate to MozillaBuild (4.0.1) dir and run start-shell.bat
5.) In the MSYS2 shell, navigate to mozilla-central
6.) ./mach clobber
7.) ./mach configure
8.) ./mach build
9.) 8:32.12 Your build was successful!

I thought maybe you had an environment variable set for python3 pointing to C:\Users\sasch\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\python.exe and that might possibly cause problems, so I added that equivalent env var on both of my machines then re-ran the scenario and reproduce what you're seeing either.

Not that it should, going by the code here and here it should always pick the intended virtual env path.

What's the value of PYTHON3 in your */obj_dir/config.status file when you're seeing the issue?

Also, if you have any additional steps I could add so that I can reproduce it that would help too. You mentioned you encounter it without VSCode so I didn't go as far as trying it out with that.

Flags: needinfo?(krosylight)

Just for context as to how I got to the PYTHON3 variable: one of your errors "config/rules.mk:851: misc Error 101" calls py_action and that maps to here, which comes back to here from the previous comment.

1.) Download and installed OpenSSH-x64-v8.9.1.0.msi on 2 of my Windows 10 machines.

You don't have to, there is a builtin OpenSSH in Windows in Apps->Optional features. https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse#install-openssh-using-windows-settings (I believe this is enabled by default, but not sure...)

> (gcm ssh).source
C:\WINDOWS\System32\OpenSSH\ssh.exe

Not sure there's any difference, though.

2.) Downloaded and installed Python3.10 from the Microsoft store on both Machines

Are you sure you have Microsoft Store-distributed python, what do you get when you do (gcm python).source on PowerShell?

> (gcm python).source
C:\Users\sasch\AppData\Local\Microsoft\WindowsApps\python.exe

I thought maybe you had an environment variable set for python3 pointing to

No, there is no PYTHON3 nor python3 as env vars. (If there was such env variable it would have affected non-SSH native terminal too.)

What's the value of PYTHON3 in your */obj_dir/config.status file when you're seeing the issue?

    'PYTHON3': 'C:/Users/sasch/Documents/GitHub/gecko-dev/obj-x86_64-pc-mingw32/_virtualenvs/build/Scripts/python.exe',
    'PYTHON3_VERSION': '3.10.4',

I tried upgrading to MozillaBuild 4.0.1 from 4.0.0 but no luck.

If this helps, I'm on Windows 11 22610.1 on my local machine (beta channel) and 22000.613 on my remote machine (stable channel, where the actual build happens), both having the system builtin OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2.

Based on pure curiosity I also tried connecting ssh from the remote machine to itself (ssh localhost) and the same issue happens, so I guess the local machine being on beta channel shouldn't affect here.

Flags: needinfo?(krosylight)

I'm going off the assumption here that we diverged at step 4 (since you mentioned PowerShell)

4.) Navigate to MozillaBuild (4.0.1) dir and run start-shell.bat
and that you're invoking calling mach build directly from PowerShell.

Going that route I didn't make it very far. The MSStore Python doesn't have the python310.dll anywhere obvious that I can access. (failed to load Python DLL python310.dll). I tried some shenanigans with putting where it actually is (deep in C:\Program Files\WindowsApps) on the Path, but since I don't have any permissions on that folder (not even read), it didn't work.

In the guide Using Mach on Windows Outside MozillaBuild there's a snippet that says

Due to issues with Python DLL import failures with pip-installed binaries, it’s not recommended to use the Windows Store release of Python.

So I uninstalled the MSStore Python, and used the official Python installer instead and retried the scenario. I was able to build successfully by invoking mach build directly through PowerShell (instead of through MozillaBuild shell) over SSH on Python 3.10.4.

Are you able to use the official Python installer instead of the MSStore one? I've got a feeling that probably won't fix your issue, but it would help reduce the number of variables. Right now my hunch is that there's something in your local environment/config that's the problem, and not something in the build scripts.

Flags: needinfo?(krosylight)

Very interesting, I'm not getting such dll error here when building without SSH. But maybe next week.

Not sure how to reproduce the dll issue, is there some way to show some log about that?

The full build succeeds with MSStore Python without SSH, I wonder why the difference. Perhaps Windows 10 vs 11?

Flags: needinfo?(krosylight)

From what I could tell when I had MSStore Python installed the Python DLLs were just not included where the python.exe was (The path that gets added to PATH). It appeared that they were in some protected folder, I think C:\Program Files\WindowsApps, which I don't even have read permissions for. It seemed like I was getting way too deep down a 'hacky' path of troubleshooting so I pulled back and just went with the official Python instead of the MS one (And everything worked fine).

I'm assuming that either you have the equivalent DLLs somewhere on Path from a separate Python install (And they're interchangeable, because they're from the same Python release as the MSStore one, Microsoft just moves them) or you went deeper than I did and found a separate workaround for getting the 'hidden' DLLs.

Have you tried with the official Python through SSH? That worked for me, but I'm also on Windows 10 (And based on your last comment I assume you're on 11).

Flags: needinfo?(krosylight)

To be clear, I have been using the exe installer distribution and want to migrate to official (yes, it's also official but somehow it's not on python.org) MSStore one as that one autoupdates. And yes the exe distribution works via SSH.

I tried uninstalling everything else and ran ./mach clobber and still the build worked with MSStore distribution without SSH. Perhaps Windows 11 allows exposing the DLL in some way while Windows 10 does not?

Flags: needinfo?(krosylight)

I had a chat with :glob and we concluded that since we're currently not recommending to use the MSStore Python in the docs that I shouldn't investigate further for the time being.

Since you said that your use case works with the official python install from python.org, I recommend that you keep using that version and just manually update as needed.

If you do keep at it and find a solution or workaround, please share it here. We can then update the docs to help others going down the non-recommended route in the future.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.