1698208 - Newtab linting scripts that rely on top level mach lint aren't working in a Windows dev environment

Status: NEW

(Firefox :: Messaging System, defect, P3)

Description

[Emily McMinn :emcminn]

While investigating bug 1694772 I came across an issue where any of the newtab yamscripts that start with cd $npm_package_config... fail to run with the error message "The system cannot find the path specified". (this includes mochitest and lint scripts.)

As well, running ./mach lint directly fails with a couple errors,
'The system cannot find the file specified`,
'AttributeError: "Process" object has no attribute "_job"'

...though this could be related to the first problem, if ./mach lint isn't running from the correct location.

I'm mostly wondering if anyone can duplicate this, or if I can put it down to an issue with my local build.

Comment 1

[Emily McMinn :emcminn]

We addressed in triage that it might be a good solution to optimize for Linux tools within Windows; to help reduce path errors and things like that.

Comment 2

[OOO until Jun 29 - Dan Mosedale (:dmosedale, :dmose)]

Interesting. It might be related to https://phabricator.services.mozilla.com/D117612 but I suspect not.

Can you execute one of these scripts like this:

`npm run --timing some-example-script-that-shows-this-behavior`

--timing should cause it create a debug log, and tell you at the end of the output where it is.

Attaching that log here could help me figure out what's going on...

Comment 3

[Emily McMinn :emcminn]

Sure, no problem! Unfortunately it's not a ton more informational: (This was ./mach npm run --timing mochitest --prefix=browser/components/newtab)

0 info it worked if it ends with ok
1 verbose cli [ 'c:\\Users\\emcminn\\.mozbuild\\node\\node.exe',
1 verbose cli   'c:\\Users\\emcminn\\.mozbuild\\node\\node_modules\\npm\\bin\\npm-cli.js',
1 verbose cli   '--scripts-prepend-node-path=auto',
1 verbose cli   'run',
1 verbose cli   '--timing',
1 verbose cli   'mochitest',
1 verbose cli   '--prefix=browser/components/newtab' ]
2 info using npm@6.14.10
3 info using node@v10.23.1
4 verbose run-script [ 'premochitest', 'mochitest', 'postmochitest' ]
5 info lifecycle activity-streams@1.14.3~premochitest: activity-streams@1.14.3
6 info lifecycle activity-streams@1.14.3~mochitest: activity-streams@1.14.3
7 verbose lifecycle activity-streams@1.14.3~mochitest: unsafe-perm in lifecycle true
8 verbose lifecycle activity-streams@1.14.3~mochitest: PATH: c:\Users\emcminn\.mozbuild\node\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;c:\mozilla-source\mozilla-unified\browser\components\newtab\node_modules\.bin;c:\Users\emcminn\.mozbuild\node;c:\Users\emcminn\.mozbuild\node:C:\mozilla-build\msys\local\bin;c:\mozilla-build\bin;c:\mozilla-build\kdiff3;c:\mozilla-build\nsis-3.01;c:\mozilla-build\python;c:\mozilla-build\python\Scripts;c:\mozilla-build\python3;c:\mozilla-build\python3\Scripts;.;C:\mozilla-build\msys\local\bin;C:\mozilla-build\msys\mingw\bin;C:\mozilla-build\msys\bin;c:\Windows\System32;c:\Windows;c:\Windows\System32\Wbem;c:\Program Files\Git\cmd\;c:\git-cinnabar;c:\php;c:\Users\emcminn\phabricator\arcanist\bin;c:\Program\ Files\Git\bin;c:\mozilla-build\Python3\Scripts;C:\mozilla-build\msys\.mozbuild\node
9 verbose lifecycle activity-streams@1.14.3~mochitest: CWD: c:\mozilla-source\mozilla-unified\browser\components\newtab
10 silly lifecycle activity-streams@1.14.3~mochitest: Args: [ '/d /s /c',
10 silly lifecycle   '(cd $npm_package_config_mc_dir && ./mach mochitest browser/components/newtab/test/browser --headless)' ]
11 silly lifecycle activity-streams@1.14.3~mochitest: Returned: code: 1  signal: null
12 info lifecycle activity-streams@1.14.3~mochitest: Failed to exec mochitest script
13 verbose stack Error: activity-streams@1.14.3 mochitest: `(cd $npm_package_config_mc_dir && ./mach mochitest browser/components/newtab/test/browser --headless)`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (c:\Users\emcminn\.mozbuild\node\node_modules\npm\node_modules\npm-lifecycle\index.js:332:16)
13 verbose stack     at EventEmitter.emit (events.js:198:13)
13 verbose stack     at ChildProcess.<anonymous> (c:\Users\emcminn\.mozbuild\node\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:198:13)
13 verbose stack     at maybeClose (internal/child_process.js:982:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
14 verbose pkgid activity-streams@1.14.3
15 verbose cwd c:\mozilla-source\mozilla-unified
16 verbose Windows_NT 10.0.19041
17 verbose argv "c:\\Users\\emcminn\\.mozbuild\\node\\node.exe" "c:\\Users\\emcminn\\.mozbuild\\node\\node_modules\\npm\\bin\\npm-cli.js" "--scripts-prepend-node-path=auto" "run" "--timing" "mochitest" "--prefix=browser/components/newtab"
18 verbose node v10.23.1
19 verbose npm  v6.14.10
20 error code ELIFECYCLE
21 error errno 1
22 error activity-streams@1.14.3 mochitest: `(cd $npm_package_config_mc_dir && ./mach mochitest browser/components/newtab/test/browser --headless)`
22 error Exit status 1
23 error Failed at the activity-streams@1.14.3 mochitest script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

Comment 4

[OOO until Jun 29 - Dan Mosedale (:dmosedale, :dmose)]

One problem is that in yamscripts.yaml (and therefore package.json, since that part of it is generated), the mochitest targets use $npm_package_config_mc_dir where they should be fixing using $npm_package_config_mc_root.

Fixing that turns the mochitest problem into the same thing as the lint problem.

What might be going on there is that the variables aren't being interpolated at all. I tried various things from https://stackoverflow.com/questions/43705195/how-can-i-use-variables-in-package-json without success.

That said, there is an error message on the console that says '.' is not recognized as an internal or external command, operable program, or batch file. The . in question might be the one at the beginning of ./mach, maybe.

Next things to try include:

• see if anything here https://www.minitool.com/news/fix-not-recognized-as-an-internal-external-command.html helps
• figure out what shell those scripts (if any?) those scripts are being executed with...
• it seems plausible that the fact we're running under msys has something to do with what's going on. dig into that...

Comment 5

[OOO until Jun 29 - Dan Mosedale (:dmosedale, :dmose)]

The all-platform mochitest bug that is part of this has been spun out to bug 1721649.

Comment 6

[Ed Lee :Mardak]

Bug 1737419 added some docs to workaround with direct invocation of lint:sasslint and testmc:unit. This bug should fix the path/invocation differences and remove the docs note (or update it for people to know they can just run individual tests).

Comment 7

[OOO until Jun 29 - Dan Mosedale (:dmosedale, :dmose)]

It's possible that some of the stuff in https://davidwells.io/blog/advanced-package-json-tips could help use work this out. Also potentially relevant is the cross-env-shell piece of the cross-env package.

Comment 8

[OOO until Jun 29 - Dan Mosedale (:dmosedale, :dmose)]

Relevent info: one of the problems in play here is that the $npm_blah_blah syntax doesn't work, but if one changes it to %npm_blah_blah% things get (at least somewhat) better.

Comment 9

[Beth Rennie [:beth] (she/her)]

Instead of embedding shell in the package.json scripts, can we replace them with a node script?

Comment 10

[OOO until Jun 29 - Dan Mosedale (:dmosedale, :dmose)]

It does sound cleaner, for sure. I guess we have to weigh adding more dev-dependencies against adding another layer of indirection in order to see what's actually going on.

Another workaround that can help is doing something like this:

../../../mach npm config set script-shell="C:\\mozilla-build\\msys2\\usr\\bin\\bash.exe"

It doesn't fix all the problems, but it does make

../../../mach npm run lint:eslint

work for me.

Comment 11

[Beth Rennie [:beth] (she/her)]

I don't think we would need to add more dev-dependencies. We could just write a JS script that invokes process.spawn

[edit] this didn't submit my entire comment

Comment 12

[Beth Rennie [:beth] (she/her)]

So I ran into some issues implementing this due to the fact that I'm using MozillaBuild via powershell.

Invoking mach from powershell resolves to mach.ps1 which invokes mach.py via py or python. This ends up executing C:\Python39\python.exe.

Then if we invoke msys bash to call mach lint -l eslint, it resolves mach to mach.py, which has a shebang of #!/usr/bin/env python3. MozillaBuild copies over python.exe to python3.exe, although regular Python does not ship with it.

When we invoke mach.py via python3.exe, we end up here because the expected python is python3.exe and the metadata on disk has python.exe. This triggers a rebuild of the virtualenv, but because the virtualenv is already in use, the script blows up:

(mozconfig:artifact)Barret@FOXYSHAZAM ~\W\s\h\m\b\c\newtab [127] $ ..\..\..\mach npm run lint:eslint

> activity-streams@1.14.3 lint:eslint
> (cd $npm_package_config_mc_root && ./mach lint -l eslint $npm_package_config_newtab_path)

Traceback (most recent call last):
  File "C:\Users\Barret\Workspace\src\hg.mozilla.org\mozilla-central\mach", line 96, in <module>
    main(sys.argv[1:])
  File "C:\Users\Barret\Workspace\src\hg.mozilla.org\mozilla-central\mach", line 88, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
  File "C:\Users\Barret\Workspace\src\hg.mozilla.org\mozilla-central\mach", line 35, in check_and_get_mach
    return load_mach(dir_path, mach_path)
  File "C:\Users\Barret\Workspace\src\hg.mozilla.org\mozilla-central\mach", line 21, in load_mach
    return mach_initialize.initialize(dir_path)
  File "C:\Users\Barret\Workspace\src\hg.mozilla.org\mozilla-central\build/mach_initialize.py", line 144, in initialize
    _activate_python_environment(
  File "C:\Users\Barret\Workspace\src\hg.mozilla.org\mozilla-central\build/mach_initialize.py", line 99, in _activate_python_environment
    mach_environment.activate()
  File "C:\Users\Barret\Workspace\src\hg.mozilla.org\mozilla-central\python\mach\mach\site.py", line 361, in activate
    self.ensure()
  File "C:\Users\Barret\Workspace\src\hg.mozilla.org\mozilla-central\python\mach\mach\site.py", line 348, in ensure
    self._build()
  File "C:\Users\Barret\Workspace\src\hg.mozilla.org\mozilla-central\python\mach\mach\site.py", line 386, in _build
    _create_venv_with_pthfile(
  File "C:\Users\Barret\Workspace\src\hg.mozilla.org\mozilla-central\python\mach\mach\site.py", line 1172, in _create_venv_with_pthfile
    shutil.rmtree(virtualenv_root)
  File "C:\Python39\lib\shutil.py", line 749, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Python39\lib\shutil.py", line 622, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Python39\lib\shutil.py", line 622, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Python39\lib\shutil.py", line 622, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Python39\lib\shutil.py", line 627, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Python39\lib\shutil.py", line 625, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Barret\\.mozbuild\\srcdirs\\mozilla-central-e60d5d44823e\\_virtualenvs\\mach\\Lib\\site-packages\\glean\\glean_ffi.dll'
(mozconfig:artifact)Barret@FOXYSHAZAM ~\W\s\h\m\b\c\newtab [1] $ Traceback (most recent call last):
  File "C:\Users\Barret\.mozbuild\srcdirs\mozilla-central-e60d5d44823e\_virtualenvs\mach\Lib\site-packages\glean\_subprocess\_process_dispatcher_helper.py", line 35, in <module>
    simple_log_level, func, args = pickle.loads(base64.b64decode(sys.argv[1]))
  File "C:\Users\Barret\.mozbuild\srcdirs\mozilla-central-e60d5d44823e\_virtualenvs\mach\Lib\site-packages\glean\__init__.py", line 17, in <module>
    from .glean import Glean
ModuleNotFoundError: No module named 'glean.glean'

It's possible using a symlinked python3.exe -> python.exe would resolve this or we would have to update MozSiteMetadata.__eq__ to not do a strict path comparison, but do something smarter to recognize these are the same python.