Unable to match regex against output: ModuleNotFoundError: No module named 'codespell_lib'
Categories
(Developer Infrastructure :: Lint and Formatting, defect)
Tracking
(firefox-esr60 unaffected, firefox-esr68 unaffected, firefox67 unaffected, firefox68 unaffected, firefox69 unaffected, firefox70 unaffected, firefox71 fixed)
Tracking | Status | |
---|---|---|
firefox-esr60 | --- | unaffected |
firefox-esr68 | --- | unaffected |
firefox67 | --- | unaffected |
firefox68 | --- | unaffected |
firefox69 | --- | unaffected |
firefox70 | --- | unaffected |
firefox71 | --- | fixed |
People
(Reporter: Mardak, Assigned: ahal)
References
(Regression)
Details
(Keywords: regression)
Attachments
(1 file)
Split from bug 1585672 comment 4:
$ ./mach lint -l codespell browser/components/newtab/
Unable to match regex against output: Traceback (most recent call last):
Unable to match regex against output: File "/mozilla-central/obj/_virtualenvs/init_py3/bin/codespell", line 7, in <module>
Unable to match regex against output: from codespell_lib import _script_main
Unable to match regex against output: ModuleNotFoundError: No module named 'codespell_lib'
✖ 0 problems (0 errors, 0 warnings)
Assignee | ||
Comment 1•5 years ago
|
||
So this happens with or without the fix in bug 1585672? (I can't reproduce this one either)
Do you have a obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib
(note change 3.7 to your version of python)? Does clobbering the _virtualenvs
dir change anything?
Thanks for helping debug/fix these btw and sorry for the inconvenience.
Reporter | ||
Comment 2•5 years ago
|
||
This happens both with and without the fix in bug 1585672, and clobbering (including _virtualenvs
) doesn't help:
$ rm -rf obj && ./mach lint -l codespell browser/components/newtab
Using base prefix '/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7'
/mozilla-central/third_party/python/virtualenv/virtualenv.py:1047: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
New python executable in /mozilla-central/obj/_virtualenvs/init_py3/bin/python3.7
Also creating executable in /mozilla-central/obj/_virtualenvs/init_py3/bin/python
Installing setuptools, pip, wheel...done.
b'running build_ext\ncopying build/lib.macosx-10.14-x86_64-3.7/psutil/_psutil_osx.cpython-37m-darwin.so -> psutil\ncopying build/lib.macosx-10.14-x86_64-3.7/psutil/_psutil_posix.cpython-37m-darwin.so -> psutil\n'
Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
Unable to match regex against output: Traceback (most recent call last):
Unable to match regex against output: File "/mozilla-central/obj/_virtualenvs/init_py3/bin/codespell", line 7, in <module>
Unable to match regex against output: from codespell_lib import _script_main
Unable to match regex against output: ModuleNotFoundError: No module named 'codespell_lib'
✖ 0 problems (0 errors, 0 warnings)
codespell_lib does exist:
$ find obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/_codespell.py
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/tests
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/tests/test_dictionary.py
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/tests/test_basic.py
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/tests/__init__.py
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/tests/__pycache__
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/tests/__pycache__/test_dictionary.cpython-37.pyc
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/tests/__pycache__/test_basic.cpython-37.pyc
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/tests/__pycache__/__init__.cpython-37.pyc
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/__init__.py
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/__pycache__
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/__pycache__/_codespell.cpython-37.pyc
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/__pycache__/__init__.cpython-37.pyc
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/data
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/data/dictionary.txt
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/data/__init__.py
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/data/__pycache__
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/data/__pycache__/__init__.cpython-37.pyc
obj/_virtualenvs/init_py3/lib/python3.7/site-packages/codespell_lib/data/linux-kernel.exclude
Reporter | ||
Comment 3•5 years ago
|
||
Checking the sys.path, I do see mozilla.pth:
and python3:
entries but not objdir:
https://searchfox.org/mozilla-central/source/build/virtualenv_packages.txt
/mozilla-central/python/mach
/mozilla-central/python/mozboot
/mozilla-central/python/mozbuild
/mozilla-central/python/mozlint
/mozilla-central/python/mozrelease
/mozilla-central/python/mozterm
/mozilla-central/python/mozversioncontrol
/mozilla-central/python/l10n
/mozilla-central/third_party/python/atomicwrites
/mozilla-central/third_party/python/attrs/src
/mozilla-central/third_party/python/biplist
/mozilla-central/third_party/python/blessings
/mozilla-central/third_party/python/Click
/mozilla-central/third_party/python/compare-locales
/mozilla-central/third_party/python/configobj
/mozilla-central/third_party/python/cram
/mozilla-central/third_party/python/dlmanager
/mozilla-central/third_party/python/enum34
/mozilla-central/third_party/python/fluent
/mozilla-central/third_party/python/funcsigs
/mozilla-central/third_party/python/mohawk
/mozilla-central/third_party/python/more-itertools
/mozilla-central/third_party/python/mozilla-version
/mozilla-central/third_party/python/pathlib2
/mozilla-central/third_party/python/gyp/pylib
/mozilla-central/third_party/python/python-hglib
/mozilla-central/third_party/python/pluggy
/mozilla-central/third_party/python/jsmin
/mozilla-central/third_party/python/psutil
/mozilla-central/third_party/python/pylru
/mozilla-central/third_party/python/which
/mozilla-central/third_party/python/pystache
/mozilla-central/third_party/python/pyyaml/lib3/
/mozilla-central/third_party/python/requests
/mozilla-central/third_party/python/requests-unixsocket
/mozilla-central/third_party/python/slugid
/mozilla-central/third_party/python/taskcluster
/mozilla-central/third_party/python/taskcluster-urls
/mozilla-central/third_party/python/py
/mozilla-central/third_party/python/pytest/src
/mozilla-central/third_party/python/pytoml
/mozilla-central/third_party/python/redo
/mozilla-central/third_party/python/six
/mozilla-central/third_party/python/taskcluster-urls
/mozilla-central/third_party/python/voluptuous
/mozilla-central/third_party/python/json-e
/mozilla-central/build
/mozilla-central/build/pymake
/mozilla-central/config
/mozilla-central/config/mozunit
/mozilla-central/dom/bindings
/mozilla-central/dom/bindings/parser
/mozilla-central/layout/tools/reftest
/mozilla-central/other-licenses/ply/
/mozilla-central/taskcluster
/mozilla-central/testing
/mozilla-central/testing/firefox-ui/harness
/mozilla-central/testing/marionette/client
/mozilla-central/testing/marionette/harness
/mozilla-central/testing/marionette/harness/marionette_harness/runner/mixins/browsermob-proxy-py
/mozilla-central/testing/marionette/puppeteer/firefox
/mozilla-central/testing/raptor
/mozilla-central/testing/talos
/mozilla-central/testing/mozbase/manifestparser
/mozilla-central/testing/mozbase/mozcrash
/mozilla-central/testing/mozbase/mozdebug
/mozilla-central/testing/mozbase/mozdevice
/mozilla-central/testing/mozbase/mozfile
/mozilla-central/testing/mozbase/mozhttpd
/mozilla-central/testing/mozbase/mozinfo
/mozilla-central/testing/mozbase/mozinstall
/mozilla-central/testing/mozbase/mozleak
/mozilla-central/testing/mozbase/mozlog
/mozilla-central/testing/mozbase/moznetwork
/mozilla-central/testing/mozbase/mozpower
/mozilla-central/testing/mozbase/mozprocess
/mozilla-central/testing/mozbase/mozprofile
/mozilla-central/testing/mozbase/mozproxy
/mozilla-central/testing/mozbase/mozrunner
/mozilla-central/testing/mozbase/mozsystemmonitor
/mozilla-central/testing/mozbase/mozscreenshot
/mozilla-central/testing/mozbase/moztest
/mozilla-central/testing/mozbase/mozversion
/mozilla-central/tools
/mozilla-central/testing/web-platform
/mozilla-central/testing/web-platform/tests/tools/third_party/html5lib
/mozilla-central/testing/web-platform/tests/tools/third_party/webencodings
/mozilla-central/testing/web-platform/tests/tools/third_party/enum
/mozilla-central/testing/web-platform/tests/tools/third_party/h2
/mozilla-central/testing/web-platform/tests/tools/third_party/hpack
/mozilla-central/testing/web-platform/tests/tools/third_party/hyperframe
/mozilla-central/testing/web-platform/tests/tools/third_party/certifi
/mozilla-central/testing/web-platform/tests/tools/wptserve
/mozilla-central/testing/web-platform/tests/tools/wptrunner
/mozilla-central/testing/web-platform/tests/tools/six
/mozilla-central/testing/xpcshell
/mozilla-central/third_party/python/mock-1.0.0
/mozilla-central/xpcom/ds/tools
/mozilla-central/tools/docs
/mozilla-central/third_party/python/cbor2
/mozilla-central/third_party/python/pyasn1
/mozilla-central/third_party/python/pyasn1-modules
/mozilla-central/third_party/python/rsa
/mozilla-central/third_party/python/PyECC
/mozilla-central/toolkit/components/telemetry/tests/marionette/harness
/mozilla-central
/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python37.zip
/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7
/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload
/usr/local/lib/python3.7/site-packages
/usr/local/Cellar/protobuf/3.9.2/libexec/lib/python3.7/site-packages
/mozilla-central/testing/awsy
Reporter | ||
Comment 4•5 years ago
•
|
||
Looks like sys.exec_prefix
when running ./mach lint
is /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7
and sys.executable: /usr/local/opt/python/bin/python3.7
But if I manually launch ./obj/_virtualenvs/init_py3/bin/python3
sys.exec_prefix: /mozilla-central/./obj/_virtualenvs/init_py3/bin/..
sys.executable: /mozilla-central/obj/_virtualenvs/init_py3/bin/python3
sys.path includes /mozilla-central/obj/_virtualenvs/init_py3/lib/python3.7/site-packages'
Comment 5•5 years ago
|
||
Assignee | ||
Comment 7•5 years ago
|
||
Fwiw it's expected that sys.executable
and sys.exec_prefix
won't point to the virtualenv. The reason for this is that we activate the virtualenv in-process using a special script that comes bundled with virtualenv
. For whatever reason, this script doesn't mess with sys.executable
(I imagine because technically the system Python was still used to start the active interpreter).
The fact that your sys.path
includes that site-packages
dir but still isn't found is really strange. I have a hunch that this issue is related to bug 1585672 and the global ImportHook. Try disabling it again (by commenting it out at the bottom of mach_bootstrap.py
) and see if that fixes this issue.
I don't know why you were seeing modules with a __file__
value of None
, but I think that might be the root of all these problems.
Assignee | ||
Comment 8•5 years ago
|
||
Oh re-reading your comment, it sounds like the objdir site-packages
is not in your sys.path. That makes more sense and should be possible to debug.
The activate_this.py script is supposed to prepend the site-packages dir to sys.path
. You can put some print debugging into it, it'll be at <virtualenv>/bin/activate_this.py
. The exact place it should get invoked in |mach lint| is here:
https://searchfox.org/mozilla-central/source/tools/lint/mach_commands.py#63
Assignee | ||
Updated•5 years ago
|
Reporter | ||
Comment 9•5 years ago
|
||
activate_this
is updating sys.path
, but looks like the way rstcheck is invoked, it loses the path:
From rst/__init__.py
, sys.path
has 114 entries including /mozilla-central/obj/_virtualenvs/init_py3/lib/python3.7/site-package
when it's calling subprocess.Popen
with cmdargs == ['/mozilla-central/obj/_virtualenvs/init_py3/bin/rstcheck', '/mozilla-central/browser/components/newtab/docs/index.rst']
Modifying /mozilla-central/obj/_virtualenvs/init_py3/bin/rstcheck
to print out sys.path
has only these 6 entries:
/mozilla-central/obj/_virtualenvs/init_py3/bin
/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python37.zip
/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7
/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload
/usr/local/lib/python3.7/site-packages
/usr/local/Cellar/protobuf/3.9.2/libexec/lib/python3.7/site-packages
Similarly, modifying /mozilla-central/obj/_virtualenvs/init_py3/bin/codespell
to print sys.path
has the same 6 entries as above. Although codespell uses ProcessHandlerMixin.run
instead:
Reporter | ||
Comment 10•5 years ago
|
||
Just to confirm, before switching to python 3, modifying /mozilla-central/obj/_virtualenvs/init/bin/rstcheck
to print sys.path
includes:
/mozilla-central/obj/_virtualenvs/init/lib/python2.7/site-packages
Assignee | ||
Comment 11•5 years ago
|
||
But why is it working for me and Shivam (and presumably everyone else who has run this in the last couple days but hasn't complained)? And why did it work before on Python 2 but start failing with Python 3? So many questions..
But this should be solvable without understanding the issue I guess. Either by calling those binaries in-process or passing sys.path
into the PYTHONPATH
environment variable.
Reporter | ||
Comment 12•5 years ago
|
||
Aha! I think I figured it out. ??
$ head -1 obj/_virtualenvs/init/bin/rstcheck
#!/mozilla-central/obj/_virtualenvs/init/bin/python2.7
$ head -1 obj/_virtualenvs/init_py3/bin/rstcheck
#!/usr/local/bin/python3
If I modify the python 3 version to use #!/mozilla-central/obj/_virtualenvs/init_py3/bin/python3
instead, it runs correctly with the expected sys.path
!
Assignee | ||
Comment 13•5 years ago
|
||
Aha! Yeah, that would do it. My init_py3/bin/rstcheck
binary points to the virtuanlenv Python.
I'm guessing that shebang gets written at install time. Rstcheck is ultimately installed here:
https://searchfox.org/mozilla-central/source/python/mozlint/mozlint/util/pip.py#13
So maybe that command is picking up your system pip
instead of the one in init_py3/bin/pip
. Can you confirm that init_py3/bin/pip
exists on your system? Maybe we just need to call it pip3
.
Assignee | ||
Comment 14•5 years ago
|
||
Hm, but rstcheck
did get installed there.. so that doesn't make a ton of sense.
Reporter | ||
Comment 15•5 years ago
•
|
||
$ ls -1 obj/_virtualenvs/init*/bin/pip*
obj/_virtualenvs/init/bin/pip
obj/_virtualenvs/init/bin/pip2
obj/_virtualenvs/init/bin/pip2.7
obj/_virtualenvs/init_py3/bin/pip
obj/_virtualenvs/init_py3/bin/pip3
obj/_virtualenvs/init_py3/bin/pip3.7
If I do subprocess.run(['/usr/bin/which', 'pip'])
from mozlint/util/pip.py, it prints out:
/mozilla-central/obj/_virtualenvs/init_py3/bin/pip
Assignee | ||
Comment 16•5 years ago
•
|
||
Ok, I'm overthinking this. We can change the ["rstcheck"]
command to [python, "rstcheck"]
(making sure to find the correct virtualenv python). Probably a similar fix needed for codespell. I'll try to get to this sometime tomorrow (unless you beat me to it).
Reporter | ||
Comment 17•5 years ago
|
||
Use distutils.spawn.find_executable to get the correct virtualenv python3.
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Comment 18•5 years ago
|
||
Pushed by elee@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/a96731900f47 [lint] Make sure codespell, rst and yaml run with virtualenv python r=ahal
Comment 19•5 years ago
|
||
bugherder |
Updated•5 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Description
•