1623741 - Pre-commit linting hook ImportError: cannot import name which

Status: RESOLVED FIXED

(Developer Infrastructure :: Lint and Formatting, defect)

Description

[Ed Lee :Mardak]

Probably similar to bug 1590328 likely regressed by bug 1622789.

Traceback (most recent call last):
  File ".git/hooks/pre-commit", line 17, in <module>
    from mozversioncontrol import get_repository_object, InvalidRepoPath
  File "mozilla-central/tools/lint/../../python/mozversioncontrol/mozversioncontrol/__init__.py", line 16, in <module>
    from mozfile import which
ImportError: cannot import name which

If I add mozbase/mozfile…

diff --git a/tools/lint/hooks_js_format.py b/tools/lint/hooks_js_format.py
--- a/tools/lint/hooks_js_format.py
+++ b/tools/lint/hooks_js_format.py
@@ -14 +14 @@ topsrcdir = os.path.join(here, os.pardir, os.pardir)
-EXTRA_PATHS = ("python/mozversioncontrol", "python/mozbuild",)
+EXTRA_PATHS = ("python/mozversioncontrol", "python/mozbuild", "testing/mozbase/mozfile",)

… it then complains:

  File "mozilla-central/tools/lint/../../python/mozversioncontrol/mozversioncontrol/__init__.py", line 58, in get_tool_path
    path = which(tool)
  File "mozilla-central/tools/lint/../../testing/mozbase/mozfile/mozfile/mozfile.py", line 338, in which
    from shutil_which import which as shutil_which
ImportError: No module named shutil_which

Should the EXTRA_PATHS be pointing to some other mozfile?

Comment 1

[Geoff Brown [:gbrown]]

I agree this was regressed by bug 1622789. Sorry about that.

testing/mozbase/mozfile is the correct mozfile.

I'm not that surprised to see a failure on shutil_which, but mozfile only attempts to import shutil_which if the import of which from shutil failed:

https://searchfox.org/mozilla-central/rev/6cd54550a27e2f6ca0755a25328f769e41e524f4/testing/mozbase/mozfile/mozfile/mozfile.py#336

I am surprised you are getting an import error on "from shutil import which as shutil_which".

What platform are you having trouble on? What version of python?

Comment 2

[Geoff Brown [:gbrown]]

Actually, maybe I shouldn't be surprised:

python
Python 2.7.17 (default, Nov  7 2019, 10:07:09) 
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from shutil import which
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name which
>>> 

(it works for me on python3, but not python 2.7)

Comment 3

[Geoff Brown [:gbrown]]

Do we also need third_party/python/backports/shutil_which in EXTRA_PATHS?

Comment 4

[Geoff Brown [:gbrown]]

:ahal - I think you know more about this area than I do.

Comment 5

[Andrew Halberstadt [:ahal]]

Yes, comment 3 looks like it would fix it. A better (but slightly more involved) solution, would be to run that script with Python 3 (by modifying the shebang).

Edit: Though you'd still need mozfile in EXTRA_PATHS I suppose.

Comment 6

[Andrew Halberstadt [:ahal]]

Just for added clarity, shutil.which was added to the stdlib in some Python 3 version, and shutil_which is the backport of this implementation for versions that are missing it. If you use mozfile.which within a mach context, it'll paper over this import. Though in this case we aren't in a mach context and therefore don't have mozfile/shutil_which on the sys.path.

Comment 7

[Ed Lee :Mardak]

Attachment: Bug 1623741 - [mozlint] Use python3 and include mozfile now that mozversioncontrol uses it r?gbrown

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2135737e9432
[mozlint] Use python3 and include mozfile now that mozversioncontrol uses it r=gbrown

Comment 9

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/2135737e9432