Closed Opened 1 year ago Closed 10 months ago

# linter broken - "ImportError: cannot import name which" in python/mozbuild/mozbuild/nodeutil.py (sometimes after running mach mozregression)

Not set
normal

## Tracking

### (Not tracked)

RESOLVED DUPLICATE of bug 1582248

## Attachments

### (1 file)

 8.45 KB, text/plain Details
$hg amend Traceback (most recent call last): File "path\to\mozilla-central\python/mozlint\mozlint\roller.py", line 39, in _run_worker res = func(paths, config, **lintargs) or [] File "path\to\mozilla-central\python/mozlint\mozlint\types.py", line 50, in __call__ return self._lint(paths, config, **lintargs) File "path\to\mozilla-central\python/mozlint\mozlint\types.py", line 133, in _lint func = findobject(config['payload']) File "path\to\mozilla-central\python/mozlint\mozlint\pathutils.py", line 216, in findobject obj = __import__(modulepath) File "path\to\mozilla-central\tools\lint\eslint\__init__.py", line 16, in <module> from eslint import setup_helper File "path\to\mozilla-central\tools\lint\eslint\setup_helper.py", line 20, in <module> from mozbuild.nodeutil import (find_node_executable, find_npm_executable, File "path\to\mozilla-central\python/mozbuild\mozbuild\nodeutil.py", line 15, in <module> from mozfile import which ImportError: cannot import name which A failure occurred in the eslint linter.  When the commit hook is configured through: [hooks] pretxncommit.lint = python:/mozilla-central/tools/lint/hooks.py:hg  on a commit based on https://hg.mozilla.org/mozilla-central/rev/db8f3ee41bdf . I tried running ./mach bootstrap to no avail. ./mach lint --outgoing produces the same error. Type: task → defect Clearing the .pyc files inside testing/mozbase/mozfile/mozfile/ doesn't help, and --verbose doesn't provide more output. In case the mixed slashes in the path didn't make it obvious, this is on Windows. Thanks for filing, would you mind applying this patch letting me know what the printed path is? diff --git a/python/mozbuild/mozbuild/nodeutil.py b/python/mozbuild/mozbuild/nodeutil.py --- a/python/mozbuild/mozbuild/nodeutil.py +++ b/python/mozbuild/mozbuild/nodeutil.py @@ -7,16 +7,18 @@ from __future__ import absolute_import, import os import subprocess import platform from distutils.version import ( StrictVersion, ) from mozboot.util import get_state_dir +import mozfile +print(mozfile.__file__) from mozfile import which NODE_MIN_VERSION = StrictVersion("8.11.0") NPM_MIN_VERSION = StrictVersion("5.6.0") def find_node_paths(): """ Determines the possible paths for node executables.  (I think clobbering <objdir>/_virtualenvs might fix it) (In reply to Andrew Halberstadt [:ahal] from comment #2) Thanks for filing, would you mind applying this patch letting me know what the printed path is? It prints twice: path\to\mozilla-central\testing/mozbase/mozfile\mozfile\__init__.pyc path\to\opt-objdir\_virtualenvs\init\Lib\site-packages\mozfile\__init__.pyc  So looks like you're probably right? At what point should such a clobber normally happen (that is, I assume that something normally takes care of this automatically?) Flags: needinfo?(ahal) I can confirm that moving the $OBJDIR/_virtualenvs file to an old_* dir fixes the issue...

I don't think there's any automatic clobbering mechanism. I'm not an expert on our virtualenv creation but I did a bit of digging and the reason this type of error doesn't happen all the time is:

1. Only mozbase modules are placed in the init virtualenv, all other modules (e.g in third_party/python) are appended to the PYTHONPATH at mach runtime. I'm not sure why this is, probably something to do with mach or the virtualenv logic depending on mozbase.

2. On non-Windows, these mozbase modules link back to the mozbase source.

Together that means this type of issue should only happen on Windows and with changes to mozbase, which changes pretty infrequently these days. Maybe one easy thing to do would be to change mach clobber python to also remove the _virtualenvs folder (in addition to removing all .pyc files). That way at least we can tell people to run a simple one-liner if they run into this.

Flags: needinfo?(ahal)

I don't think there's any automatic clobbering mechanism. I'm not an expert on our virtualenv creation but I did a bit of digging and the reason this type of error doesn't happen all the time is:

1. Only mozbase modules are placed in the init virtualenv, all other modules (e.g in third_party/python) are appended to the PYTHONPATH at mach runtime. I'm not sure why this is, probably something to do with mach or the virtualenv logic depending on mozbase.

2. On non-Windows, these mozbase modules link back to the mozbase source.

Together that means this type of issue should only happen on Windows and with changes to mozbase, which changes pretty infrequently these days. Maybe one easy thing to do would be to change mach clobber python to also remove the _virtualenvs folder (in addition to removing all .pyc files). That way at least we can tell people to run a simple one-liner if they run into this.

FWIW, I just saw the same thing on mac, but I suspect that's due to bug 1549186...

I also ran into this on Windows today, removing \$OBJDIR/_virtualenvs did the trick.

Attached file mozregression log

I seem to have found another way to reproduce that is... unfortunate:

1. set up mozconfig with clean objdir, build
2. run ./mach mozregression --help, which will set up mozregression and installs a bunch of stuff in the virtualenv
3. touch some cpp file and run ./mach build binaries

Fortunately, deleting the virtualenv fixes this and "only" requires another configure step plus a minor amount of additional build work, rather than a full-on clobber. But it's unfortunate.

Can we make ./mach mozregression not break stuff like this?

Flags: needinfo?(ahal)

I've actually never touched mozregression before, but yeah.. we could probably give it its own dedicated virtualenv and it could use a Pipfile to specify whatever packages it needs. Similar to this:
https://searchfox.org/mozilla-central/source/tools/docs/mach_commands.py#65

When I ran ./mach mozregression locally it didn't seem to install anything (even after clobbering).. but maybe that's because I'm on Linux? Might be good to file a bug under Testing :: mozregression and cc :wlach (I'm not sure if he's still the owner or not, but he can at least direct you to whoever is).

Flags: needinfo?(ahal)
Summary: linter broken - "ImportError: cannot import name which" in python/mozbuild/mozbuild/nodeutil.py → linter broken - "ImportError: cannot import name which" in python/mozbuild/mozbuild/nodeutil.py (sometimes after running mach mozregression)
Depends on: 1580625

OK, filed bug 1580625 for the mozregression thingy.

I am pretty sure bug 1582248 should workaround this particular issue, though it's admittedly only a short term solution. If it works as I hope, maybe we can mark this as a dupe? Bug 1580625 remains open for a more permanent fix.

I guess we could have just continued using this bug rather than filing bug 1580625.. but oh well. Since most of the mozregression talk is over there let's use that to solve properly.

Status: NEW → RESOLVED
Closed: 10 months ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1582248
You need to log in before you can comment on or make changes to this bug.