1562920 - ImportError: cannot import name ConverterMapping on running linter

Status: RESOLVED WORKSFORME

(Developer Infrastructure :: Lint and Formatting, defect, P3)

Description

[Andreas Tolfsen ❲:ato❳]

Running the linter locally on any directory causes this traceback
to be printed before the lint results:

% ./mach lint -funix remote
pulling from https://hg.mozilla.org/l10n/gecko-strings
searching for changes
adding changesets
adding manifests
adding file changes
added 22 changesets with 44 changes to 36 files
new changesets 64b4e680cbb2:d0131144d6c4
(run 'hg update' to get a working copy)
36 files updated, 0 files merged, 10 files removed, 0 files unresolved
Traceback (most recent call last):
  File "/home/ato/src/gecko/python/mozlint/mozlint/roller.py", line 39, in _run_worker
    res = func(paths, config, **lintargs) or []
  File "/home/ato/src/gecko/python/mozlint/mozlint/types.py", line 50, in __call__
    return self._lint(paths, config, **lintargs)
  File "/home/ato/src/gecko/python/mozlint/mozlint/types.py", line 134, in _lint
    return func(files, config, **lintargs)
  File "/home/ato/src/gecko/tools/lint/python/flake8.py", line 80, in lint
    from flake8.main.application import Application
  File "/home/ato/src/gecko/build/mach_bootstrap.py", line 403, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/src/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/lib/python2.7/site-packages/flake8/main/application.py", line 15, in <module>
    from flake8.main import options
  File "/home/ato/src/gecko/build/mach_bootstrap.py", line 403, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/src/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/lib/python2.7/site-packages/flake8/main/options.py", line 4, in <module>
    from flake8.main import vcs
  File "/home/ato/src/gecko/build/mach_bootstrap.py", line 403, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/src/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/lib/python2.7/site-packages/flake8/main/vcs.py", line 4, in <module>
    from flake8.main import mercurial
  File "/home/ato/src/gecko/build/mach_bootstrap.py", line 403, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/src/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/lib/python2.7/site-packages/flake8/main/mercurial.py", line 7, in <module>
    import configparser
  File "/home/ato/src/gecko/build/mach_bootstrap.py", line 403, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/src/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/lib/python2.7/site-packages/configparser.py", line 12, in <module>
    from backports.configparser import (
ImportError: cannot import name ConverterMapping
remote/domains/content/Runtime.jsm:236:10: no-undef error: 'rv' is not defined.

The hg pull only happened the first time, and subsequently disappeared.

Comment 1

[Andrew Halberstadt [:ahal]]

Chris, I think you ran into this same issue. Did you ever end up figuring it out (or should this be duped)?

Comment 2

[Andrew Halberstadt [:ahal]]

Andreas, I pointed Chris at https://github.com/spack/spack/issues/8343 earlier. While not identical, one possibility is that you might have a module called backports on your PYTHONPATH somewhere. So flake8 might be finding that before the one that is installed as part of the virtualenv.

I'm not sure if this tip helped Chris or not, but maybe it's true in your case.

Comment 3

[Andrew Halberstadt [:ahal]]

I guess another sanity check would be to inspect the virtualenv. I.e, run ./mach lint -l flake8, then look at:

<objdir>/_virtualenvs/init/lib/python2.7/site-packages/backports/configparser/__init__.py

for ConverterMapping.

Comment 4

[Andreas Tolfsen ❲:ato❳]

Thanks for the pointers, ahal!

It looks like uninstalling the Debian package python-configparser
makes the error go away, which as you hinted, must mean that there
is some confusion with the PYTHONPATH. Perhaps the virtualenv-vendored
backports.configparser is accidentally loading an old version?

Comment 5

[Chris Manchester (limited bugmail, email directly)]

I ended up fixing these without being able to run things locally. I did notice that PYTHONPATH was empty in my environment.

Comment 6

[Andrew Halberstadt [:ahal]]

Thanks for figuring it out Andreas!

It is indeed using an older version of configparser, but not by accident:
https://searchfox.org/mozilla-central/source/tools/lint/python/flake8_requirements.txt#18

We do this to make sure everyone is using the same versions and we don't have some people seeing lint issues while others don't. I'm kind of surprised that the virtualenv allows system packages.. maybe we should make sure it doesn't (though that seems like it'll have far reaching implications).

I'm not sure how to solve this problem generally, but in this particular case we could try upgrading flake8 (or maybe just configparser if flake8 doesn't pin it) so that the python-configparser package is compatible.

Comment 7

[Andrew Halberstadt [:ahal]]

Actually, the latest version of configparser does have ConverterMapping. So it's the system package that's out of date.

Comment 9

[Mark Banner (:standard8)]

Given we no longer using flake8, I'm going to mark this as WFM.