1761507 - `./mach lint` is still broken on Python 3.10

Status: RESOLVED FIXED

(Developer Infrastructure :: Lint and Formatting, defect)

Description

[Kagami Rosylight [:saschanaz] (they/them)]

Still about typed-ast.

> ./mach lint --fix docs/code-quality/lint/linters/eslint-plugin-mozilla/use-isInstance.rst
created virtual environment CPython3.10.1.final.0-64 in 34ms
  creator CPython3Windows(dest=C:\Users\sasch\.mozbuild\srcdirs\gecko-dev-f1e71310be22\_virtualenvs\mach, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator
Collecting glean-sdk==44.0.0
  Using cached glean_sdk-44.0.0-py3-none-win_amd64.whl (1.4 MB)
Collecting cffi>=1.13.0
  Using cached cffi-1.15.0-cp310-cp310-win_amd64.whl (180 kB)
Collecting glean-parser==5.0.1
  Using cached glean_parser-5.0.1-py3-none-any.whl (88 kB)
Requirement already satisfied: jsonschema>=3.0.2 in c:\users\sasch\documents\github\gecko-dev\third_party\python\jsonschema (from glean-parser==5.0.1->glean-sdk==44.0.0) (3.2.0)
Requirement already satisfied: PyYAML>=5.3.1 in c:\users\sasch\documents\github\gecko-dev\third_party\python\pyyaml\lib3 (from glean-parser==5.0.1->glean-sdk==44.0.0) (5.4.1)
Requirement already satisfied: Jinja2>=2.10.1 in c:\users\sasch\documents\github\gecko-dev\third_party\python\jinja2 (from glean-parser==5.0.1->glean-sdk==44.0.0) (2.11.3)
Requirement already satisfied: diskcache>=4 in c:\users\sasch\documents\github\gecko-dev\third_party\python\diskcache (from glean-parser==5.0.1->glean-sdk==44.0.0) (4.1.0)
Requirement already satisfied: Click>=7 in c:\users\sasch\documents\github\gecko-dev\third_party\python\click (from glean-parser==5.0.1->glean-sdk==44.0.0) (7.1.2)
Requirement already satisfied: appdirs>=1.4 in c:\users\sasch\documents\github\gecko-dev\third_party\python\appdirs (from glean-parser==5.0.1->glean-sdk==44.0.0) (1.4.4)
Requirement already satisfied: yamllint>=1.18.0 in c:\users\sasch\documents\github\gecko-dev\third_party\python\yamllint (from glean-parser==5.0.1->glean-sdk==44.0.0) (1.23.0)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Requirement already satisfied: MarkupSafe>=0.23 in c:\users\sasch\documents\github\gecko-dev\third_party\python\markupsafe\src (from Jinja2>=2.10.1->glean-parser==5.0.1->glean-sdk==44.0.0) (1.1.1)
Requirement already satisfied: pyrsistent>=0.14.0 in c:\users\sasch\documents\github\gecko-dev\third_party\python\pyrsistent (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (0.16.0)
Requirement already satisfied: attrs>=17.4.0 in c:\users\sasch\documents\github\gecko-dev\third_party\python\attrs (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (19.2.0)
Requirement already satisfied: six>=1.11.0 in c:\users\sasch\documents\github\gecko-dev\third_party\python\six (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (1.13.0)
Requirement already satisfied: setuptools in c:\users\sasch\documents\github\gecko-dev\third_party\python\setuptools (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (51.2.0)
Requirement already satisfied: pathspec>=0.5.3 in c:\users\sasch\documents\github\gecko-dev\third_party\python\pathspec (from yamllint>=1.18.0->glean-parser==5.0.1->glean-sdk==44.0.0) (0.9.0)
Installing collected packages: pycparser, glean-parser, cffi, glean-sdk
Successfully installed cffi-1.15.0 glean-parser-5.0.1 glean-sdk-44.0.0 pycparser-2.21
Collecting psutil<=5.8.0,>=5.4.2
  Using cached psutil-5.8.0-cp310-cp310-win_amd64.whl
Installing collected packages: psutil
Successfully installed psutil-5.8.0
Collecting zstandard<=0.17.0,>=0.11.1
  Using cached zstandard-0.17.0-cp310-cp310-win_amd64.whl (611 kB)
Installing collected packages: zstandard
Successfully installed zstandard-0.17.0
created virtual environment CPython3.10.1.final.0-64 in 31ms
  creator CPython3Windows(dest=C:\Users\sasch\Documents\GitHub\gecko-dev\obj-x86_64-pc-mingw32\_virtualenvs\common, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator
Collecting black==21.11b1
  Using cached black-21.11b1-py3-none-any.whl (155 kB)
Collecting click==8.0.3
  Using cached click-8.0.3-py3-none-any.whl (97 kB)
Collecting dataclasses==0.6
  Using cached dataclasses-0.6-py3-none-any.whl (14 kB)
Collecting mypy-extensions==0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Requirement already satisfied: pathspec==0.9.0 in c:\users\sasch\documents\github\gecko-dev\third_party\python\pathspec (from -r C:\Users\sasch\Documents\GitHub\gecko-dev\tools\lint\python\black_requirements.txt (line 23)) (0.9.0)
Collecting platformdirs==2.4.0
  Using cached platformdirs-2.4.0-py3-none-any.whl (14 kB)
Collecting regex==2021.11.10
  Using cached regex-2021.11.10-cp310-cp310-win_amd64.whl (273 kB)
Collecting tomli==1.2.2
  Using cached tomli-1.2.2-py3-none-any.whl (12 kB)
Collecting typed-ast==1.4.3
  Using cached typed_ast-1.4.3.tar.gz (210 kB)
Collecting typing-extensions==3.10.0.2
  Using cached typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
Requirement already satisfied: colorama in c:\users\sasch\documents\github\gecko-dev\third_party\python\colorama (from click==8.0.3->-r C:\Users\sasch\Documents\GitHub\gecko-dev\tools\lint\python\black_requirements.txt (line 11)) (0.4.4)
Building wheels for collected packages: typed-ast
  Building wheel for typed-ast (setup.py): started
  Building wheel for typed-ast (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: 'C:\Users\sasch\Documents\GitHub\gecko-dev\obj-x86_64-pc-mingw32\_virtualenvs\common\Scripts\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\sasch\\AppData\\Local\\Temp\\pip-install-_aa0gd58\\typed-ast_e6a71b0b0465406
c9d4935bfe01d348c\\setup.py'"'"'; __file__='"'"'C:\\Users\\sasch\\AppData\\Local\\Temp\\pip-install-_aa0gd58\\typed-ast_e6a71b0b0465406c9d4935bfe01d348c\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'
from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\sasch\AppData\Local\Temp\pip-wheel-kiu1054x'
       cwd: C:\Users\sasch\AppData\Local\Temp\pip-install-_aa0gd58\typed-ast_e6a71b0b0465406c9d4935bfe01d348c\
  Complete output (102 lines):
( ... looooooooong logs ... )

Unable to install correct version of pylint
Try to install it manually with:
    $ pip install -U --require-hashes -r C:\Users\sasch\Documents\GitHub\gecko-dev\tools\lint\python\pylint_requirements.txt
error: problem with lint setup, skipping android-api-lint, android-checkstyle, android-format, android-javadoc, android-lint, android-test, black, l10n, pylint

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

( ... looooooooong logs ... )

Could you attach those logs, so it's possible to identify the (new?) root cause? :)

Comment 2

[Kagami Rosylight [:saschanaz] (they/them)]

Attachment: build failure log

Sure, here's the full log.

Comment 3

[Kagami Rosylight [:saschanaz] (they/them)]

Attachment: build failure log 2

(Grr, the weird linebreaks)

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Upstream issue
Appears to be fixed in 1.5.0

Comment 5

[Kagami Rosylight [:saschanaz] (they/them)]

typed_ast will not be updated to support parsing Python 3.8 and newer. Instead, it is recommended to use the stdlib ast module there, which has been augmented to support extracting type comments and has limited support for parsing older versions of Python 3.

Huh, this looks significant.

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

True, the solution here is to only install typed-ast on platforms/python versions that need it, and avoid it otherwise.
black's version requirements specify this:

Requires-Dist: typed-ast (>=1.4.2) ; python_version < "3.8" and implementation_name == "cpython"

pip-compile doesn't respect these "environment markers" - fortunately, poetry does, so when ./mach lint is ported to use the centralized Python dep management, this issue should be resolved.

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1761507: Bump typed-ast version

This should resolve a ./mach lint incompatibility with
Python 3.10.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b5414dcbc2a5
Bump typed-ast version r=ahal

Comment 9

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/b5414dcbc2a5