Closed Bug 1460402 Opened 5 years ago Closed 5 years ago

codespell: in treeherder, use the pip package instead of the distro package (too old)

Categories

(Developer Infrastructure :: Lint and Formatting, enhancement)

Product:
Developer Infrastructure
Component:
Lint and Formatting
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox62 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla62
Tracking Flags:
Tracking Status
firefox62 --- fixed

People

(Reporter: Sylvestre, Assigned: Sylvestre)

Details

Attachments

(4 files)

Bug 1460402 - Create a new class to manage pip install
59 bytes, text/x-review-board-request
ahal
: review+
Details
Bug 1460402 - Update the flake8 support to use the pip class
59 bytes, text/x-review-board-request
ahal
: review+
Details
Bug 1460402 - Update the codespell support to use pip instead of the package
59 bytes, text/x-review-board-request
ahal
: review+
Details
Bug 1460402 - Update the CI to use codespell with pip instead of the apt packages (too old)
59 bytes, text/x-review-board-request
ahal
: review+
Details 
For example, in https://taskcluster-artifacts.net/cUrSg7yGTIq_iP7lSO4oUQ/0/public/logs/live_backing.log
it fails because of:
 [task 2018-05-09T12:04:42.050Z] Unable to match regex against output: codespell: error: no such option: --ignore-words

which shipped in codespell 1.11

I guess we are using a too old version (the Debian & Ubuntu are pretty old).
Comment on attachment 8974513 [details]
Bug 1460402 - Create a new class to manage pip install

https://reviewboard.mozilla.org/r/242850/#review248698


Code analysis found 2 defects in this patch:
 - 2 defects found by mozlint

You can run this analysis locally with:
 - `./mach lint path/to/file` (JS/Python)


If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx


::: python/mozlint/mozlint/pip.py:1
(Diff revision 1)
> +#!/usr/bin/env python

Error: Missing from __future__ import absolute_import [py2: require absolute_import]

::: python/mozlint/mozlint/pip.py:1
(Diff revision 1)
> +#!/usr/bin/env python

Error: Missing from __future__ import print_function [py2: require print_function]
Comment on attachment 8974513 [details]
Bug 1460402 - Create a new class to manage pip install

https://reviewboard.mozilla.org/r/242850/#review248700


Code analysis found 2 defects in this patch:
 - 2 defects found by mozlint

You can run this analysis locally with:
 - `./mach lint path/to/file` (JS/Python)


If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx


::: python/mozlint/mozlint/pip.py:1
(Diff revision 2)
> +# This Source Code Form is subject to the terms of the Mozilla Public

Error: Missing from __future__ import absolute_import [py2: require absolute_import]

::: python/mozlint/mozlint/pip.py:1
(Diff revision 2)
> +# This Source Code Form is subject to the terms of the Mozilla Public

Error: Missing from __future__ import print_function [py2: require print_function]
Assignee: nobody → sledru
Summary: codespell: too old version of codespell used in treeherder → codespell: in treeherder, use the pip package instead of the distro package (too old)
Comment on attachment 8974513 [details]
Bug 1460402 - Create a new class to manage pip install

https://reviewboard.mozilla.org/r/242850/#review248922

Thanks for the cleanup!

Eventually I'd like if we could use the build system's virtualenv management infrastructure, but that is definitely follow-up fodder.

::: python/mozlint/mozlint/pip.py:1
(Diff revision 2)
> +# This Source Code Form is subject to the terms of the Mozilla Public

Please move this file to `python/mozlint/mozlint/util/pip.py`, as it isn't really related to the core mozlint library at all.
Attachment #8974513 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8974514 [details]
Bug 1460402 - Update the flake8 support to use the pip class

https://reviewboard.mozilla.org/r/242852/#review248924

::: tools/lint/python/__init__.py:13
(Diff revision 2)
>  from collections import defaultdict
>  
>  import which
>  from mozprocess import ProcessHandlerMixin
>  
> -from mozlint import result
> +from mozlint import pip, result

This will need to be `from mozlint.util import pip` based on the issue in the previous commit.
Attachment #8974514 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8974515 [details]
Bug 1460402 - Update the codespell support to use pip instead of the package

https://reviewboard.mozilla.org/r/242854/#review248920

::: tools/lint/spell/__init__.py:22
(Diff revision 2)
>  
> -from mozlint import result
> +from mozlint import pip, result
>  from mozprocess import ProcessHandlerMixin
>  
> +here = os.path.abspath(os.path.dirname(__file__))
> +CODESPELL_REQUIREMENTS_PATH = os.path.join(here, 'codespell_requirements.txt')

I think you forgot to add this file.
Attachment #8974515 - Flags: review?(ahalberstadt) → review-
Comment on attachment 8974516 [details]
Bug 1460402 - Update the CI to use codespell with pip instead of the apt packages (too old)

https://reviewboard.mozilla.org/r/242856/#review248926
Attachment #8974516 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8974515 [details]
Bug 1460402 - Update the codespell support to use pip instead of the package

https://reviewboard.mozilla.org/r/242854/#review249000

Thanks for fixing this, lgtm!
Attachment #8974515 - Flags: review?(ahalberstadt) → review+
Pushed by sledru@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/06ceda084d69
Create a new class to manage pip install r=ahal
https://hg.mozilla.org/integration/autoland/rev/e1caff997e5a
Update the flake8 support to use the pip class r=ahal
https://hg.mozilla.org/integration/autoland/rev/17526c61b995
Update the codespell support to use pip instead of the package r=ahal
https://hg.mozilla.org/integration/autoland/rev/5b40f3f18f42
Update the CI to use codespell with pip instead of the apt packages (too old) r=ahal
Backed out 4 changesets (bug 1460402) for breaking taskcluster images. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer.html#?job_id=177938228&repo=autoland&lineNumber=1596

[task 2018-05-10T20:25:19.075Z] Successfully built configparser
[task 2018-05-10T20:25:19.076Z] Installing collected packages: pyflakes, enum34, configparser, pycodestyle, mccabe, flake8, setuptools
[task 2018-05-10T20:25:19.190Z]   Found existing installation: setuptools 20.7.0
[task 2018-05-10T20:25:19.192Z]     Not uninstalling setuptools at /usr/lib/python2.7/dist-packages, outside environment /usr
[task 2018-05-10T20:25:19.383Z] Successfully installed configparser-3.5.0 enum34-1.1.6 flake8-3.5.0 mccabe-0.6.1 pycodestyle-2.3.1 pyflakes-1.5.0 setuptools-38.5.1
[task 2018-05-10T20:25:19.592Z] You are using pip version 8.1.1, however version 10.0.1 is available.
[task 2018-05-10T20:25:19.592Z] You should consider upgrading via the 'pip install --upgrade pip' command.
[task 2018-05-10T20:25:19.614Z] 
[task 2018-05-10T20:25:19.615Z] ###
[task 2018-05-10T20:25:19.615Z] # codespell Setup
[task 2018-05-10T20:25:19.615Z] ###
[task 2018-05-10T20:25:19.615Z] 
[task 2018-05-10T20:25:19.615Z] cd /setup
[task 2018-05-10T20:25:19.615Z] 
[task 2018-05-10T20:25:19.615Z] pip install --require-hashes -r /tmp/codespell_requirements.txt
[task 2018-05-10T20:25:20.161Z] Could not open requirements file: [Errno 2] No such file or directory: '/tmp/codespell_requirements.txt'
[task 2018-05-10T20:25:20.320Z] You are using pip version 8.1.1, however version 10.0.1 is available.
[task 2018-05-10T20:25:20.320Z] You should consider upgrading via the 'pip install --upgrade pip' command.
[task 2018-05-10T20:25:21.411Z] Traceback (most recent call last):
[task 2018-05-10T20:25:21.411Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 447, in build_image
[task 2018-05-10T20:25:21.411Z]     build_image(image_name, tag, os.environ)
[task 2018-05-10T20:25:21.411Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/docker.py", line 83, in build_image
[task 2018-05-10T20:25:21.411Z]     docker.post_to_docker(buf.getvalue(), '/build', nocache=1, t=tag)
[task 2018-05-10T20:25:21.411Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/docker.py", line 107, in post_to_docker
[task 2018-05-10T20:25:21.411Z]     raise Exception(data['error'])
[task 2018-05-10T20:25:21.411Z] Exception: The command [/bin/sh -c bash /tmp/system-setup.sh] returned a non-zero code: 1
kill 8
+ kill 8
[taskcluster 2018-05-10 20:25:21.654Z] === Task Finished ===
[taskcluster 2018-05-10 20:25:21.765Z] Artifact "public/image.tar.zst" not found at "/builds/worker/workspace/artifacts/image.tar.zst"
[taskcluster 2018-05-10 20:25:22.307Z] Unsuccessful task run with exit code: 1 completed in 342.827 seconds

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=5b40f3f18f42e292d68bad895bc08bad184e847c

Backout:
https://hg.mozilla.org/integration/autoland/rev/bca28818826ba017ad63fc50a1ce2783b0c17d0a
Flags: needinfo?(sledru)
Pushed by sledru@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3e50885329c4
Create a new class to manage pip install r=ahal
https://hg.mozilla.org/integration/autoland/rev/bb12ffd4b96e
Update the flake8 support to use the pip class r=ahal
https://hg.mozilla.org/integration/autoland/rev/3676e913dbff
Update the codespell support to use pip instead of the package r=ahal
https://hg.mozilla.org/integration/autoland/rev/c2e8fbd72ca6
Update the CI to use codespell with pip instead of the apt packages (too old) r=ahal
Backed out 4 changesets (bug 1460402) for lint failure on intl/locales/en-US/hyphenation/hyph_en_US.dic. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer.html#?job_id=177948127&repo=autoland&lineNumber=445

task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/pbmode.ico
[task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/default128.png
[task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/firefox.icns
[task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/newwindow.ico
[task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/document.icns
[task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/background.png
[task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/wizWatermark.bmp
[task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/default64.png
[task 2018-05-10T21:38:25.322Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/default16.png
[task 2018-05-10T21:38:25.322Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/disk.icns
[task 2018-05-10T21:38:25.322Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/VisualElements_70.png
[task 2018-05-10T21:38:25.322Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/document.ico
[task 2018-05-10T21:38:25.322Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/newtab.ico
[task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/dsstore
[task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/wizHeader.bmp
[task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/firefox.ico
[task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/wizHeaderRTL.bmp
[task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/content/about-logo@2x.png
[task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/content/about-background.png
[task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/content/about-logo.png
[task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/content/about.png
[task 2018-05-10T21:38:25.339Z] TEST-UNEXPECTED-WARNING | intl/locales/en-US/hyphenation/hyph_en_US.dic:2039 | nd  ==> and, 2nd (codespell)
[task 2018-05-10T21:38:25.339Z] TEST-UNEXPECTED-WARNING | intl/locales/en-US/hyphenation/hyph_en_US.dic:11040 | nd  ==> and, 2nd (codespell)
[task 2018-05-10T21:38:25.340Z] TEST-UNEXPECTED-WARNING | intl/locales/en-US/hyphenation/hyph_en_US.dic:11098 | nd  ==> and, 2nd (codespell)
[taskcluster 2018-05-10 21:38:25.670Z] === Task Finished ===
[taskcluster 2018-05-10 21:38:25.670Z] Unsuccessful task run with exit code: 1 completed in 292.611 seconds

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=c2e8fbd72ca649e29403c31c8cee9e36cf83ebd6

Backout:
https://hg.mozilla.org/integration/autoland/rev/5a8eea8a92bbfff1c4b324e1137f73e9956f9643
Pushed by sledru@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c1f425788ae6
Create a new class to manage pip install r=ahal
https://hg.mozilla.org/integration/autoland/rev/b69e58c4e0fd
Update the flake8 support to use the pip class r=ahal
https://hg.mozilla.org/integration/autoland/rev/3f8ad0565db2
Update the codespell support to use pip instead of the package r=ahal
https://hg.mozilla.org/integration/autoland/rev/8def71624ff1
Update the CI to use codespell with pip instead of the apt packages (too old) r=ahal
Landed!
Flags: needinfo?(sledru)
Product: Firefox Build System → Developer Infrastructure
You need to log in before you can comment on or make changes to this bug.