Closed Bug 1460402 Opened 7 years ago Closed 7 years ago

codespell: in treeherder, use the pip package instead of the distro package (too old)

Categories

(Developer Infrastructure :: Lint and Formatting, enhancement)

Product:
Developer Infrastructure
Component:
Lint and Formatting
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox62 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla62
Tracking Flags:
Tracking Status
firefox62 --- fixed

People

(Reporter: Sylvestre, Assigned: Sylvestre)

Details

Attachments

(4 files)

Bug 1460402 - Create a new class to manage pip install
59 bytes, text/x-review-board-request
ahal
: review+
Details
Bug 1460402 - Update the flake8 support to use the pip class
59 bytes, text/x-review-board-request
ahal
: review+
Details
Bug 1460402 - Update the codespell support to use pip instead of the package
59 bytes, text/x-review-board-request
ahal
: review+
Details
Bug 1460402 - Update the CI to use codespell with pip instead of the apt packages (too old)
59 bytes, text/x-review-board-request
ahal
: review+
Details
For example, in https://taskcluster-artifacts.net/cUrSg7yGTIq_iP7lSO4oUQ/0/public/logs/live_backing.log it fails because of: [task 2018-05-09T12:04:42.050Z] Unable to match regex against output: codespell: error: no such option: --ignore-words which shipped in codespell 1.11 I guess we are using a too old version (the Debian & Ubuntu are pretty old).
Comment on attachment 8974513 [details] Bug 1460402 - Create a new class to manage pip install https://reviewboard.mozilla.org/r/242850/#review248698 Code analysis found 2 defects in this patch: - 2 defects found by mozlint You can run this analysis locally with: - `./mach lint path/to/file` (JS/Python) If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx ::: python/mozlint/mozlint/pip.py:1 (Diff revision 1) > +#!/usr/bin/env python Error: Missing from __future__ import absolute_import [py2: require absolute_import] ::: python/mozlint/mozlint/pip.py:1 (Diff revision 1) > +#!/usr/bin/env python Error: Missing from __future__ import print_function [py2: require print_function]
Comment on attachment 8974513 [details] Bug 1460402 - Create a new class to manage pip install https://reviewboard.mozilla.org/r/242850/#review248700 Code analysis found 2 defects in this patch: - 2 defects found by mozlint You can run this analysis locally with: - `./mach lint path/to/file` (JS/Python) If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx ::: python/mozlint/mozlint/pip.py:1 (Diff revision 2) > +# This Source Code Form is subject to the terms of the Mozilla Public Error: Missing from __future__ import absolute_import [py2: require absolute_import] ::: python/mozlint/mozlint/pip.py:1 (Diff revision 2) > +# This Source Code Form is subject to the terms of the Mozilla Public Error: Missing from __future__ import print_function [py2: require print_function]
Assignee: nobody → sledru
Summary: codespell: too old version of codespell used in treeherder → codespell: in treeherder, use the pip package instead of the distro package (too old)
Comment on attachment 8974513 [details] Bug 1460402 - Create a new class to manage pip install https://reviewboard.mozilla.org/r/242850/#review248922 Thanks for the cleanup! Eventually I'd like if we could use the build system's virtualenv management infrastructure, but that is definitely follow-up fodder. ::: python/mozlint/mozlint/pip.py:1 (Diff revision 2) > +# This Source Code Form is subject to the terms of the Mozilla Public Please move this file to `python/mozlint/mozlint/util/pip.py`, as it isn't really related to the core mozlint library at all.
Attachment #8974513 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8974514 [details] Bug 1460402 - Update the flake8 support to use the pip class https://reviewboard.mozilla.org/r/242852/#review248924 ::: tools/lint/python/__init__.py:13 (Diff revision 2) > from collections import defaultdict > > import which > from mozprocess import ProcessHandlerMixin > > -from mozlint import result > +from mozlint import pip, result This will need to be `from mozlint.util import pip` based on the issue in the previous commit.
Attachment #8974514 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8974515 [details] Bug 1460402 - Update the codespell support to use pip instead of the package https://reviewboard.mozilla.org/r/242854/#review248920 ::: tools/lint/spell/__init__.py:22 (Diff revision 2) > > -from mozlint import result > +from mozlint import pip, result > from mozprocess import ProcessHandlerMixin > > +here = os.path.abspath(os.path.dirname(__file__)) > +CODESPELL_REQUIREMENTS_PATH = os.path.join(here, 'codespell_requirements.txt') I think you forgot to add this file.
Attachment #8974515 - Flags: review?(ahalberstadt) → review-
Comment on attachment 8974516 [details] Bug 1460402 - Update the CI to use codespell with pip instead of the apt packages (too old) https://reviewboard.mozilla.org/r/242856/#review248926
Attachment #8974516 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8974515 [details] Bug 1460402 - Update the codespell support to use pip instead of the package https://reviewboard.mozilla.org/r/242854/#review249000 Thanks for fixing this, lgtm!
Attachment #8974515 - Flags: review?(ahalberstadt) → review+
Pushed by sledru@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/06ceda084d69 Create a new class to manage pip install r=ahal https://hg.mozilla.org/integration/autoland/rev/e1caff997e5a Update the flake8 support to use the pip class r=ahal https://hg.mozilla.org/integration/autoland/rev/17526c61b995 Update the codespell support to use pip instead of the package r=ahal https://hg.mozilla.org/integration/autoland/rev/5b40f3f18f42 Update the CI to use codespell with pip instead of the apt packages (too old) r=ahal
Backed out 4 changesets (bug 1460402) for breaking taskcluster images. CLOSED TREE Log: https://treeherder.mozilla.org/logviewer.html#?job_id=177938228&repo=autoland&lineNumber=1596 [task 2018-05-10T20:25:19.075Z] Successfully built configparser [task 2018-05-10T20:25:19.076Z] Installing collected packages: pyflakes, enum34, configparser, pycodestyle, mccabe, flake8, setuptools [task 2018-05-10T20:25:19.190Z] Found existing installation: setuptools 20.7.0 [task 2018-05-10T20:25:19.192Z] Not uninstalling setuptools at /usr/lib/python2.7/dist-packages, outside environment /usr [task 2018-05-10T20:25:19.383Z] Successfully installed configparser-3.5.0 enum34-1.1.6 flake8-3.5.0 mccabe-0.6.1 pycodestyle-2.3.1 pyflakes-1.5.0 setuptools-38.5.1 [task 2018-05-10T20:25:19.592Z] You are using pip version 8.1.1, however version 10.0.1 is available. [task 2018-05-10T20:25:19.592Z] You should consider upgrading via the 'pip install --upgrade pip' command. [task 2018-05-10T20:25:19.614Z] [task 2018-05-10T20:25:19.615Z] ### [task 2018-05-10T20:25:19.615Z] # codespell Setup [task 2018-05-10T20:25:19.615Z] ### [task 2018-05-10T20:25:19.615Z] [task 2018-05-10T20:25:19.615Z] cd /setup [task 2018-05-10T20:25:19.615Z] [task 2018-05-10T20:25:19.615Z] pip install --require-hashes -r /tmp/codespell_requirements.txt [task 2018-05-10T20:25:20.161Z] Could not open requirements file: [Errno 2] No such file or directory: '/tmp/codespell_requirements.txt' [task 2018-05-10T20:25:20.320Z] You are using pip version 8.1.1, however version 10.0.1 is available. [task 2018-05-10T20:25:20.320Z] You should consider upgrading via the 'pip install --upgrade pip' command. [task 2018-05-10T20:25:21.411Z] Traceback (most recent call last): [task 2018-05-10T20:25:21.411Z] File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 447, in build_image [task 2018-05-10T20:25:21.411Z] build_image(image_name, tag, os.environ) [task 2018-05-10T20:25:21.411Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/docker.py", line 83, in build_image [task 2018-05-10T20:25:21.411Z] docker.post_to_docker(buf.getvalue(), '/build', nocache=1, t=tag) [task 2018-05-10T20:25:21.411Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/docker.py", line 107, in post_to_docker [task 2018-05-10T20:25:21.411Z] raise Exception(data['error']) [task 2018-05-10T20:25:21.411Z] Exception: The command [/bin/sh -c bash /tmp/system-setup.sh] returned a non-zero code: 1 kill 8 + kill 8 [taskcluster 2018-05-10 20:25:21.654Z] === Task Finished === [taskcluster 2018-05-10 20:25:21.765Z] Artifact "public/image.tar.zst" not found at "/builds/worker/workspace/artifacts/image.tar.zst" [taskcluster 2018-05-10 20:25:22.307Z] Unsuccessful task run with exit code: 1 completed in 342.827 seconds Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=5b40f3f18f42e292d68bad895bc08bad184e847c Backout: https://hg.mozilla.org/integration/autoland/rev/bca28818826ba017ad63fc50a1ce2783b0c17d0a
Flags: needinfo?(sledru)
Pushed by sledru@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/3e50885329c4 Create a new class to manage pip install r=ahal https://hg.mozilla.org/integration/autoland/rev/bb12ffd4b96e Update the flake8 support to use the pip class r=ahal https://hg.mozilla.org/integration/autoland/rev/3676e913dbff Update the codespell support to use pip instead of the package r=ahal https://hg.mozilla.org/integration/autoland/rev/c2e8fbd72ca6 Update the CI to use codespell with pip instead of the apt packages (too old) r=ahal
Backed out 4 changesets (bug 1460402) for lint failure on intl/locales/en-US/hyphenation/hyph_en_US.dic. CLOSED TREE Log: https://treeherder.mozilla.org/logviewer.html#?job_id=177948127&repo=autoland&lineNumber=445 task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/pbmode.ico [task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/default128.png [task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/firefox.icns [task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/newwindow.ico [task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/document.icns [task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/background.png [task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/wizWatermark.bmp [task 2018-05-10T21:38:25.321Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/default64.png [task 2018-05-10T21:38:25.322Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/default16.png [task 2018-05-10T21:38:25.322Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/disk.icns [task 2018-05-10T21:38:25.322Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/VisualElements_70.png [task 2018-05-10T21:38:25.322Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/document.ico [task 2018-05-10T21:38:25.322Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/newtab.ico [task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/dsstore [task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/wizHeader.bmp [task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/firefox.ico [task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/wizHeaderRTL.bmp [task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/content/about-logo@2x.png [task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/content/about-background.png [task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/content/about-logo.png [task 2018-05-10T21:38:25.323Z] Unable to match regex against output: WARNING: Binary file: /builds/worker/checkouts/gecko/browser/branding/unofficial/content/about.png [task 2018-05-10T21:38:25.339Z] TEST-UNEXPECTED-WARNING | intl/locales/en-US/hyphenation/hyph_en_US.dic:2039 | nd ==> and, 2nd (codespell) [task 2018-05-10T21:38:25.339Z] TEST-UNEXPECTED-WARNING | intl/locales/en-US/hyphenation/hyph_en_US.dic:11040 | nd ==> and, 2nd (codespell) [task 2018-05-10T21:38:25.340Z] TEST-UNEXPECTED-WARNING | intl/locales/en-US/hyphenation/hyph_en_US.dic:11098 | nd ==> and, 2nd (codespell) [taskcluster 2018-05-10 21:38:25.670Z] === Task Finished === [taskcluster 2018-05-10 21:38:25.670Z] Unsuccessful task run with exit code: 1 completed in 292.611 seconds Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=c2e8fbd72ca649e29403c31c8cee9e36cf83ebd6 Backout: https://hg.mozilla.org/integration/autoland/rev/5a8eea8a92bbfff1c4b324e1137f73e9956f9643
Pushed by sledru@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/c1f425788ae6 Create a new class to manage pip install r=ahal https://hg.mozilla.org/integration/autoland/rev/b69e58c4e0fd Update the flake8 support to use the pip class r=ahal https://hg.mozilla.org/integration/autoland/rev/3f8ad0565db2 Update the codespell support to use pip instead of the package r=ahal https://hg.mozilla.org/integration/autoland/rev/8def71624ff1 Update the CI to use codespell with pip instead of the apt packages (too old) r=ahal
Landed!
Flags: needinfo?(sledru)
Product: Firefox Build System → Developer Infrastructure
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: