1712819 - Avoid pip's "outdated" warning in virtualenvs

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement)

Description

[Mitchell Hentges [:mhentges] 🦀]

When running commands line ./mach lint --fix, I see pip's outdated warning multiple times:

 (hg)-[default]-~/d/firefox % ./mach lint --fix    
created virtual environment CPython3.9.5.final.0-64 in 126ms
  creator CPython3Posix(dest=/home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/common, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mitch/.local/share/virtualenv)
    added seed packages: pip==20.3.1, setuptools==51.0.0, wheel==0.36.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
warning: linting the entire repo takes a long time, using --outgoing and --workdir instead. If you want to lint the entire repo, run `./mach lint .`
WARNING: You are using pip version 20.3.1; however, version 21.1.2 is available.
You should consider upgrading via the '/home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/common/bin/python -m pip install --upgrade pip' command.
WARNING: You are using pip version 20.3.1; however, version 21.1.2 is available.
You should consider upgrading via the '/home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/common/bin/python -m pip install --upgrade pip' command.
WARNING: You are using pip version 20.3.1; however, version 21.1.2 is available.
You should consider upgrading via the '/home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/common/bin/python -m pip install --upgrade pip' command.
WARNING: You are using pip version 20.3.1; however, version 21.1.2 is available.
You should consider upgrading via the '/home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/common/bin/python -m pip install --upgrade pip' command.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
jsonschema 3.2.0 requires attrs>=17.4.0, which is not installed.
jsonschema 3.2.0 requires pyrsistent>=0.14.0, which is not installed.
WARNING: You are using pip version 20.3.1; however, version 21.1.2 is available.
You should consider upgrading via the '/home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/common/bin/python -m pip install --upgrade pip' command.
...

This is not ideal for two reasons:

• It would be obnoxious for developers to have to manually update every virtualenv's pip every time an env was generated. It's not viable.
• It's better to have pip be pinned (due to its embedded nature in virtualenv)

Reported a feature request to virtualenv.

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1712819: Fix VirtualenvManager not being expandable in debugger

At least in PyCharm, expanding a VirtualenvManager instance means
resolving all the properties and fields of the instance.
However, if that property is calling a subprocess, the debugger
wouldn't run that subprocess while we're stopped at a breakpoint.
So, the instance would sit there with the "Collecting data..." text.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1712819: Avoid pip's "outdated" warning in virtualenvs

Now, when running mach commands that invoke pip, it will no longer
inform the user that it needs an update.

We reach into distutils to determine the "site-packages" directory
pattern, then apply it to our virtualenv.

Depends on D115935

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/40c109bb5f9a
Fix VirtualenvManager not being expandable in debugger r=ahal
https://hg.mozilla.org/integration/autoland/rev/f51d72e5f0b8
Avoid pip's "outdated" warning in virtualenvs r=ahal

Comment 4

[Dorel Luca [:dluca]]

Backed out 2 changesets (bug 1712819) for Python failures. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer?job_id=340969366&repo=autoland&lineNumber=667

https://treeherder.mozilla.org/logviewer?job_id=340972607&repo=autoland&lineNumber=339

https://treeherder.mozilla.org/logviewer?job_id=340969341&repo=autoland&lineNumber=708

https://treeherder.mozilla.org/logviewer?job_id=340969413&repo=autoland&lineNumber=598

Push with failures:
https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=f51d72e5f0b89c4b4e32db9ed047ff291553a43f

Backout:
https://hg.mozilla.org/integration/autoland/rev/034912384c8dfba9622a2ee586912e2b6afbe767

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Was an issue with self.virtualenv_root having both / and \\ separators. Fantastic.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9c25bbd07129
Fix VirtualenvManager not being expandable in debugger r=ahal
https://hg.mozilla.org/integration/autoland/rev/3a5b335bdc8c
Avoid pip's "outdated" warning in virtualenvs r=ahal

Comment 7

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/9c25bbd07129
https://hg.mozilla.org/mozilla-central/rev/3a5b335bdc8c

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1712819: Remove redundant pip warning suppression

Now that we've deployed a virtualenv configuration to turn off
pip's "outdated" warning, we no longer need to apply the suppression
as a CLI argument.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4335f275d7b4
Remove redundant pip warning suppression r=ahal

Comment 10

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/4335f275d7b4