1723031 - Assert pip packages are available with MACH_USE_SYSTEM_PYTHON

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement)

Description

[Mitchell Hentges [:mhentges] 🦀]

With requirements being defined up-front, we can fail fast if insufficient packages are installed.

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723031: Vendors `packaging`, `pyparsing` library

This will allow us to parse and compare pip package versions the same
way that pip does.

Depends on D122887

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723031: Allow flexible dependency-specification in the Mach venv

There's some trade-offs in play here: the major issue is that we can't
pin psutil's because it's pre-installed on some CI workers with a
different version (5.4.2).

Additionally, we can't "just" bump that version, because CI workers jump
between different revisions to do work, so a specific pinned version
won't work when we try to update such a package.

One option is that we could avoid validating package versions in CI, but
then that will cause surprises (heck, I didn't know we were still using
psutil==5.4.2 instead of 5.8.0 until now). By doing validation, we
make it more explicit and avoid accidentally depending on behaviour of
too new of such a package.

However, in most cases, we manage the installation environment and can
pin dependencies. So, I've made the top-level Mach virtualenv the only
one that is able to use requirement specifiers other than "==".

Depends on D122888

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723031: In CI, assert Mach pypi package deps using system Python

When using MOZ_AUTOMATION or MACH_USE_SYSTEM_PYTHON, validate
that required dependencies are installed.

Depends on D122889

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:mhentges, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723031: Update Jinja2 pin to be compatible with `glean-parser`

glean-parser==3.6.0 requires Jinja2>=2.10.1.

Depends on D122890

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723031: Set zstandard requirement lower bound to 0.11.1

Windows and Mac workers have zstandard==0.11.1 installed.

Depends on D126283

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723031: Remove pytest from vendored packages

pytest has platform-specific dependencies (colorama on Windows), but
our vendoring logic currently doesn't support having platform-specific
packages.

Since python-test jobs don't have to be isolated from the network,
migrate pytest to be a pypi: dependency.

Depends on D126284

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723031: [WILL SQUASH] Remove `pytest` + deps from `third_party`

Results from ./mach vendor python.

Depends on D126285

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723031: Associate `perftest-test` command with `python-test` venv

The pytest==3.6.2 dependency was moved from common into a
python-test-specific virtualenv. Since perftest-test uses pytest, it
needs to use a virtualenv that has it installed.

Depends on D126286

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6e1bde40e3b4
Vendors `packaging`, `pyparsing` library r=ahal
https://hg.mozilla.org/integration/autoland/rev/784c94c2f528
Allow flexible dependency-specification in the Mach venv r=ahal
https://hg.mozilla.org/integration/autoland/rev/9bfeb01bcc9a
Update Jinja2 pin to be compatible with `glean-parser` r=ahal
https://hg.mozilla.org/integration/autoland/rev/394152994966
Set zstandard requirement lower bound to 0.11.1 r=ahal
https://hg.mozilla.org/integration/autoland/rev/7f64d538701b
In CI, assert Mach pypi package deps using system Python r=ahal

Comment 11

[Noemi Erli[:noemi_erli]]

Backed out 10 changesets (Bug 1712151, Bug 1724279, Bug 1730712, Bug 1717051, Bug 1723031, Bug 1731145) for causing failures on test_yaml.py

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&collapsedPushes=842382&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&classifiedState=classified&revision=7f64d538701b65a841643349899057576adc15a5&selectedTaskRun=W2_veJEgQJmuxcZ91N3RdQ.0

Log: https://treeherder.mozilla.org/logviewer?job_id=352859436&repo=autoland&lineNumber=925
Bustage: https://treeherder.mozilla.org/logviewer?job_id=352865415&repo=autoland&lineNumber=1148

Backout: https://hg.mozilla.org/integration/autoland/rev/b59ab731aecfcc3b9cb4e1388950b70039bf95ac

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3aafa68a8527
Vendors `packaging`, `pyparsing` library r=ahal
https://hg.mozilla.org/integration/autoland/rev/bebfc139b5a9
Allow flexible dependency-specification in the Mach venv r=ahal
https://hg.mozilla.org/integration/autoland/rev/39514bc52fe2
Update Jinja2 pin to be compatible with `glean-parser` r=ahal
https://hg.mozilla.org/integration/autoland/rev/77c9a55337fa
Set zstandard requirement lower bound to 0.11.1 r=ahal

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a8e890ec4273
Associate `perftest-test` command with `python-test` venv r=ahal
https://hg.mozilla.org/integration/autoland/rev/11ebaddfc95a
Remove pytest from vendored packages r=ahal

Comment 14

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/3aafa68a8527
https://hg.mozilla.org/mozilla-central/rev/bebfc139b5a9
https://hg.mozilla.org/mozilla-central/rev/39514bc52fe2
https://hg.mozilla.org/mozilla-central/rev/77c9a55337fa
https://hg.mozilla.org/mozilla-central/rev/a8e890ec4273
https://hg.mozilla.org/mozilla-central/rev/11ebaddfc95a

Comment 15

[Mitchell Hentges [:mhentges] 🦀]

Almost there, but have one more patch to land.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/68940497078c
In CI, assert Mach pypi package deps using system Python r=ahal

Comment 17

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/68940497078c

Comment 18

[Narcis Beleuzu [:NarcisB]]

Backed out for causing pip setup issues (bug 1733142)

Backout link: https://hg.mozilla.org/integration/autoland/rev/60dcd1171dd2b522effa87f53abb36bdde3892fe

Comment 19

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The change got reverted because it also broke the release bouncer check task.

Comment 20

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1723031: Resolve update-verify image python package incompatibility

Due to having apt install "recommended packages", we're having the
python3-gi package installed.
However, though the apt package of python3-gi doesn't depend on
anything, the underlying Python PyGObject package does depend on
pycairo.

By installing python3-cairo, we resolve the missing Python package
dependency, allowing pip check to complete successfully.

Comment 21

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout merge to beta: https://hg.mozilla.org/releases/mozilla-beta/rev/417146e6ebfd8856bd8020c7fd512edd088f1840

Comment 22

[Cristian Tuns]

Backout merged to central: https://hg.mozilla.org/mozilla-central/rev/60dcd1171dd2

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9670952cdb0a
Resolve update-verify image python package incompatibility r=glandium

Comment 24

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/9670952cdb0a

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/98067437416e
In CI, assert Mach pypi package deps using system Python r=ahal

Comment 26

[Cristina Cozmuta (:CrissCozmuta)]

https://hg.mozilla.org/mozilla-central/rev/98067437416e