Closed Bug 1723031 Opened 3 months ago Closed 5 days ago

Assert pip packages are available with MACH_USE_SYSTEM_PYTHON

Categories

(Firefox Build System :: Mach Core, enhancement)

enhancement

Tracking

(firefox95 fixed)

RESOLVED FIXED
95 Branch
Tracking Status
firefox95 --- fixed

People

(Reporter: mhentges, Assigned: mhentges)

References

Details

Attachments

(8 files, 1 obsolete file)

48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review

With requirements being defined up-front, we can fail fast if insufficient packages are installed.

Assignee: nobody → mhentges
Status: NEW → ASSIGNED

This will allow us to parse and compare pip package versions the same
way that pip does.

Depends on D122887

There's some trade-offs in play here: the major issue is that we can't
pin psutil's because it's pre-installed on some CI workers with a
different version (5.4.2).

Additionally, we can't "just" bump that version, because CI workers jump
between different revisions to do work, so a specific pinned version
won't work when we try to update such a package.

One option is that we could avoid validating package versions in CI, but
then that will cause surprises (heck, I didn't know we were still using
psutil==5.4.2 instead of 5.8.0 until now). By doing validation, we
make it more explicit and avoid accidentally depending on behaviour of
too new of such a package.

However, in most cases, we manage the installation environment and can
pin dependencies. So, I've made the top-level Mach virtualenv the only
one that is able to use requirement specifiers other than "==".

Depends on D122888

When using MOZ_AUTOMATION or MACH_USE_SYSTEM_PYTHON, validate
that required dependencies are installed.

Depends on D122889

Attachment #9236702 - Attachment description: Bug 1723031: Vendors `packaging` library → Bug 1723031: Vendors `packaging`, `pyparsing` library
Attachment #9236703 - Attachment description: Bug 1723031: Allow flexible dependency-specificiation in the Mach venv → Bug 1723031: Allow flexible dependency-specification in the Mach venv
See Also: → 1726825

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:mhentges, could you have a look please?
For more information, please visit auto_nag documentation.

Flags: needinfo?(mhentges)
Flags: needinfo?(ahal)
Flags: needinfo?(ahal)
Flags: needinfo?(mhentges)
Attachment #9236704 - Attachment description: Bug 1723031: Assert package dependencies with system Python → Bug 1723031: In CI, assert Mach pypi package deps using system Python

glean-parser==3.6.0 requires Jinja2>=2.10.1.

Depends on D122890

Windows and Mac workers have zstandard==0.11.1 installed.

Depends on D126283

pytest has platform-specific dependencies (colorama on Windows), but
our vendoring logic currently doesn't support having platform-specific
packages.

Since python-test jobs don't have to be isolated from the network,
migrate pytest to be a pypi: dependency.

Depends on D126284

Results from ./mach vendor python.

Depends on D126285

The pytest==3.6.2 dependency was moved from common into a
python-test-specific virtualenv. Since perftest-test uses pytest, it
needs to use a virtualenv that has it installed.

Depends on D126286

Attachment #9242353 - Attachment is obsolete: true
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6e1bde40e3b4
Vendors `packaging`, `pyparsing` library r=ahal
https://hg.mozilla.org/integration/autoland/rev/784c94c2f528
Allow flexible dependency-specification in the Mach venv r=ahal
https://hg.mozilla.org/integration/autoland/rev/9bfeb01bcc9a
Update Jinja2 pin to be compatible with `glean-parser` r=ahal
https://hg.mozilla.org/integration/autoland/rev/394152994966
Set zstandard requirement lower bound to 0.11.1 r=ahal
https://hg.mozilla.org/integration/autoland/rev/7f64d538701b
In CI, assert Mach pypi package deps using system Python r=ahal
Flags: needinfo?(mhentges)
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3aafa68a8527
Vendors `packaging`, `pyparsing` library r=ahal
https://hg.mozilla.org/integration/autoland/rev/bebfc139b5a9
Allow flexible dependency-specification in the Mach venv r=ahal
https://hg.mozilla.org/integration/autoland/rev/39514bc52fe2
Update Jinja2 pin to be compatible with `glean-parser` r=ahal
https://hg.mozilla.org/integration/autoland/rev/77c9a55337fa
Set zstandard requirement lower bound to 0.11.1 r=ahal
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a8e890ec4273
Associate `perftest-test` command with `python-test` venv r=ahal
https://hg.mozilla.org/integration/autoland/rev/11ebaddfc95a
Remove pytest from vendored packages r=ahal

Almost there, but have one more patch to land.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/68940497078c
In CI, assert Mach pypi package deps using system Python r=ahal
Regressions: 1733142
See Also: → 1733144
See Also: 1733144
Status: REOPENED → RESOLVED
Closed: 20 days ago19 days ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Flags: needinfo?(mhentges)
Resolution: FIXED → ---
Target Milestone: 94 Branch → ---
Flags: needinfo?(mhentges)

The change got reverted because it also broke the release bouncer check task.

Due to having apt install "recommended packages", we're having the
python3-gi package installed.
However, though the apt package of python3-gi doesn't depend on
anything, the underlying Python PyGObject package does depend on
pycairo.

By installing python3-cairo, we resolve the missing Python package
dependency, allowing pip check to complete successfully.

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9670952cdb0a
Resolve update-verify image python package incompatibility r=glandium
Status: REOPENED → RESOLVED
Closed: 19 days ago13 days ago
Resolution: --- → FIXED
Target Milestone: --- → 95 Branch
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: 95 Branch → ---
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/98067437416e
In CI, assert Mach pypi package deps using system Python r=ahal
Status: REOPENED → RESOLVED
Closed: 13 days ago5 days ago
Resolution: --- → FIXED
Target Milestone: --- → 95 Branch
You need to log in before you can comment on or make changes to this bug.