1712151 - Validate in CI virtualenvs don't have dependency collisions with "global" virtualenv, or between vendored + pypi'd deps

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement, P2)

Description

[Mitchell Hentges [:mhentges] 🦀]

First, some backstory: there's a two different ways of activating the command's virtual environment:

1. Use an independent Python process.
2. Re-use the existing process and activate the new virtualenv within it.

Using an independent Python process fully insulates the command (and its dependencies) from the global mach operations' dependencies. However, in addition to the performance hit (roughly ~100ms Python startup time on a fast Linux machine), it has some inflexibilities:

1. Passing data (moz.build parsed goodies, context, etc) becomes more difficult - it has to be encoded, transferred to the other process, and decoded.
2. Behaviour that would be trivial in a single Python process becomes harder:
   • Having sentry capture exceptions from either mach or the active command.
   • Passing a glean handle into virtualenvs (we'd have to ensure that commands are using the same version of glean as global mach, as well as find a nice way of sharing the initialization logic).

Now, the downside of re-using the same python process is that import state is maintained: if we import glean-sdk==38.0.1 (mach venv), and that imports glean_parser==3.2.0 (mach venv), and that imports PyYAML==3.13 (mach venv), then import yaml will always return PyYAML==3.13 from the (mach venv), even after the virtualenv changes. (Technically, imports can be re-loaded, but then that can cause issues if glean-sdk performs an import from within a function and gets the wrong package).

Fortunately, this issue is completely manageable - if we validate that each virtualenv's dependencies are compatible with the global mach venv's dependencies, then we won't have issues.

From my perspective, the "validation" technique is the most manageable: we still get to write a single cohesive Python application, but we are able to have the different commands have their own distinct dependencies which are incompatible with each other.

This ticket is to capture the automatic validation of the command virtualenvs in CI.
Note that, due to system-specific requirements, we probably need to run these tests in:

• All supported Pythons (Python 3.6 through 3.9)
• All primary platforms (Mac, Windows, Linux)

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

We're going to need to validate command dependency compatibility with both:

• Global Mach dependencies and
• Python testing dependencies

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1712151: Use compatible version of pyasn1-modules

The existing version of pyasn1-modules (0.1.5) is incompatible with
our version of pyasn1 (0.4.8).

By bumping pyasn1-modules to 0.2.8, we now meet its compatibility
requirements.

Depends on D122896

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1712151: Vendor pystache automatically

Add pystache to vendor requirements.in so that it's vendored according
to ./mach vendor python "ignore" rules.
This ensures that sufficient files are vendored such that installing the
package from it's setup.py file is possible.

Depends on D122897

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1712151: Bump glean-sdk version to 40.0.0

We vendor glean_parser==3.6.0, and that was incompatible with
glean-sdk==36.0.0's requirement of glean_parser==2.5.0.
glean-sdk==40.0.0 expects glean_parser==3.6.0, which is perfect.

Depends on D122898

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1712151: Add test to verify virtualenv compatibility

This adds two main compatibility guarantees:

1. Vendored dependencies <=> Pypi-downloaded dependencies
2. Global Mach dependencies <=> command-specific dependencies

As part of this, a new vendored: action was added to the virtualenv
definition format. Otherwise similar to pth: paths, vendored:
packages are assumed to be "pip install"-able.

Some validation (the .dist-info check) was added to requirements.py
to verify that pth: and vendored: are correctly used. However, I
couldn't add a setup.py check because some of our un-installable
internal modules (mozbase, etc) have out-of-date and potentially
inaccurate setup.py files. Though it would be nice to leverage each
internal module as an independent package with its own scoped set of
dependencies, that's a project for another time.

Depends on D122899

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3f85c601e434
Bump glean-sdk version to 40.0.0 r=ahal

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9ccf99dae3c7
Use compatible version of pyasn1-modules r=ahal
https://hg.mozilla.org/integration/autoland/rev/2c6edefd8417
Vendor pystache automatically r=ahal

Comment 8

[Cosmin Sabou [:CosminS]]

Backed out for causing invalid handle xpcshell failures.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=U48eqelWQge_ZBWzSGPX-A.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel&revision=90b9ff20b311609163c786cdee830ff9a132d575

Failure log: https://treeherder.mozilla.org/logviewer?job_id=350477627&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/6b58b83b626e01f7b2af85156c293c9bccd82816

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Bump glean-sdk version to 40.0.0 r=ahal
https://hg.mozilla.org/integration/autoland/rev/3f85c601e434331a3b06eac39c216bfadc39cf7f
https://hg.mozilla.org/mozilla-central/rev/3f85c601e434

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c196c1ecdff7
Use compatible version of pyasn1-modules r=ahal
https://hg.mozilla.org/integration/autoland/rev/089c1a0d9ec8
Vendor pystache automatically r=ahal

Comment 11

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/c196c1ecdff7
https://hg.mozilla.org/mozilla-central/rev/089c1a0d9ec8

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7adf7e2136a3
Add test to verify virtualenv compatibility r=ahal

Comment 13

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b59ab731aecf
Backed out 10 changesets (bug 1712151, bug 1724279, bug 1730712, bug 1717051, bug 1723031, bug 1731145) for causing failures on test_yaml.py

Comment 14

[Noemi Erli[:noemi_erli]]

Backed out 10 changesets (Bug 1712151, Bug 1724279, Bug 1730712, Bug 1717051, Bug 1723031, Bug 1731145) for causing failures on test_yaml.py

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&collapsedPushes=842382&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&classifiedState=classified&revision=7f64d538701b65a841643349899057576adc15a5&selectedTaskRun=W2_veJEgQJmuxcZ91N3RdQ.0

Log: https://treeherder.mozilla.org/logviewer?job_id=352859436&repo=autoland&lineNumber=925
Bustage: https://treeherder.mozilla.org/logviewer?job_id=352865415&repo=autoland&lineNumber=1148

Backout: https://hg.mozilla.org/integration/autoland/rev/b59ab731aecfcc3b9cb4e1388950b70039bf95ac

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1860400b41c4
Add test to verify virtualenv compatibility r=ahal

Comment 16

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/1860400b41c4

Comment 17

[Mitchell Hentges [:mhentges] 🦀]

Note that, due to system-specific requirements, we probably need to run these tests in:

• All supported Pythons (Python 3.6 through 3.9)
• All primary platforms (Mac, Windows, Linux)

I'm going to defer the cross-platform-test part of this ticket.