1758584 - Raise error if active site is marked as out-of-date

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement)

Description

[Mitchell Hentges [:mhentges] 🦀]

There's some cases in CI where the "system python" is used, and the build site is activated multiple times.
Confusingly, the build site is recreated each time, and this causes a Windows-specific error in CI (Access is denied: '$topsrcdir\\_virtualenvs\\build\\Scripts\\python.exe').

Let's guard against this in a platform-specific way, then resolve the issue.

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1758584: Raise error if active site is out-of-date

Unless either the site's requirements or its on-disk virtualenv
changes, it shouldn't become out-of-date.
This will also catch situations where sites are incorrectly
being marked out-of-date due to sys.path mismanagement.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1758584: Add in-proc venv activation paths to the end of `sys.path`

So far, we've been using virtualenv's activate_this.py script.
However, unlike earlier expectations, it adds its sys.path additions
to the front, not the back! This breaks our prioritization
requirements, such as:

• When using any package from the system environment, import all
  possible packages from the system to avoid compatibility issues.
• Use vendored packages instead of virtualenv-installed packages
  wherever possible, because it more-closely matches developer
  expectations ("why is this package vendored if it's not used?")

Define an activate_virtualenv() function that replicates the logic
of activate_this.py [1], except for three differences:

• Don't modify sys.real_prefix, since it's a non-standard change
• Only add seen-with-venv-module paths to the sys.path ($prefix,
  $prefix/.../$site_packages_dir) - don't do the paths in-between.
• And, of course, append instead of prepend sys.path entries.

As an aside, this is one of the few remaining blockers from allowing
us to fully embrace venv instead of virtualenv - the last piece is
waiting on the fix for bug 1697833 to propagate.

[1]
https://github.com/pypa/virtualenv/blob/20.7.2/src/virtualenv/activation/python/activate_this.py

Depends on D140578

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1758584: Fix virtualenv-path scrubbing from command _pthfile_lines

We want _pthfile_lines() to consistently and accurately represent a
virtualenv's sys.path modifications.

However, this becomes tricky when comparing expected pthfile_lines
//before// activating a virtualenv versus //afterwards//, especially
since Python implicitly adds some paths (such as the path to
site-packages).

The current solution made this scrubbing only happen if we were
pip install-ing into the site. Unfortunately, it //doesn't// work for
the "use system Python for the build site", because:

• Pre-activation result: site-packages isn't added, because we aren't
  using it
• Post-activation result, implicitly-added site-packages isn't
  scrubbed, because scrubbing only happened for
  SitePackagesSource.VENV

Stepping back, the solution here is:

• pthfile_lines only represents Mach modifications //on top// of
  implicit virtualenv behaviour: since site-packages is always added
  by default, it shouldn't be in our explicit pthfile_lines.
• The only time when implicit sys.path additions throws us off is when
  SitePackagesSource.SYSTEM: so, move the scrubbing to happen in that
  case instead.

Finally refactor the "conditional deprioritization" comment to be more
useful and more accurate: we've implemented the "nontrivial complexity"
of purging site-packages, and the other piece about nothing being
pip install-ed feels self-explanatory enough.

Depends on D140579

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d0c262934162
Raise error if active site is out-of-date r=ahal
https://hg.mozilla.org/integration/autoland/rev/1337faa66745
Add in-proc venv activation paths to the end of `sys.path` r=ahal
https://hg.mozilla.org/integration/autoland/rev/1883182d614f
Fix virtualenv-path scrubbing from command _pthfile_lines r=ahal

Comment 5

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/d0c262934162
https://hg.mozilla.org/mozilla-central/rev/1337faa66745
https://hg.mozilla.org/mozilla-central/rev/1883182d614f