1656614 - Virtualenv `.pth` files don't get recreated on virtualenv "re-creation"

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Comment 1

[Ricky Stewart]

Attachment: Bug 1656614 - Delete existing `virtualenv` before re-creating it

I wrote this patch because I noticed that the .pth files in my objdir virtualenvs were extremely repetitive, containing multiple references to the same directories. This happens because we append to the in-virtualenv .pth files when calling populate(), but we don't ever clean up the old ones, meaning that whenever we determine that the virtualenvs are out of date and need to be recreated, we actually leave A LOT of state lying around on-disk that is going to go on to impact further uses of the virtualenv. Concretely, how this manifested is that when I erroneously removed an entry from virtualenv_packages.txt, the build actually succeeded because that entry was still in the .pth file in the virtualenv; instead of "creating" a new virtualenv with the correct .pth files, it just appended the new .pth data to the old, stale data.

I've chosen to address this by completely deleting the entire virtualenv when we try to re-create it. Another way you might solve this problem is by doing a find $VIRTUALENV -name '*.pth' | xargs rm before doing the virtualenv re-creation, but I'm suggesting we do it this way because we have had a long history of difficulty with virtualenv persistence. Bug 1628498 is an obvious example; note that we would never have encountered that bug if we always unconditionally deleted the virtualenv before creating a new one, as in this patch. A patch that is laser-targeted at handling the issue with .pth files might be fine for now but this is more foolproof and future-proof.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/951942b47734
Delete existing `virtualenv` before re-creating it r=mhentges,ahal

Comment 3

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/951942b47734