1590745 - [mozbuild] Make the $PYTHON3 build var use a venv instead of a raw python binary

Status: RESOLVED FIXED

(Firefox Build System :: General, task)

Description

[Māris Fogels [:mars] (please needinfo)]

To port the build process and all of it's related scripts to Python 3 the $PYTHON3 build var should point to a Python binary inside a virtualenv just like the $PYTHON build var does.

Once this is done build tasks that rely on libraries installed in the Python virtualenv can be switched to Python 3 by configuring them to use $PYTHON3 in instead of $PYTHON.

See virtualenv_python() in build/moz.configure/init.configure for how the $PYTHON build var and its virtualenv are created.

See python3() in build/moz.configure/init.configure for how the $PYTHON3 build var is set. It points to the raw python binary and carries no virtualenv with it.

Comment 1

[Māris Fogels [:mars] (please needinfo)]

Attachment: Bug 1590745 - Make the $PYTHON3 build var use a virtualenv r?mshal

Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.

The new virtualenv is given a name unique to the build process and
unique to the Python version so future upgrades of the build process
interpreter are decoupled from upgrades to the Python used to run mach.

This patch also adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3034ec8681ac
Make the $PYTHON3 build var use a virtualenv r=mshal

Comment 3

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/3034ec8681ac

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out from central: https://hg.mozilla.org/mozilla-central/rev/d1001fea6e4c66b98bb4983df49c6e47d2db5ceb

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout from beta: https://hg.mozilla.org/releases/mozilla-beta/rev/a5748e4de94bcc3a88008bc8c1d0bda23f31a3dc

Comment 6

[Māris Fogels [:mars] (please needinfo)]

Landing this is blocked until we fix bug 1464038.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a439caeaa0e1
Make the $PYTHON3 build var use a virtualenv r=mshal

Comment 8

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/a439caeaa0e1