Closed Bug 1627163 Opened 1 year ago Closed 1 year ago

Switch python configure to python 3

Categories

(Firefox Build System :: General, task)

task
Not set
normal

Tracking

(firefox77 fixed)

RESOLVED FIXED
mozilla77
Tracking Status
firefox77 --- fixed

People

(Reporter: glandium, Assigned: glandium)

References

(Regressed 1 open bug)

Details

Attachments

(16 files)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
No description provided.

This was cargo-culted from virtualenv_python2, but this is not
necessary: currently, the only way configure code will execute with
python 3 is if something uses the configure sandbox from python 3, not
configure. Only configure itself is meant to re-execute itself with
python, so we don't want python 3 to re-execute.

We'll be swapping virtualenv_python2 and virtualenv_python3 later, when
switching to python 3.

It was cargo-culted from virtualenv_python2, but is not necessary
because virtualenv_python3 is not going to re-execute python, which is
what the dependency is there to avoid.

This is yet another cargo-cult from virtualenv_python2, which is not
necessary because by the time we reach virtualenv_python3, the mozconfig
configure has been injected already.

Using @checking doesn't go well with the virtualenv creation, so print
the version information after the fact. And do so uniformely for python
2 and python 3.

Also remove unused import of itertools.ifilterfalse.

It's in both in python 2, but only in the former in python 3.

ConfigureSandbox._implied_options is a list of ReadOnlyNamespaces.
In python 3.5, ReadOnlyNamespaces end up with no guarantee in the order
of their dict. So when comparing

ReadOnlyNamespace(a=1, b=2)

and

ReadOnlyNamespace(a=3, b=4)

It's not guaranteed that the a's are compared before the b's.

In ConfigureSandbox._implied_options, some of those ReadOnlyNamespace
fields are SandboxDependsFunctions, which actually raise an error when
they're being compared, because we don't want that to happen in the
sandbox.

So when using python 3, configure would randomly fail when trying to
remove items from the ConfigureSandbox._implied_options list because
removing an item from a list scans the list to find the first element
that matches.

And ConfigureSandbox._implied_options needs to be ordered, which is why
it's currently a list.

There is actually something better we can do instead of removing by
value: we can actually remove by index, assuming we adjust for the
number of elements we already removed during the loop.

Python 3 doesn't like that.

Some things remain to iron out, but this is presumably green everywhere.

I appreciate the removal of unnecessary cargo-culted stuff.

Blocks: 1617748

The title of the bug and the contents of some of the patches seem to indicate we want this to block bug 1617748, so I made that change. (I have been testing the patch for bug 1617748 for WEEKS, and I'm a little concerned about why none of this stuff -- like the iteritems() issues in configure -- appeared in try pushes before.)

If I'm not understanding this correctly and it's unrelated to bug 1617748 though, feel free to set me straight though.

I think the problem in bug 1617748 is that switching mach build and mach configure themselves is almost a no-op, because they re-execute something else (respectively make -f client.mk and make -f client.mk configure), and that, in turn, returns to python2. However, mach build-backend executes config.status with the python that is used to call it, so bug 1617748 is effectively switching running a python2-generated config.status with python3, but mach build-backend is also not something that runs all the time.

This bug changes make -f client.mk configure to end up using python3. You'll see the last patch also switches mach build-backend as a consequence.

Keywords: leave-open
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/50d99e9fdae6
Do not re-execute configure with python 3. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/e075af33170e
Remove the --help dependency on virtualenv_python3. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/7a25c7487ffa
Remove mozconfig dependency from virtualenv_python3. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/06ef786c150d
Unify how python version is shown for py2 and py3. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/31b7ae9a38da
Don't use binary mode to read/write json files in mozwebidlcodegen. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/77cb1d547938
Use generator expressions instead of itertools.imap. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/6c0a9c89b833
Use POINTER from ctypes rather than wintypes. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/592ef8e830ad
Fix a couple python 3 incompatibilities in old.configure. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/542f439cf528
Add support for python 3 to mozbuild.configure.libstdcxx. r=firefox-build-system-reviewers,rstewart

Many values we get out from configure are of types that look like
lists/tuples, that write_indented_repr will serialize as lists... but
only in python 2, because the alternative implementation for python 3
is not doing that. So sanitize first.

Attachment #9138007 - Attachment description: Bug 1627163 - WIP → Bug 1627163 - Switch python configure to python 3.
Keywords: leave-open
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/46cc9d8517dd
Avoid scanning ConfigureSandbox._implied_options when handling implied options. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/d1913f539e6b
Add comments to virtualenv_python2 that can be seen in virtualenv_python3. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/b8987765a42d
Avoid a comparison between a str and an int in gyp. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/cb049a611bf7
Fix a few more python 3 incompatibilities in some .configure files. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/832c792b08f4
Do not rely on write_indented_repr serializing list-like items as lists. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/d3eb1d4ffd63
Fix a few python 3 issues in mozbuild.backend.visualstudio. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/407894bc5f9c
Switch python configure to python 3. r=firefox-build-system-reviewers,rstewart
Regressed by: 1628519
No longer regressed by: 1628519
Regressions: 1628519
Regressions: 1628726
Regressions: 1628748
Regressions: 1628498
Regressions: 1631633
Regressions: 1635481
Regressions: 1661624
You need to log in before you can comment on or make changes to this bug.