1627163 - Switch python configure to python 3

Status: RESOLVED FIXED

(Firefox Build System :: General, task)

Comment 1

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Do not re-execute configure with python 3.

This was cargo-culted from virtualenv_python2, but this is not
necessary: currently, the only way configure code will execute with
python 3 is if something uses the configure sandbox from python 3, not
configure. Only configure itself is meant to re-execute itself with
python, so we don't want python 3 to re-execute.

We'll be swapping virtualenv_python2 and virtualenv_python3 later, when
switching to python 3.

Comment 2

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Remove the --help dependency on virtualenv_python3.

It was cargo-culted from virtualenv_python2, but is not necessary
because virtualenv_python3 is not going to re-execute python, which is
what the dependency is there to avoid.

Comment 3

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Remove mozconfig dependency from virtualenv_python3.

This is yet another cargo-cult from virtualenv_python2, which is not
necessary because by the time we reach virtualenv_python3, the mozconfig
configure has been injected already.

Comment 4

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Unify how python version is shown for py2 and py3.

Using @checking doesn't go well with the virtualenv creation, so print
the version information after the fact. And do so uniformely for python
2 and python 3.

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Don't use binary mode to read/write json files in mozwebidlcodegen.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Use generator expressions instead of itertools.imap.

Also remove unused import of itertools.ifilterfalse.

Comment 7

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Use POINTER from ctypes rather than wintypes.

It's in both in python 2, but only in the former in python 3.

Comment 8

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Fix a couple python 3 incompatibilities in old.configure.

Comment 9

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Add support for python 3 to mozbuild.configure.libstdcxx.

Comment 10

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Avoid scanning ConfigureSandbox._implied_options when handling implied options.

ConfigureSandbox._implied_options is a list of ReadOnlyNamespaces.
In python 3.5, ReadOnlyNamespaces end up with no guarantee in the order
of their dict. So when comparing

ReadOnlyNamespace(a=1, b=2)

and

ReadOnlyNamespace(a=3, b=4)

It's not guaranteed that the a's are compared before the b's.

In ConfigureSandbox._implied_options, some of those ReadOnlyNamespace
fields are SandboxDependsFunctions, which actually raise an error when
they're being compared, because we don't want that to happen in the
sandbox.

So when using python 3, configure would randomly fail when trying to
remove items from the ConfigureSandbox._implied_options list because
removing an item from a list scans the list to find the first element
that matches.

And ConfigureSandbox._implied_options needs to be ordered, which is why
it's currently a list.

There is actually something better we can do instead of removing by
value: we can actually remove by index, assuming we adjust for the
number of elements we already removed during the loop.

Comment 11

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Add comments to virtualenv_python2 that can be seen in virtualenv_python3.

Comment 12

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Avoid a comparison between a str and an int in gyp.

Python 3 doesn't like that.

Comment 13

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Switch python configure to python 3.

Some things remain to iron out, but this is presumably green everywhere.

Comment 14

[Ricky Stewart]

I appreciate the removal of unnecessary cargo-culted stuff.

Comment 15

[Ricky Stewart]

The title of the bug and the contents of some of the patches seem to indicate we want this to block bug 1617748, so I made that change. (I have been testing the patch for bug 1617748 for WEEKS, and I'm a little concerned about why none of this stuff -- like the iteritems() issues in configure -- appeared in try pushes before.)

Comment 16

[Ricky Stewart]

If I'm not understanding this correctly and it's unrelated to bug 1617748 though, feel free to set me straight though.

Comment 17

[Mike Hommey [:glandium]]

I think the problem in bug 1617748 is that switching mach build and mach configure themselves is almost a no-op, because they re-execute something else (respectively make -f client.mk and make -f client.mk configure), and that, in turn, returns to python2. However, mach build-backend executes config.status with the python that is used to call it, so bug 1617748 is effectively switching running a python2-generated config.status with python3, but mach build-backend is also not something that runs all the time.

This bug changes make -f client.mk configure to end up using python3. You'll see the last patch also switches mach build-backend as a consequence.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/50d99e9fdae6
Do not re-execute configure with python 3. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/e075af33170e
Remove the --help dependency on virtualenv_python3. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/7a25c7487ffa
Remove mozconfig dependency from virtualenv_python3. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/06ef786c150d
Unify how python version is shown for py2 and py3. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/31b7ae9a38da
Don't use binary mode to read/write json files in mozwebidlcodegen. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/77cb1d547938
Use generator expressions instead of itertools.imap. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/6c0a9c89b833
Use POINTER from ctypes rather than wintypes. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/592ef8e830ad
Fix a couple python 3 incompatibilities in old.configure. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/542f439cf528
Add support for python 3 to mozbuild.configure.libstdcxx. r=firefox-build-system-reviewers,rstewart

Comment 19

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Fix a few more python 3 incompatibilities in some .configure files.

Comment 20

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Do not rely on write_indented_repr serializing list-like items as lists.

Many values we get out from configure are of types that look like
lists/tuples, that write_indented_repr will serialize as lists... but
only in python 2, because the alternative implementation for python 3
is not doing that. So sanitize first.

Comment 21

[Mike Hommey [:glandium]]

Attachment: Bug 1627163 - Fix a few python 3 issues in mozbuild.backend.visualstudio.

Comment 22

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/50d99e9fdae6
https://hg.mozilla.org/mozilla-central/rev/e075af33170e
https://hg.mozilla.org/mozilla-central/rev/7a25c7487ffa
https://hg.mozilla.org/mozilla-central/rev/06ef786c150d
https://hg.mozilla.org/mozilla-central/rev/31b7ae9a38da
https://hg.mozilla.org/mozilla-central/rev/77cb1d547938
https://hg.mozilla.org/mozilla-central/rev/6c0a9c89b833
https://hg.mozilla.org/mozilla-central/rev/592ef8e830ad
https://hg.mozilla.org/mozilla-central/rev/542f439cf528

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/46cc9d8517dd
Avoid scanning ConfigureSandbox._implied_options when handling implied options. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/d1913f539e6b
Add comments to virtualenv_python2 that can be seen in virtualenv_python3. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/b8987765a42d
Avoid a comparison between a str and an int in gyp. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/cb049a611bf7
Fix a few more python 3 incompatibilities in some .configure files. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/832c792b08f4
Do not rely on write_indented_repr serializing list-like items as lists. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/d3eb1d4ffd63
Fix a few python 3 issues in mozbuild.backend.visualstudio. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/407894bc5f9c
Switch python configure to python 3. r=firefox-build-system-reviewers,rstewart

Comment 24

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/46cc9d8517dd
https://hg.mozilla.org/mozilla-central/rev/d1913f539e6b
https://hg.mozilla.org/mozilla-central/rev/b8987765a42d
https://hg.mozilla.org/mozilla-central/rev/cb049a611bf7
https://hg.mozilla.org/mozilla-central/rev/832c792b08f4
https://hg.mozilla.org/mozilla-central/rev/d3eb1d4ffd63
https://hg.mozilla.org/mozilla-central/rev/407894bc5f9c