1210157 - bootstrap.py raises syntax error on Python 3

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[altblitz]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0
Build ID: 20150921073455

Steps to reproduce:

http://areweeveryoneyet.org/onramp/desktop-linux.html

'If you don’t have one yet, create a "src" directory for yourself under your home directory: cd ~ && mkdir src' - done.
'Next download the bootstrap.py script and save it in your src/ directory.' - done
'And finally, in your terminal: cd ~/src and start the bootstrapper like this:
python bootstrap.py' - failed

[foo@ba ~/src ] uname  -a
Linux inode 4.2.1-1-ARCH #1 SMP PREEMPT Tue Sep 22 06:57:07 CEST 2015 x86_64 GNU/Linux
[foo@ba ~/src ] python --version
Python 3.4.3




Actual results:

[foo@bar ~/src ] python bootstrap.py  
Could not load the bootstrap Python environment.
This should never happen. Consider filing a bug.
Missing parentheses in call to 'print' (archlinux.py, line 67)



Expected results:

'... follow the prompts.'

Comment 1

[altblitz]

In Arch python bootstrap.py should be python2 bootstrap.py

Your version of Mercurial (3.5.1) is sufficiently modern.
Your version of Python (2.7.10) is new enough.

Your system should be ready to build Firefox for Desktop! If you have not already,
obtain a copy of the source code by running:
    hg clone https://hg.mozilla.org/mozilla-central
Or, if you prefer Git:
    git clone https://git.mozilla.org/integration/gecko-dev.git

python2 bootstrap.py  6.71s user 1.21s system 12% cpu 1:03.48 total

Comment 2

[Gregory Szorc [:gps]]

Yeah, this is a Python 3 compatibility problem. We need to make the bootstrapper Python 3 compatible as a lot of distros are shipping Python 3 by default now.

We should at least detect Python 3 and raise an actionable error until we have full Python 3 support.

Comment 3

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

(In reply to Gregory Szorc [:gps] from comment #2)
> Yeah, this is a Python 3 compatibility problem. We need to make the
> bootstrapper Python 3 compatible as a lot of distros are shipping Python 3
> by default now.
> 
> We should at least detect Python 3 and raise an actionable error until we
> have full Python 3 support.

This was done in bug 1257095.

Comment 4

[Māris Fogels [:mars] (please needinfo)]

Attachment: Bug 1210157 - Make mozboot call mach artifact with default Python

mozboot calls out to 'mach artifact' to install dependencies on the host. Make
mozboot use the default interpreter for 'mach artifact' (Python 2 for now)
instead of using the interpreter that mozboot was called with.

This allows us to port the mozboot code to Python 3 without having to port
everything 'mach artifact' pulls in as well (which is a lot).

Comment 5

[Māris Fogels [:mars] (please needinfo)]

Attachment: Bug 1210157 - Make Linux dev environment bootstrap support Python 3

Make the mozboot base modules and Debian dev environment
bootstrap routines compatible with both Python 2.7 and Python 3.6+.

• Remove Python 2.6 support from bootstrap.py and mozboot.
• Mark minimum versions for bootstrap.py as Python 2.7 and Python 3.6+.
• Make bootstrap.py support Python 3.
• Make mozboot.base and mozboot.bootstrap support Python 3.
• Make the Debian-based Linux distro bootstrap routines support
  Python 3.
• Make "mach bootstrap" use the Python 3 interpreter.
• Use unicode_literals in all mozboot modules.

Depends on D39080

Comment 6

[Māris Fogels [:mars] (please needinfo)]

Attachment: Bug 1210157 - Use unicode_literals in all mozboot modules

Add unicode_literals to all mozboot module future statements to support
running the modules under Python 3. Remove comments about unicode_literals and
Python 2.6 support as Python 2.6 is no longer supported in tree.

Comment 7

[Māris Fogels [:mars] (please needinfo)]

Attachment: Bug 1210157 - Add Python 3 support to mozboot.base and mozboot.bootstrap

Add support for both Python 3 and Python 2.7 to the mozboot.base and
mozboot.bootstrap modules. Remove legacy Python 2.6 code or mark it for later
removal.

Depends on D39359

Comment 8

[Māris Fogels [:mars] (please needinfo)]

Attachment: Bug 1210157 - Add Python 3 support to the mozboot Debian bootstrapper

Add support for Python 3 and Python 2.7 to the Debian-based linux distro
bootstrap routines.

Depends on D39360

Comment 9

[Māris Fogels [:mars] (please needinfo)]

Attachment: Bug 1210157 - Add Python 3 support to mozboot Android toolchain bootstrap

Support installing the android development toolchain with Python 3 as well as
Python 2.7.

Depends on D39361

Comment 10

[Māris Fogels [:mars] (please needinfo)]

Attachment: Bug 1210157 - Add a --debug option to bootstrap.py

Add an option to run bootstrap.py with '--debug'. This will print full
tracebacks if the bootstrap process encounters an uncaught exception. It should
be useful for developers who are working on the bootstrap code as well as users
when writing bug reports.

Depends on D39362

Comment 11

[Māris Fogels [:mars] (please needinfo)]

Attachment: Bug 1210157 - Make bootstrap.py work with Python 3.5+ and Python 2.7

Make bootstrap.py compatible with both Python 3.6+ and Python 2.7. Remove
Python 2.6 support as Python 2.6 is no longer supported in the Firefox source
tree.

Depends on D39363

Comment 12

[Māris Fogels [:mars] (please needinfo)]

Attachment: Bug 1210157 - Fix Python 3 issues in mozboot.gentoo and mozboot.archlinux

Fix some library calls and syntax that are required to support Python 3.

Depends on D39364

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f044d6116869
Use unicode_literals in all mozboot modules r=firefox-build-system-reviewers,mshal
https://hg.mozilla.org/integration/autoland/rev/5d26b3d7d4a0
Add Python 3 support to mozboot.base and mozboot.bootstrap r=firefox-build-system-reviewers,mshal
https://hg.mozilla.org/integration/autoland/rev/25aef37f3556
Add Python 3 support to the mozboot Debian bootstrapper r=firefox-build-system-reviewers,mshal
https://hg.mozilla.org/integration/autoland/rev/a65032b442bc
Add Python 3 support to mozboot Android toolchain bootstrap r=nalexander
https://hg.mozilla.org/integration/autoland/rev/0f5311bbd75b
Add a --debug option to bootstrap.py r=nalexander
https://hg.mozilla.org/integration/autoland/rev/06d452ac1d16
Make bootstrap.py work with Python 3.5+ and Python 2.7 r=firefox-build-system-reviewers,mshal
https://hg.mozilla.org/integration/autoland/rev/eec9458e553d
Fix Python 3 issues in mozboot.gentoo and mozboot.archlinux r=firefox-build-system-reviewers,mshal

Comment 14

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out 8 changesets (Bug 1569046, Bug 1210157) for flake8 failure on base.py

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linting%2Copt%2Csource-test-mozlint-py-flake8%2C%28f8%29&fromchange=a59737bc6d819c1a9da579977c1afe6145964690&tochange=3c49b0c27b1558011f0927f6169fa522b2117c04

Backout link: https://hg.mozilla.org/integration/autoland/rev/3c49b0c27b1558011f0927f6169fa522b2117c04

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=262549430&repo=autoland&lineNumber=288
[task 2019-08-20T21:03:03.028Z] building 'psutil._psutil_posix' extension
[task 2019-08-20T21:03:03.028Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-08-20T21:03:03.028Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-08-20T21:03:03.028Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2019-08-20T21:03:03.028Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2019-08-20T21:03:03.028Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2019-08-20T21:03:03.029Z]
[task 2019-08-20T21:03:03.029Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2019-08-20T21:06:15.592Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/python/mozboot/mozboot/base.py:27:1 | too many blank lines (3) (E303)
[taskcluster 2019-08-20 21:06:16.128Z] === Task Finished ===
[taskcluster 2019-08-20 21:06:17.135Z] Unsuccessful task run with exit code: 1 completed in 462.916 seconds

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fd1ce26da310
Use unicode_literals in all mozboot modules r=firefox-build-system-reviewers,mshal
https://hg.mozilla.org/integration/autoland/rev/41bebeeebc27
Add Python 3 support to mozboot.base and mozboot.bootstrap r=firefox-build-system-reviewers,mshal
https://hg.mozilla.org/integration/autoland/rev/6a5f5594422f
Add Python 3 support to the mozboot Debian bootstrapper r=firefox-build-system-reviewers,mshal
https://hg.mozilla.org/integration/autoland/rev/d79183b963e7
Add Python 3 support to mozboot Android toolchain bootstrap r=nalexander
https://hg.mozilla.org/integration/autoland/rev/7c2b0c902854
Add a --debug option to bootstrap.py r=nalexander
https://hg.mozilla.org/integration/autoland/rev/f0f92aecee18
Make bootstrap.py work with Python 3.5+ and Python 2.7 r=firefox-build-system-reviewers,mshal
https://hg.mozilla.org/integration/autoland/rev/d00d447a0c29
Fix Python 3 issues in mozboot.gentoo and mozboot.archlinux r=firefox-build-system-reviewers,mshal

Comment 16

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/fd1ce26da310
https://hg.mozilla.org/mozilla-central/rev/41bebeeebc27
https://hg.mozilla.org/mozilla-central/rev/6a5f5594422f
https://hg.mozilla.org/mozilla-central/rev/d79183b963e7
https://hg.mozilla.org/mozilla-central/rev/7c2b0c902854
https://hg.mozilla.org/mozilla-central/rev/f0f92aecee18
https://hg.mozilla.org/mozilla-central/rev/d00d447a0c29