1622963 - Bump up minimum required Python 3 versions for builds from 3.5 to 3.6

Status: RESOLVED FIXED

(Firefox Build System :: General, task)

Description

[Ricky Stewart]

Our minimum required version of Python 3 has been 3.5 for a while, since at least bug 1451065, and 3.5 is a fairly old version of Python anyway so the bump doesn't seem especially aggressive. This comes on the heels of bug 1617748 where getting everything to work under Python 3 on all platforms has been troublesome, so consolidating our supported versions of Python as much as possible before officially making this transition can only make our lives easier in the short run.

Python 3.6 has a number of improvements/behavioral changes that should make a meaningful difference for us. The biggest pain point in the Python 3 migration by far has been difficulties with text encoding, especially in different environments including automation, and Python 3.6 is just better all-around in every respect with regard to text encoding, particularly on Windows (see PEP 528 and 529). The most important of these is the addition of the encoding argument to Popen.__init__(), which we used in the resolution to bug 1622021. (We could probably find a workaround that doesn't include setting encoding when calling into subprocesses, but this is the "correct" solution to the problem given our constraints and I don't see that trying to work around it so we can preserve Python 3.5 compatibility makes any sense.) All of this is in addition to inherently positive changes like perf improvements in hash tables and so on.

This will require making sure we install Python 3.6 for all builders in CI.

Comment 1

[Mike Hommey [:glandium]]

A couple data points about downstreams:

• Ubuntu 16.04, supported until April 2021, has 3.5.
• Debian 9, supported until June 2022, has 3.5.

Comment 2

[Ricky Stewart]

(In reply to Mike Hommey [:glandium] from comment #1)

A couple data points about downstreams:

• Ubuntu 16.04, supported until April 2021, has 3.5.
• Debian 9, supported until June 2022, has 3.5.

By "has 3.5" do you mean "has 3.5 installed by default"? My understanding has been that they can just do apt-get install python3.6 to get a more recent Python, so we shouldn't be blocked or anything.

Comment 3

[Mike Hommey [:glandium]]

I mean they have 3.5 and that's all. You can't apt-get install python3.6 unless you upgrade to the next LTS, so respectively Ubuntu 18.04 or Debian 10.

Comment 4

[James Graham [:jgraham]]

For web-platform-tests needing to support 3.5 is proving to be somewhat problematic:

• We don't currently have any CI set up for this configuration. This means we have been relying on some changes in 3.6 in our code. Although we obviously can set up some it's imposing additional burden on the project and is only providing value to Gecko; everyone else is targeting more modern versions. There is particular skepticism about supporting Python 3.5 past its EOL in September 2020.
• Some dependencies we have don't work with Python < 3.6. None of these are currently used in Gecko CI, but that may change (e.g. we may want to start running the QUIC server).

So I'm pretty worried that whilst we're still using 3.5 for gecko there's likely to be breakage that is going to require me to do an emergency fix in order to unblock the sync. As far as I can tell we're using two principal parts of the code in 3.5:

• The manifest generation, used by the TestResolver
• The actual test execution, in jsshell

Of these the first part is easier to support; standing up the manifest unittests in 3.5 is relatively low effort (although will involve e.g. updating the docker image to actually include Python 3.5), only touches a subset of the code, and doesn't have large ongoing costs. The second part is harder to support and doing it well requires at least infrastructure tests to be run under 3.5.

So from our point of view, it would save a great deal of trouble to get this work prioritised. In particular any changes to move the js tests off 3.5 would be extremely welcome, since that removes the biggest headache in terms of ongoing support.

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1622963 - Install python 3.6 in build docker images.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1622963 - Require python 3.6 for the build system.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2931388675f0
Install python 3.6 in build docker images. CLOSED TREE

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/83ee79caab92
Require python 3.6 for the build system. r=firefox-build-system-reviewers,rstewart

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out changeset 83ee79caab92 (bug 1622963) for python related bustages

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=83ee79caab927e6b4ff13c9ed55fd2d8d93ed133&searchStr=build&tochange=0f765a8fefd4f14c333517890015732e4cdbc068&selectedTaskRun=Vz3JBUGVSCCGITScbPDUOA-0

Backout link: https://hg.mozilla.org/integration/autoland/rev/0f765a8fefd4f14c333517890015732e4cdbc068

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=304871951&repo=autoland&lineNumber=1200

[task 2020-06-03T07:20:24.837Z] 07:20:24     INFO -  /builds/worker/checkouts/gecko/configure
[task 2020-06-03T07:20:25.098Z] 07:20:25     INFO -  ERROR: One of the following Python versions are required to build:
[task 2020-06-03T07:20:25.099Z] 07:20:25    ERROR -  Traceback (most recent call last):
[task 2020-06-03T07:20:25.099Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/configure.py", line 181, in <module>
[task 2020-06-03T07:20:25.099Z] 07:20:25     INFO -      sys.exit(main(sys.argv))
[task 2020-06-03T07:20:25.099Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/configure.py", line 52, in main
[task 2020-06-03T07:20:25.099Z] 07:20:25     INFO -      sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
[task 2020-06-03T07:20:25.099Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/configure/__init__.py", line 445, in run
[task 2020-06-03T07:20:25.099Z] 07:20:25     INFO -      self.include_file(path)
[task 2020-06-03T07:20:25.099Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/configure/__init__.py", line 436, in include_file
[task 2020-06-03T07:20:25.099Z] 07:20:25     INFO -      exec_(code, self)
[task 2020-06-03T07:20:25.099Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/util.py", line 53, in exec_
[task 2020-06-03T07:20:25.099Z] 07:20:25     INFO -      exec(object, globals, locals)
[task 2020-06-03T07:20:25.099Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/moz.configure", line 7, in <module>
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -      include('build/moz.configure/init.configure')
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/configure/__init__.py", line 773, in include_impl
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -      self.include_file(what)
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/configure/__init__.py", line 436, in include_file
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -      exec_(code, self)
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/util.py", line 53, in exec_
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -      exec(object, globals, locals)
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/build/moz.configure/init.configure", line 253, in <module>
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -      @imports(_from='six', _import='ensure_text')
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/configure/__init__.py", line 756, in decorator
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -      depends = DependsFunction(self, func, dependencies, when=when)
[task 2020-06-03T07:20:25.100Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/configure/__init__.py", line 129, in __init__
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -      sandbox._value_for(self)
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/configure/__init__.py", line 542, in _value_for
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -      return self._value_for_depends(obj)
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/util.py", line 1018, in method_call
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -      cache[args] = self.func(instance, *args)
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/configure/__init__.py", line 551, in _value_for_depends
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -      value = obj.result()
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/util.py", line 1018, in method_call
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -      cache[args] = self.func(instance, *args)
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/configure/__init__.py", line 155, in result
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -      return self._func(*resolved_args)
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/configure/__init__.py", line 1161, in wrapped
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -      return new_func(*args, **kwargs)
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/build/moz.configure/init.configure", line 292, in virtualenv_python3
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -      verify_python_version(out)
[task 2020-06-03T07:20:25.101Z] 07:20:25     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 694, in verify_python_version
[task 2020-06-03T07:20:25.102Z] 07:20:25     INFO -      for minver in MINIMUM_PYTHON_VERSIONS.values():
[task 2020-06-03T07:20:25.102Z] 07:20:25    ERROR -  NameError: name 'MINIMUM_PYTHON_VERSIONS' is not defined
[task 2020-06-03T07:20:25.109Z] 07:20:25     INFO -  *** Fix above errors and then restart with\
[task 2020-06-03T07:20:25.109Z] 07:20:25     INFO -                 "./mach build"
[task 2020-06-03T07:20:25.109Z] 07:20:25     INFO -  client.mk:111: recipe for target 'configure' failed
[task 2020-06-03T07:20:25.109Z] 07:20:25     INFO -  make: *** [configure] Error 1
[task 2020-06-03T07:20:25.180Z] 07:20:25    ERROR - Return code: 2
[task 2020-06-03T07:20:25.180Z] 07:20:25  WARNING - setting return code to 2
[task 2020-06-03T07:20:25.180Z] 07:20:25    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2020-06-03T07:20:25.180Z] 07:20:25    FATAL - Running post_fatal callback...
[task 2020-06-03T07:20:25.180Z] 07:20:25    FATAL - Exiting -1
[task 2020-06-03T07:20:25.180Z] 07:20:25     INFO - [mozharness: 2020-06-03 07:20:25.180913Z] Finished build step (failed)

Comment 10

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/2931388675f0

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/f3649e01726f
Port bug 1622963 - Install python 3.6 in build docker images. rs=bustage-fix

Comment 12

[Mike Hommey [:glandium]]

So, the Debian stretch-based docker images are still on python 3.5, which is why the second half didn't stick. Installing python3.6 is not totally straightforward, because it has an impact on other docker images that try to install other python packages, which we'd probably need to have to rebuild against python3.6, and that doesn't sound really great. It's probably going to be easier to update those images to Debian buster, and if we're going to do that, I feels like we might as well go all in and update all the Debian stretch images to buster. As a matter of fact, I already have WIP patches for that. I'll file a separate bug for this.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6700a8dfaeb1
Require python 3.6 for the build system. r=firefox-build-system-reviewers,rstewart

Comment 14

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/6700a8dfaeb1