1451106 - Install Python 3.6 on macOS machines

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Gregory Szorc [:gps]]

Per bug 1388447, we want to make Python 3.5+ a build (and test) requirement for Firefox.

We already have Python 3.5 installed in Linux Docker images. So that platform is (mostly) taken care of.

This bug tracks installing Python 3.5+ on *all* macOS machines.

I'm requesting the install of Python 3.6, preferably the latest released version (3.6.5 at the time I write this), not Python 3.5, because we should be using the latest/greatest Python release.

Because macOS machines are test only and we don't yet have any firm plans to require Python 3.5+ on the test side of things, this is not a blocker to requiring Python 3.5+ on the build side. But it is a blocker to whoever wants to be the first person to use Python 3.5+ for tests.

Comment 1

[Dragos Crisan [:dragrom]]

created and added python36-3.6.5-1.dmg and patch

Comment 2

[Dragos Crisan [:dragrom]]

Attachment: Bug_1451106_Install_Python36_on_macOS_machines.patch

- created a module, called python36.pp, to install python 3.6 on OSX workers
- created a shell script, called python36-dmg.sh, to generate dmg package

Comment 3

[Jake Watkins [:dividehex]]

Comment on attachment 8968205 [details] [diff] [review]
Bug_1451106_Install_Python36_on_macOS_machines.patch

This looks good to me but I'd like to have Aki peek at it also since I think he was the last to roll the python3.5 packages

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 8968205 [details] [diff] [review]
Bug_1451106_Install_Python36_on_macOS_machines.patch

Thanks!

- we may want a /usr/local/bin/python3
- we may want to make sure the /usr/local/bin/python link doesn't cause bustage, from python2 scripts picking up the python3 path
- we probably want to roll this out to a canary portion of the pool and see if anything unexpected breaks
- we will probably be adding `pip.conf` to our releng infra. we likely will want to do the same here. Sadly, (aiui) older python2.7 (e.g. py 2.7.3) doesn't support newer tls and newer python>=2.7.[9?] doesn't support http, so we're going to need to be careful about which urls we publish in the pip.conf until we have new python2 and python3 rolled out everywhere.

Comment 5

[Dragos Crisan [:dragrom]]

Attachment: Bug_1451106_add_python36_module.patch

Puppet module to configure python 3.6 environment

Comment 6

[:dhouse]

Comment on attachment 8972861 [details] [diff] [review]
Bug_1451106_add_python36_module.patch

Review of attachment 8972861 [details] [diff] [review]:
-----------------------------------------------------------------

r+ except I don't know about the windows pieces.

Comment 7

[Aki Sasaki (not active)]

What's the timing on this? There's bug 1450037 where I'm renaming python35 -> python3, and updating it to 3.6.5. That's specifically targeting the existing centos hosts though.

Comment 8

[Dragos Crisan [:dragrom]]

I'll put this on hold until you will finish  bug 1450037.

Comment 9

[Aki Sasaki (not active)]

Thanks! I'll try to wrap that up this week.

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8968205 [details] [diff] [review]
Bug_1451106_Install_Python36_on_macOS_machines.patch

Review of attachment 8968205 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/packages/manifests/mozilla/python36.spec
@@ +56,5 @@
> +# add /usr/local/bin links
> +mkdir -p $RPM_BUILD_ROOT/usr/local/bin
> +set -x
> +ln -s %{_prefix}/bin/python%{pyver} $RPM_BUILD_ROOT/usr/local/bin/python%{pyver}
> +ln -s %{_prefix}/bin/python%{pyver} $RPM_BUILD_ROOT%{_prefix}/bin/python

Assuming this creates `/usr/local/bin/python`, then this is somewhat wrong because it is against the recommendations of PEP 394 (https://www.python.org/dev/peps/pep-0394/), which say that `python` should refer to Python 2.

The PEP reflects real world concerns, where many Python scripts use `#!/usr/bin/env python`. You are supposed to use `python2` to refer to the latest Python 2 and `python3` to refer to the latest Python 3. If you want a specific Python, you use `pythonX.Y`.

A Python 3 install should alias `python3` and `python3.Y`. That also means this patch is wrong because it only aliases `python3.6` and not `python3`.

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 8972861 [details] [diff] [review]
Bug_1451106_add_python36_module.patch

Clearing review, since this patch will likely change after bug 1450037 rolls out.

Comment 12

[Dragos Crisan [:dragrom]]

Comment on attachment 8972861 [details] [diff] [review]
Bug_1451106_add_python36_module.patch

Removed this review request, need to do other changes based on 1450037 aki's work

Comment 13

[Dragos Crisan [:dragrom]]

Attachment: Bug_1451106_Install_Python36_on_macOS_machines.patch

Install python 3.6.5 on Mac OS yosemite workers

Comment 14

[Aki Sasaki (not active)]

Comment on attachment 8975452 [details] [diff] [review]
Bug_1451106_Install_Python36_on_macOS_machines.patch

Thanks!

We're going to try to update our python version regularly [1] in our releng infra. The way this is defined looks like it allows us to update osx simultaneously or at its own cadence, which is a nice feature :)

[1] https://wiki.mozilla.org/ReleaseEngineering/Python_Standards#Supported_versions

Comment 15

[Jake Watkins [:dividehex]]

Comment on attachment 8975452 [details] [diff] [review]
Bug_1451106_Install_Python36_on_macOS_machines.patch

Review of attachment 8975452 [details] [diff] [review]:
-----------------------------------------------------------------

Fix lint error, otherwise it looks great!

::: modules/packages/manifests/mozilla/python3.pp
@@ +51,5 @@
> +                default: {
> +                    fail("Cannot install on Darwin version ${::macosx_productversion_major}")
> +                }
> +            }
> +}

human lint: This closing curly bracket needs to be properly space indented

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/build/puppet/rev/6cb868b2fc5c
Install Python 3.6 on macOS machines, r=jwatkins, aki

Comment 17

[Dragos Crisan [:dragrom]]

Comment on attachment 8975452 [details] [diff] [review]
Bug_1451106_Install_Python36_on_macOS_machines.patch

Pushed to default: https://hg.mozilla.org/build/puppet/rev/6cb868b2fc5c7839bef11726be4fc03885659025
Pushed to production: https://hg.mozilla.org/build/puppet/rev/531f2665b93d6102bd87738d8a79661592bff947