1466535 - Install python3 on OSX generic-worker hosts

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Andrew Halberstadt [:ahal]]

+++ This bug was initially created as a clone of Bug #1436767 +++

I'm trying to get run-task [1] based tasks working with taskcluster's generic-worker. But the OSX hosts don't have python3 installed.

We'll need at least python 3.5+, though I assume 3.6 would be better.

[1] https://searchfox.org/mozilla-central/source/taskcluster/docker/recipes/run-task

Comment 1

[Kendall Libby [:fubar] (he/him)]

Python 3.6 was installed on our OSX machines in bug 1451106.

Comment 2

[Dragos Crisan [:dragrom]]

[[e-mail] ~]# which python3.6
/usr/local/bin/python3.6

Comment 3

[Andrew Halberstadt [:ahal]]

I don't have a loaner so have been try debugging, but here `which python3` returns 1:
https://taskcluster-artifacts.net/dFL_PhP4QwmcRy0SuT1CzQ/0/public/logs/live_backing.log

And in this push, `ls /usr/local/bin` has no python3 in it:
https://treeherder.mozilla.org/logviewer.html#?job_id=181718941&repo=try

Is it possible this is a different pool?

Comment 4

[Dragos Crisan [:dragrom]]

ahal: can you try please pyhon3.6?

Comment 5

[Andrew Halberstadt [:ahal]]

https://taskcluster-artifacts.net/bAXi6dzCRduA0qfqpw8lGQ/0/public/logs/live_backing.log

That push runs:
- ls /tools
- ls /usr/local/bin
- which python3.6

All three show no sign of python3.

Comment 6

[Andrew Halberstadt [:ahal]]

(In reply to Dragos Crisan [:dragrom] from comment #2)
> [[e-mail] ~]# which python3.6
> /usr/local/bin/python3.6

I believe these tasks are running as the cltbld user for some reason (at least the commands are run from /home/cltbld). Maybe python 3 was only installed for root?

It's possible that the solution here is to run these tasks with root, but I think that will require a change to the taskcluster worker that is beyond my area of expertise.

Comment 7

[Andrew Halberstadt [:ahal]]

Hey :dragrom,

Could you check whether python3 is installed for the cltbld user? Thanks!

Comment 8

[Andrew Halberstadt [:ahal]]

Any updates here? This is becoming higher priority as more people are starting to need bug 1436037.

Comment 9

[Dragos Crisan [:dragrom]]

looks like python 3.6 was not installed on OSX workers (created the class to install, but not added include in worker.pp). I installed python3.6 on staging workers. To find this workers, please see: https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010-beta

The worker type for this workers is: gecko-t-osx-1010-beta.
O this workers, python3 is installed and visible for cltbld user:

[[e-mail] ~]$ which python3.6
/usr/local/bin/python3.6

I also created a PR to include the python3 package on the workers

Comment 10

[Dragos Crisan [:dragrom]]

Attachment: Install python3 on OSX generic worker

Install python3 on OSX generic worker

Comment 11

[Gregory Szorc [:gps]]

Does `which python3` yield /usr/local/bin/python3 and is that file a symlink or copy of /usr/local/bin/python3.6?

Comment 12

[Jake Watkins [:dividehex]]

[[e-mail] tools]# which python3
[[e-mail] tools]# which python3.6
/usr/local/bin/python3.6
[[e-mail] tools]# ls -l /usr/local/bin/python3.6
lrwxr-xr-x  1 root  wheel  29 Jul 26 07:23 /usr/local/bin/python3.6 -> /tools/python36/bin/python3.6
[[e-mail] tools]# ls -l /tools/python36/bin/python3.6
-rwxr-xr-x  2 root  wheel  9420 Apr 16 05:55 /tools/python36/bin/python3.6
[[e-mail] tools]# /tools/python36/bin/python3.6 -V
Python 3.6.5
[[e-mail] tools]# ls -l /usr/local/bin/python3
ls: /usr/local/bin/python3: No such file or directory


In addition to including the python36 package in the toplevel worker class, it also looks like we need to add a symlink for /usr/local/bin/python3 -> /tools/python36/bin/python3.6

Comment 13

[Jake Watkins [:dividehex]]

Comment on attachment 8995202 [details] [review]
Install python3 on OSX generic worker

r+ and merged

Comment 14

[Dragos Crisan [:dragrom]]

Attachment: Created symlink for /usr/local/bin/python3

Add a symlink for /usr/local/bin/python3 -> /tools/python36/bin/python3.6
Because we are in the migration process from taskcluster-worker to generic-worker I added the case entry for Ubuntu, to avoid the error message: Cannot install on Ubuntu, when run puppet on Ubuntu machines
Added python3 information, to avoid error: $python3 must be defined in this manifest file when run puppet on Ubuntu machines
Python3 installation on Ubuntu machines will be part os other bug

Comment 15

[Dragos Crisan [:dragrom]]

[[e-mail] ~]$ which python3
/usr/local/bin/python3

Comment 16

[Dragos Crisan [:dragrom]]

merged https://github.com/mozilla-releng/build-puppet/pull/139

Comment 17

[Andrew Halberstadt [:ahal]]

Can this bug be closed?

Comment 18

[Dragos Crisan [:dragrom]]

This bug can be closed. I closed it.