Installing newer version of python on buildduty-tools breaks puppet

RESOLVED FIXED

Status

Release Engineering
Buildduty
RESOLVED FIXED
8 months ago
7 months ago

People

(Reporter: arr, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

8 months ago
Dave noticed that buildduty-tools did not update httpd when all of the other machines. Running puppet by hand yielded a bunch of python errors for yumhelper. It looks like someone installed python 2.7 in /usr/local/bin, breaking puppet.

https://tickets.puppetlabs.com/browse/PUP-2144

This is because root has /usr/local/bin before /usr/bin in its path:
$PATH = /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin

I verified this by hardcoding the python version in /usr/lib/ruby/site_ruby/1.8/puppet/provider/package/yum.rb and that ran without error.

I'm not sure what uses python 2.7 on this machine or how you'd like to fix this moving forward.

Comment 1

8 months ago
Pretty sure this wasn't intentional. cc-ing buildduty folk so they can decide how to fix this host.
Created attachment 8844831 [details]
python.PNG

Indeed root has /usr/local/bin before /usr/bin in its path where python 2.7.12 is installed while /usr/bin/python version is 2.6.6(check the printscreen attached).
As the newer python version breaks puppet I see 2 solution:
-change /etc/profile of root to first look for /usr/bin where the older version of python is available
-remove the symlink of python or the newer python version from /usr/local/bin
(Reporter)

Comment 3

7 months ago
If you chose to do the first option or first choice in the second option, note that it is likely to be overwritten by package/configuration upgrades and/or reinstantation unless you enforce that using puppet. If there's no need for python 2.7, I recommend removing it all together.
Nick do you know if we still need python 2.7 on buildduty-tools or we can remove it?
Flags: needinfo?(nthomas)
I don't have any knowledge about that, wasn't involved in setting the machine up. Why do we hit this problem on this machine but not other places we have python 2.7 ?
Flags: needinfo?(nthomas)
I'll take a look at this tomorrow morning. Adding a ni to me as a reminder.
Flags: needinfo?(aselagea)
We needed to update python on both cruncher-aws and buildduty-tools in order to avoid spamming mails for alerts like: "warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0)". Those started coming with the migration to hg 3.9.1 (see bug 1304813).

The errors received when running puppet look like this:
"Error: Could not get latest version: Execution of '/usr/local/bin/python /usr/lib/ruby/site_ruby/1.8/puppet/provider/package/yumhelper.py' returned 1: <type 'exceptions.AttributeError'>"

As mentioned above, running a script like 'yumhelper.py' will only work with Python 2.6 (/usr/bin/python). I don't know how all those symlinks got there (they were dated back Sep 7 2016), but I moved all of them under home/aselagea/fix_buildduty_tools folder and ran puppet again.

Result: puppet completed without any issues.
Flags: needinfo?(aselagea)
Since the initial task was to update httpd on buildduty_tools machine, I took a look at the puppet module responsible for configuring this host and noticed that httpd was not defined there. So the apache version running was old and most likely came with the AMI when the instance was launched.

[root@buildduty-tools.srv.releng.usw2.mozilla.com ~]# httpd -v
Server version: Apache/2.2.15 (Unix)
Server built:   Apr  3 2014 23:56:16

[root@buildduty-tools.srv.releng.usw2.mozilla.com bin]# rpm -q httpd
httpd-2.2.15-30.el6.centos.x86_64
Comment hidden (mozreview-request)
I added the puppet manifests for checking out httpd from our repos (in a similar way to how it's done for cruncher) and tested the patch against my environment.

Result: httpd was indeed updated to the proper version.
[root@buildduty-tools.srv.releng.usw2.mozilla.com ~]# httpd -v
Server version: Apache/2.2.15 (Unix)
Server built:   Jan 12 2017 17:09:39
[root@buildduty-tools.srv.releng.usw2.mozilla.com ~]# rpm -q httpd
httpd-2.2.15-56.el6.centos.3.x86_64

Note: for my tests, I also increased the yum repo flag in order to trigger a refresh. However, I didn't included that in the patch since httpd is already updated.

Comment 11

7 months ago
Thank you Alin!

Comment 12

7 months ago
Do we actually use the webserver on buildduty-tools directly? IIRC we simply put the slave_health files in place, regenerating the buildduty reportand then rsync the dir over to the web cluster.

Am I missing something here? Is buildduty using the httpd server on this host to serve something else?

If not, I think we can simply remove the httpd server from this host entirely and continue rsyncing the single dir.
(In reply to Chris Cooper [:coop] from comment #12)
> Do we actually use the webserver on buildduty-tools directly? IIRC we simply
> put the slave_health files in place, regenerating the buildduty reportand
> then rsync the dir over to the web cluster.

We don't use the webserver on this host at all atm. It is periodically used to run the script that restarts the buildbot masters (slave_health, allthethings and reportor are all running on cruncher). I was thinking setting up httpd in the case we'll be needing it in the future, but I guess that can be easily done if the situation calls. 

> If not, I think we can simply remove the httpd server from this host
> entirely and continue rsyncing the single dir.

Removed httpd from this host. Re-ran puppet manually and it completed without any errors.
Status: NEW → RESOLVED
Last Resolved: 7 months ago
Resolution: --- → FIXED
Attachment #8846670 - Attachment is obsolete: true
Attachment #8846670 - Flags: review?(coop)
You need to log in before you can comment on or make changes to this bug.