Closed Bug 1344939 Opened 7 years ago Closed 7 years ago

Installing newer version of python on buildduty-tools breaks puppet

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: arich, Unassigned)

Details

Attachments

(1 file, 1 obsolete file)

Dave noticed that buildduty-tools did not update httpd when all of the other machines. Running puppet by hand yielded a bunch of python errors for yumhelper. It looks like someone installed python 2.7 in /usr/local/bin, breaking puppet.

https://tickets.puppetlabs.com/browse/PUP-2144

This is because root has /usr/local/bin before /usr/bin in its path:
$PATH = /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin

I verified this by hardcoding the python version in /usr/lib/ruby/site_ruby/1.8/puppet/provider/package/yum.rb and that ran without error.

I'm not sure what uses python 2.7 on this machine or how you'd like to fix this moving forward.
Pretty sure this wasn't intentional. cc-ing buildduty folk so they can decide how to fix this host.
Attached image python.PNG
Indeed root has /usr/local/bin before /usr/bin in its path where python 2.7.12 is installed while /usr/bin/python version is 2.6.6(check the printscreen attached).
As the newer python version breaks puppet I see 2 solution:
-change /etc/profile of root to first look for /usr/bin where the older version of python is available
-remove the symlink of python or the newer python version from /usr/local/bin
If you chose to do the first option or first choice in the second option, note that it is likely to be overwritten by package/configuration upgrades and/or reinstantation unless you enforce that using puppet. If there's no need for python 2.7, I recommend removing it all together.
Nick do you know if we still need python 2.7 on buildduty-tools or we can remove it?
Flags: needinfo?(nthomas)
I don't have any knowledge about that, wasn't involved in setting the machine up. Why do we hit this problem on this machine but not other places we have python 2.7 ?
Flags: needinfo?(nthomas)
I'll take a look at this tomorrow morning. Adding a ni to me as a reminder.
Flags: needinfo?(aselagea)
We needed to update python on both cruncher-aws and buildduty-tools in order to avoid spamming mails for alerts like: "warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0)". Those started coming with the migration to hg 3.9.1 (see bug 1304813).

The errors received when running puppet look like this:
"Error: Could not get latest version: Execution of '/usr/local/bin/python /usr/lib/ruby/site_ruby/1.8/puppet/provider/package/yumhelper.py' returned 1: <type 'exceptions.AttributeError'>"

As mentioned above, running a script like 'yumhelper.py' will only work with Python 2.6 (/usr/bin/python). I don't know how all those symlinks got there (they were dated back Sep 7 2016), but I moved all of them under home/aselagea/fix_buildduty_tools folder and ran puppet again.

Result: puppet completed without any issues.
Flags: needinfo?(aselagea)
Since the initial task was to update httpd on buildduty_tools machine, I took a look at the puppet module responsible for configuring this host and noticed that httpd was not defined there. So the apache version running was old and most likely came with the AMI when the instance was launched.

[root@buildduty-tools.srv.releng.usw2.mozilla.com ~]# httpd -v
Server version: Apache/2.2.15 (Unix)
Server built:   Apr  3 2014 23:56:16

[root@buildduty-tools.srv.releng.usw2.mozilla.com bin]# rpm -q httpd
httpd-2.2.15-30.el6.centos.x86_64
I added the puppet manifests for checking out httpd from our repos (in a similar way to how it's done for cruncher) and tested the patch against my environment.

Result: httpd was indeed updated to the proper version.
[root@buildduty-tools.srv.releng.usw2.mozilla.com ~]# httpd -v
Server version: Apache/2.2.15 (Unix)
Server built:   Jan 12 2017 17:09:39
[root@buildduty-tools.srv.releng.usw2.mozilla.com ~]# rpm -q httpd
httpd-2.2.15-56.el6.centos.3.x86_64

Note: for my tests, I also increased the yum repo flag in order to trigger a refresh. However, I didn't included that in the patch since httpd is already updated.
Thank you Alin!
Do we actually use the webserver on buildduty-tools directly? IIRC we simply put the slave_health files in place, regenerating the buildduty reportand then rsync the dir over to the web cluster.

Am I missing something here? Is buildduty using the httpd server on this host to serve something else?

If not, I think we can simply remove the httpd server from this host entirely and continue rsyncing the single dir.
(In reply to Chris Cooper [:coop] from comment #12)
> Do we actually use the webserver on buildduty-tools directly? IIRC we simply
> put the slave_health files in place, regenerating the buildduty reportand
> then rsync the dir over to the web cluster.

We don't use the webserver on this host at all atm. It is periodically used to run the script that restarts the buildbot masters (slave_health, allthethings and reportor are all running on cruncher). I was thinking setting up httpd in the case we'll be needing it in the future, but I guess that can be easily done if the situation calls. 

> If not, I think we can simply remove the httpd server from this host
> entirely and continue rsyncing the single dir.

Removed httpd from this host. Re-ran puppet manually and it completed without any errors.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Attachment #8846670 - Attachment is obsolete: true
Attachment #8846670 - Flags: review?(coop)
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: