Closed
Bug 1344939
Opened 7 years ago
Closed 7 years ago
Installing newer version of python on buildduty-tools breaks puppet
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Infrastructure & Operations Graveyard
CIDuty
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: arich, Unassigned)
Details
Attachments
(1 file, 1 obsolete file)
30.85 KB,
image/png
|
Details |
Dave noticed that buildduty-tools did not update httpd when all of the other machines. Running puppet by hand yielded a bunch of python errors for yumhelper. It looks like someone installed python 2.7 in /usr/local/bin, breaking puppet. https://tickets.puppetlabs.com/browse/PUP-2144 This is because root has /usr/local/bin before /usr/bin in its path: $PATH = /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin I verified this by hardcoding the python version in /usr/lib/ruby/site_ruby/1.8/puppet/provider/package/yum.rb and that ran without error. I'm not sure what uses python 2.7 on this machine or how you'd like to fix this moving forward.
Comment 1•7 years ago
|
||
Pretty sure this wasn't intentional. cc-ing buildduty folk so they can decide how to fix this host.
Comment 2•7 years ago
|
||
Indeed root has /usr/local/bin before /usr/bin in its path where python 2.7.12 is installed while /usr/bin/python version is 2.6.6(check the printscreen attached). As the newer python version breaks puppet I see 2 solution: -change /etc/profile of root to first look for /usr/bin where the older version of python is available -remove the symlink of python or the newer python version from /usr/local/bin
Reporter | ||
Comment 3•7 years ago
|
||
If you chose to do the first option or first choice in the second option, note that it is likely to be overwritten by package/configuration upgrades and/or reinstantation unless you enforce that using puppet. If there's no need for python 2.7, I recommend removing it all together.
Comment 4•7 years ago
|
||
Nick do you know if we still need python 2.7 on buildduty-tools or we can remove it?
Flags: needinfo?(nthomas)
Comment 5•7 years ago
|
||
I don't have any knowledge about that, wasn't involved in setting the machine up. Why do we hit this problem on this machine but not other places we have python 2.7 ?
Flags: needinfo?(nthomas)
Comment 6•7 years ago
|
||
I'll take a look at this tomorrow morning. Adding a ni to me as a reminder.
Flags: needinfo?(aselagea)
Comment 7•7 years ago
|
||
We needed to update python on both cruncher-aws and buildduty-tools in order to avoid spamming mails for alerts like: "warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0)". Those started coming with the migration to hg 3.9.1 (see bug 1304813). The errors received when running puppet look like this: "Error: Could not get latest version: Execution of '/usr/local/bin/python /usr/lib/ruby/site_ruby/1.8/puppet/provider/package/yumhelper.py' returned 1: <type 'exceptions.AttributeError'>" As mentioned above, running a script like 'yumhelper.py' will only work with Python 2.6 (/usr/bin/python). I don't know how all those symlinks got there (they were dated back Sep 7 2016), but I moved all of them under home/aselagea/fix_buildduty_tools folder and ran puppet again. Result: puppet completed without any issues.
Flags: needinfo?(aselagea)
Comment 8•7 years ago
|
||
Since the initial task was to update httpd on buildduty_tools machine, I took a look at the puppet module responsible for configuring this host and noticed that httpd was not defined there. So the apache version running was old and most likely came with the AMI when the instance was launched. [root@buildduty-tools.srv.releng.usw2.mozilla.com ~]# httpd -v Server version: Apache/2.2.15 (Unix) Server built: Apr 3 2014 23:56:16 [root@buildduty-tools.srv.releng.usw2.mozilla.com bin]# rpm -q httpd httpd-2.2.15-30.el6.centos.x86_64
Comment hidden (mozreview-request) |
Comment 10•7 years ago
|
||
I added the puppet manifests for checking out httpd from our repos (in a similar way to how it's done for cruncher) and tested the patch against my environment. Result: httpd was indeed updated to the proper version. [root@buildduty-tools.srv.releng.usw2.mozilla.com ~]# httpd -v Server version: Apache/2.2.15 (Unix) Server built: Jan 12 2017 17:09:39 [root@buildduty-tools.srv.releng.usw2.mozilla.com ~]# rpm -q httpd httpd-2.2.15-56.el6.centos.3.x86_64 Note: for my tests, I also increased the yum repo flag in order to trigger a refresh. However, I didn't included that in the patch since httpd is already updated.
Comment 11•7 years ago
|
||
Thank you Alin!
Comment 12•7 years ago
|
||
Do we actually use the webserver on buildduty-tools directly? IIRC we simply put the slave_health files in place, regenerating the buildduty reportand then rsync the dir over to the web cluster. Am I missing something here? Is buildduty using the httpd server on this host to serve something else? If not, I think we can simply remove the httpd server from this host entirely and continue rsyncing the single dir.
Comment 13•7 years ago
|
||
(In reply to Chris Cooper [:coop] from comment #12) > Do we actually use the webserver on buildduty-tools directly? IIRC we simply > put the slave_health files in place, regenerating the buildduty reportand > then rsync the dir over to the web cluster. We don't use the webserver on this host at all atm. It is periodically used to run the script that restarts the buildbot masters (slave_health, allthethings and reportor are all running on cruncher). I was thinking setting up httpd in the case we'll be needing it in the future, but I guess that can be easily done if the situation calls. > If not, I think we can simply remove the httpd server from this host > entirely and continue rsyncing the single dir. Removed httpd from this host. Re-ran puppet manually and it completed without any errors.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Updated•7 years ago
|
Attachment #8846670 -
Attachment is obsolete: true
Attachment #8846670 -
Flags: review?(coop)
Updated•6 years ago
|
Product: Release Engineering → Infrastructure & Operations
Updated•4 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•