1133138 - Alias python/pip/easy_install to their Python 2.7 versions on stage and prod

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

Please could we alias python to Python2.7 rather than Python2.6?

Currently:

[[e-mail]3 ~]$ multi treeherder python --version
...
[treeherder1.webapp.scl3.mozilla.com] err: Python 2.6.6
[treeherder2.webapp.scl3.mozilla.com] err: Python 2.6.6
[treeherder3.webapp.scl3.mozilla.com] err: Python 2.6.6
[treeherder-processor1.private.scl3.mozilla.com] err: Python 2.6.6
[treeherder-processor2.private.scl3.mozilla.com] err: Python 2.6.6
[treeherder-processor3.private.scl3.mozilla.com] err: Python 2.6.6
[treeherder-etl1.private.scl3.mozilla.com] err: Python 2.6.6
[treeherder-etl2.private.scl3.mozilla.com] err: Python 2.6.6
[treeherder-rabbitmq1.private.scl3.mozilla.com] err: Python 2.6.6

[[e-mail]3 ~]$ multi treeherder-stage python --version
...
[treeherder1.stage.webapp.scl3.mozilla.com] err: Python 2.6.6
[treeherder2.stage.webapp.scl3.mozilla.com] err: Python 2.6.6
[treeherder3.stage.webapp.scl3.mozilla.com] err: Python 2.6.6
[treeherder-processor1.stage.private.scl3.mozilla.com] err: Python 2.6.6
[treeherder-processor2.stage.private.scl3.mozilla.com] err: Python 2.6.6
[treeherder-processor3.stage.private.scl3.mozilla.com] err: Python 2.6.6
[treeherder-etl1.stage.private.scl3.mozilla.com] err: Python 2.6.6
[treeherder-etl2.stage.private.scl3.mozilla.com] err: Python 2.6.6
[treeherder-rabbitmq1.stage.private.scl3.mozilla.com] err: Python 2.6.6

And unfortunately:

[[e-mail]3 ~]$ multi treeherder-stage "ps axo command | egrep '^/usr/bin/python ' | cut -d' ' -f-2"
...
[treeherder1.stage.webapp.scl3.mozilla.com] out: /usr/bin/python /usr/bin/supervisord
[treeherder2.stage.webapp.scl3.mozilla.com] out: /usr/bin/python /usr/bin/supervisord
[treeherder3.stage.webapp.scl3.mozilla.com] out: /usr/bin/python /usr/bin/gunicorn
[treeherder3.stage.webapp.scl3.mozilla.com] out: /usr/bin/python /usr/bin/gunicorn
[treeherder3.stage.webapp.scl3.mozilla.com] out: /usr/bin/python /usr/bin/gunicorn
[treeherder3.stage.webapp.scl3.mozilla.com] out: /usr/bin/python /usr/bin/gunicorn
[treeherder3.stage.webapp.scl3.mozilla.com] out: /usr/bin/python /usr/bin/gunicorn
[treeherder3.stage.webapp.scl3.mozilla.com] out: /usr/bin/python /usr/bin/gunicorn
[treeherder3.stage.webapp.scl3.mozilla.com] out: /usr/bin/python /usr/bin/supervisord
[treeherder-etl1.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/supervisord
[treeherder-etl2.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/supervisord
[treeherder-processor1.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/celery
[treeherder-processor1.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/celery
[treeherder-processor1.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/supervisord
[treeherder-processor2.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/celery
[treeherder-processor2.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/celery
[treeherder-processor2.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/celery
[treeherder-processor2.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/celery
[treeherder-processor2.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/celery
[treeherder-processor2.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/celery
[treeherder-processor2.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/celery
[treeherder-processor2.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/celery
[treeherder-processor2.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/supervisord
[treeherder-processor3.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/celery
[treeherder-processor3.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/supervisord
[treeherder-rabbitmq1.stage.private.scl3.mozilla.com] out: /usr/bin/python /usr/bin/supervisord

Note some of the above appears to be stuck processes, presumably a la bug 1131059, and they'd probably be fine after being killed:
[treeherder-processor2.stage.private.scl3.mozilla.com] out: 10-04:06:29 /usr/bin/python /usr/bin/celery -A treeherder worker -Q log_parser_fail,log_parser,log_parser_hp --logfile=/var/log/celery/celery_worker_log_parser.log -l INFO --maxtasksperchild=500 -n log_parser.%h
[treeherder-processor2.stage.private.scl3.mozilla.com] out: 10-04:37:24 /usr/bin/python /usr/bin/celery -A treeherder worker -Q log_parser_fail,log_parser,log_parser_hp --logfile=/var/log/celery/celery_worker_log_parser.log -l INFO --maxtasksperchild=500 -n log_parser.%h

However running gunicorn with Python 2.6 is causing things like:

https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/3098137500/similar_errors?original_error_id=3098137500

Comment 1

[Ed Morley [:emorley]]

And also for pip:

[[e-mail]3 ~]$ multi treeherder 'pip --version'
...
[treeherder-etl1.private.scl3.mozilla.com] out: pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)
[treeherder-etl2.private.scl3.mozilla.com] out: pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)
[treeherder-processor1.private.scl3.mozilla.com] out: pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)
[treeherder-processor2.private.scl3.mozilla.com] out: pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)
[treeherder-processor3.private.scl3.mozilla.com] out: pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)
[treeherder-rabbitmq1.private.scl3.mozilla.com] out: pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)
[treeherder1.webapp.scl3.mozilla.com] out: pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)
[treeherder2.webapp.scl3.mozilla.com] out: pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)
[treeherder3.webapp.scl3.mozilla.com] out: pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)

Since otherwise we'll end up operating on the 2.7 libs directory rather than the 2.7 if we don't remember to use pip2.7 (as I didn't when doing a |pip freeze| just now, and getting quite confused to see Django 1.5 listed for stage).

Comment 2

[Ed Morley [:emorley]]

[[e-mail]3 ~]$ multi treeherder-stage 'cat /usr/bin/gunicorn | head -n 1'
...
[treeherder-etl1.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder-etl2.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder-processor1.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder-processor2.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder-processor3.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder-rabbitmq1.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder1.stage.webapp.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder2.stage.webapp.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder3.stage.webapp.scl3.mozilla.com] out: #!/usr/bin/python

[[e-mail]3 ~]$ multi treeherder-stage 'cat /usr/bin/celery | head -n 1'
...
[treeherder-etl1.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder-etl2.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder-processor1.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder-processor2.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder-processor3.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder-rabbitmq1.stage.private.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder1.stage.webapp.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder2.stage.webapp.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder3.stage.webapp.scl3.mozilla.com] out: #!/usr/bin/python

Seems like it's just treeherder3.stage.webapp.scl3.mozilla.com that's the odd child out wrt shebangs.

Btw I tried to restart the chief & pushbot services on the admin node, in the hope it stopped them being quite so chatty on IRC, but the chief service wouldn't start again. Inspecting /var/log/chief-supervisor.log revealed that we were missing the chief deps {redis, flask, wtforms} from the python 2.7 site-packages directory, I installed these on the admin node using |pip2.7 install redis flask wtforms| and then was able to start the chief service again. Hope that was ok.

Comment 3

[Ed Morley [:emorley]]

Hopefully some of the issues you've seen with pip fubar, will go away (a) partly due to the switch to using Python2.7, (b) due to updating to pip 6.0.8 in python2.7's site-packages, rather than the 1.3.1 (!!) that was installed for Python2.6 - and it was a similar story for setuptools.

Comment 4

[Ed Morley [:emorley]]

gunicorn shebang corrected using:
[[e-mail]3 ~]$ multi treeherder-stage-web 'pip2.7 install --upgrade --no-deps --force-reinstall gunicorn==17.5'

[[e-mail]3 ~]$ multi treeherder-stage-web 'cat /usr/bin/gunicorn | head -n 1'
...
[treeherder3.stage.webapp.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder1.stage.webapp.scl3.mozilla.com] out: #!/usr/bin/python2.7
[treeherder2.stage.webapp.scl3.mozilla.com] out: #!/usr/bin/python2.7

Perhaps the previous pip install (post update to Python 2.7) was done using |pip install| rather than |pip2.7 install|?

Have done a |restart-jobs -s all| after that and the errors on newrelic are subsiding :-)

Comment 5

[Ed Morley [:emorley]]

To consolidate the above comments:

1) Please could we alias python to Python2.7 rather than Python2.6, plus alias pip to pip2.7, to avoid us accidentally using python2.6? (Or remove 2.6 entirely)
2) I had to pip2.7 install some packages on the admin node (see last paragraph of comment 2 for which) since python2.7's site-packages was missing things needed by chief and was preventing the stage deploy over the weekend. Not sure if this needs updating in puppet somewhere to reflect what's installed - could you do so if needed?
3) I had to force reinstall gunicorn on treeherder3.stage.webapp using pip2.7 since the shebang was wrong in the wrapper script compared to all the other nodes, so it was using python->python2.6 instead and causing errors after the deploy to stage during the weekend. Not sure if this was deployed via puppet, if so, could you updated the puppet script for this too?

Thanks :-)

Comment 6

[Kendall Libby [:fubar] (he/him)]

1) Where? /etc/profile.d/treeherder.sh? That still won't fix anything that calls it with the full path. And removing 2.6 is a straight up WONTFIX as that's the system default and god knows what might break if it goes away. RHEL-- again.
2) Ah, another case where RHEL packaging screws things up, and the last-installed version wins in /usr/bin/. Thank you for fixing.
3) Not done by puppet, so I may have fat fingered that one. Puppet *could* do the webapp nodes, but most of the others it can't due to network restrictions. Doing it manually is less work than trying to "fix" puppet to work around it, atm.

Comment 7

[Ed Morley [:emorley]]

(In reply to Kendall Libby [:fubar] from comment #6)
> 1) Where? /etc/profile.d/treeherder.sh? That still won't fix anything that
> calls it with the full path. And removing 2.6 is a straight up WONTFIX as
> that's the system default and god knows what might break if it goes away.
> RHEL-- again.

I don't know where is best to be honest - anything to avoid us accidentally using pip rather than pip2.7 is probably the biggest win (even without python->python2.7)

Comment 8

[Kendall Libby [:fubar] (he/him)]

Added aliases in treeherder.sh for python, pip, and easy_install; will be applied when puppet runs.

Comment 9

[Ed Morley [:emorley]]

That's great, thank you :-)