1156838 - Remove unused globally installed Python packages

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Description

[Ed Morley [:emorley]]

Now that we're using virtualenvs for everything apart from the Chief web app server & the newrelic-plugin-agent (not this is not the same thing as the newrelic python agent), we can remove the old packages from the global python site-packages for both python2.6 and python2.7, on all {admin,prod,stage} treeherder nodes.

Bug 1156731 stopped puppet enforcing the old out-of-repo requirements files, so we can freely pip uninstall them now. Since |pip2.6| doesn't exist, and |pip| uses python2.7, we'll need to explicitly specify the interpreter to clean up the python2.6 site-packages.

eg:

Admin...
* py2.6: sudo python2.6 /usr/bin/pip uninstall -s <packages>
* py2.7: sudo pip2.7 uninstall -s <packages>

Stage...
* py2.6: multi treeherder-stage "python2.6 /usr/bin/pip uninstall -s <packages>"
* py2.7: multi treeherder-stage "pip2.7 uninstall -s <packages>"

Prod...
* py2.6: multi treeherder "python2.6 /usr/bin/pip uninstall -s <packages>"
* py2.7: multi treeherder "pip2.7 uninstall -s <packages>"

Where <packages> is all currently installed packages, minus...

For the admin node:
  - virtualenv
  - Flask
  - gunicorn
  - itsdangerous
  - Jinja2
  - MarkupSafe
  - redis
  - Werkzeug
  - WTForms

For all non-admin nodes:
  - newrelic-plugin-agent
  - helper
  - requests 
  - PyYAML

Comment 1

[Ed Morley [:emorley]]

Additions to the admin node list above:
  - commander (since chief needs it but just to be helpful, doesn't include it in their requirements.txt, sigh)

I've also excluded things like mercurial, supervisor, and supervisor'd dep 'meld3'.
  Plus 'supervisor' and it's dep meld3.

Therefore, list of packages to remove from both python2.6 and python2.7 on admin node:
https://emorley.pastebin.mozilla.org/8830862

Likewise, but for the non-admin nodes:
https://emorley.pastebin.mozilla.org/8830863

fubar, is there anything in those two lists that is actually required by things outside of the treeherder repo, that I've missed? eg MIG or similar? I guess the puppet run would add them back if there was anything required, so it wouldn't be the end of the world. Presuming you were fine with these two lists, I'll run them towards the end of this week, once we're sure the virtualenv stuff is ok.

Comment 2

[Ed Morley [:emorley]]

Flower is in the second list, but it needs celery (go figure) and so I think it would make sense to add it to the virtualenv packages if we actually wanted to use it more regularly. (My understanding is that we don't really use it at the moment, since we have the admin webUI plus new relic reporting, right?)

Comment 3

[Kendall Libby [:fubar] (he/him)]

correct, re: flower. it needs celery==3.1.17, actually, so isn't used at all atm.

things that are required or at least pre-date treeherder:

backports.ssl-match-hostname (part of the py2.7.8 suite)
ethtool
iniparse
iwlib
logutils
M2Crypto
newrelic (managed by rpm and would cause a re-install breaking things)
pycurl
pygpgme
pyOpenSSL
python-dateutil
python-dmidecode
PyXML
rhnlib
urllib3 (I have a note somewhere that says newrelic-* needed that)
yum-metadata-parser


puppet *should* return anything important but in theory, practice and theory are the same thing ;-)

Comment 4

[Ed Morley [:emorley]]

That's useful - thank you :-) I've also pruned anything from the list that I don't remember being in Treeherder's requirements files at some point, just to be safe. My main motivation here is to just remove the 90% of stuff that we definitely don't need, to avoid footguns by people running commands (either manually on a node or via a new script that hasn't updated PATH correctly) outside of the virtualenv and not realising it - since once we start updating versions in the venv, that will cause surprises.

Updated:

To remove from all nodes...

amqp
amqplib
anyjson
argparse
billiard
carrot
celery
celerymon
certifi
cov-core
coverage
Cython
datadiff
datasource
Django
django-browserid
django-celery
django-cors-headers
django-extensions
django-rest-swagger
django-webtest
djangorestframework
docutils
fancy-tag
gevent
gevent-socketio
gevent-websocket
greenlet
jsonfield
kombu
mock
MozillaPulse
MySQL-python
oauth2
py
Pygments
pyinotify
pytest
pytest-cov
python-memcached
six
South
Sphinx
sphinxcontrib-httpdomain
tornado
treeherder-client
Unipath
WebOb
WebTest

Additional removals for the non-admin nodes...

flower
Flask
gunicorn
Jinja2
MarkupSafe
redis
Werkzeug
WTForms

Comment 5

[Ed Morley [:emorley]]

Attachment: Admin node removals

Comment 6

[Ed Morley [:emorley]]

Attachment: Stage node removals

Comment 7

[Ed Morley [:emorley]]

Attachment: Prod node removals