1125838 - Determine which packages have updates available and update them if appropriate

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P4)

Description

[Ed Morley [:emorley]]

We should go through requirements/*.txt and:
1) Look for cases where we are not pinning a specific version (eg >=) and decide if that is still appropriate.
2) If there are any remaining unpinned dependencies in pure.txt that have updates (currently 'requests' and 'six'), re-run generate-vendor-lib.py and check them into the repo, since vendor/ is currently out of sync.
3) Make a list of pinned-version packages that have updates, and file dep bugs for evaluating whether we should update them.

Comment 1

[Ed Morley [:emorley]]

We'll also have to try and deal with bug 1070470 - since I'm presuming we don't even use the in-repo requirements files in some cases.

Comment 2

[Mauro Doglio [:mdoglio]]

For 3) you can get a list of all the outdated packages running:
pip list -o
if the list parameter is not available, upgrade your pip version with:
pip install --upgrade pip

Comment 3

[Ed Morley [:emorley]]

#1-2 have been done by bug 1070470.

As for #3, the dev.txt parts are being done in bug 1143033 and the docs.txt parts were done in bug 1070470.

This just leaves checking for updates in common.txt and checked-in.txt

Comment 4

[Ed Morley [:emorley]]

From common.txt/dev.txt:

Cython (Current: 0.19.2 Latest: 0.22)
v0.19.2 was released in 2013-10-13!
https://github.com/cython/cython/blob/master/CHANGES.rst

kombu (Current: 3.0.23 Latest: 3.0.24)
https://github.com/celery/kombu/blob/3.0/Changelog

celery (Current: 3.1.16 Latest: 3.1.17)
http://docs.celeryproject.org/en/latest/changelog.html

requests (Current: 2.4.1 Latest: 2.6.0)
https://github.com/kennethreitz/requests/blob/master/HISTORY.rst

pytz (Current: 2014.10 Latest: 2015.2)
http://bazaar.launchpad.net/~stub/pytz/devel/view/head:/src/CHANGES.txt (but it's out of date, sigh)

Django (Current: 1.7.7 Latest: 1.8)
https://docs.djangoproject.com/en/1.8/releases/1.8/
(But we'll likely want to wait for a point release before even considering this)

django-extensions (Current: 1.5.1 Latest: 1.5.2)
https://github.com/django-extensions/django-extensions/blob/master/CHANGELOG.md

simplejson (Current: 3.3.0 Latest: 3.6.5)
https://github.com/simplejson/simplejson/blob/master/CHANGES.txt

pep8 (Current: 1.5.7 Latest: 1.6.2)
https://github.com/jcrocholl/pep8/blob/master/CHANGES.txt

gunicorn (Current: 17.5 Latest: 19.3.0)
http://docs.gunicorn.org/en/latest/news.html
older: http://docs.gunicorn.org/en/latest/2014-news.html
older: http://docs.gunicorn.org/en/latest/2013-news.html

pytest (Current: 2.6.4 Latest: 2.7.0)
http://pytest.org/latest/changelog.html

Currently in checked-in.txt (though these will move soon):

Unipath (Current: 1.0 Latest: 1.1)
https://github.com/mikeorr/Unipath/blob/master/CHANGES

django-cors-headers (Current: 0.11 Latest: 1.0.0)
https://github.com/ottoyiu/django-cors-headers/releases
older: https://github.com/ottoyiu/django-cors-headers#changelog

drf-extensions (Current: 0.2.5 Latest: 0.2.7)
http://chibisov.github.io/drf-extensions/docs/#release-notes

django-browserid (Current: 0.9 Latest: 0.11.1)
https://github.com/mozilla/django-browserid/blob/master/CHANGELOG.rst

django-rest-swagger (Current: 0.1.11 Latest: 0.2.9)
https://github.com/marcgibbons/django-rest-swagger/blob/master/CHANGELOG.md

httplib2 (Current: 0.7.4 Latest: 0.9.1)
https://github.com/jcgregorio/httplib2/blob/master/CHANGELOG

jsonfield (Current: 0.9.20 Latest: 1.0.3)
https://github.com/bradjasper/django-jsonfield#changes

mozlog (Current: 2.10 Latest: 2.11)
https://github.com/mozilla/gecko-dev/commits/master/testing/mozbase/mozlog

python-memcached (Current: 1.48 Latest: 1.54)
https://github.com/linsomniac/python-memcached/blob/master/ChangeLog

djangorestframework (Current: 2.3.12 Latest: 3.1.1)
http://www.django-rest-framework.org/topics/release-notes/
older: http://tomchristie.github.io/rest-framework-2-docs/topics/release-notes

docs.txt:

Sphinx (Current: 1.3 Latest: 1.3.1)
http://sphinx-doc.org/changes.html

pytz (Current: 2014.10 Latest: 2015.2)
See above

alabaster (Current: 0.7.2 Latest: 0.7.3)
https://github.com/bitprophet/alabaster#changelog

Comment 5

[Ed Morley [:emorley]]

prod.txt:

newrelic (Current: 2.44.0.36 Latest: 2.50.0.39)
https://docs.newrelic.com/docs/release-notes/agent-release-notes/python-release-notes

Comment 6

[Ed Morley [:emorley]]

Revised list after the recent changes (I've excluded things we don't want to update or have bugs already):

Unipath (Current: 1.0 Latest: 1.1 [wheel])
django-cors-headers (Current: 0.11 Latest: 1.1.0 [sdist])
flake8 (Current: 2.4.0 Latest: 2.4.1 [wheel])
django-rest-swagger (Current: 0.1.11 Latest: 0.2.9 [sdist])
simplejson (Current: 3.6.5 Latest: 3.7.1 [sdist])
pytz (Current: 2014.10 Latest: 2015.4 [wheel])
httplib2 (Current: 0.7.4 Latest: 0.9.1 [sdist])
django-extensions (Current: 1.5.2 Latest: 1.5.5 [wheel])
Django (Current: 1.7.7 Latest: 1.8.2 [wheel])
  -> Even though we're not ready to move to 1.8, there's a 1.7.8 out
jsonfield (Current: 0.9.20 Latest: 1.0.3 [sdist])
mozlog (Current: 2.10 Latest: 2.11 [sdist])
python-memcached (Current: 1.48 Latest: 1.54 [wheel])
billiard (Current: 3.3.0.19 Latest: 3.3.0.20 [sdist])
gunicorn (Current: 17.5 Latest: 19.3.0 [wheel])
requests (Current: 2.6.2 Latest: 2.7.0 [wheel])

Comment 7

[Ed Morley [:emorley]]

On latest master:

vagrant ~/treeherder $ pip list -o
django-browserid (Current: 0.10 Latest: 1.0.0 [sdist])
pep8 (Current: 1.5.7 Latest: 1.6.2 [wheel])
Django (Current: 1.7.7 Latest: 1.8.3 [wheel])
djangorestframework (Current: 2.4.5 Latest: 3.1.3 [wheel])
pip (Current: 6.1.1 Latest: 7.1.0 [wheel])
django-cors-headers (Current: 0.11 Latest: 1.1.0 [sdist])

All of those bar pep8/pip (and Django 1.8 vs the point update) have bugs filed.

We don't want to update pep8, since it doesn't play nicely with the current version of flake8. We don't want to update pip yet, since peep is not yet compatible with pip 7.x.

Calling this done :-)

Comment 8

[Ed Morley [:emorley]]

A quote from the 2015 Data Breach Investigations Report (http://www.verizonenterprise.com/resources/reports/rp_data-breach-investigation-report-2015_en_xg.pdf) made me think of this bug:
"We found that 99.9% of the exploited vulnerabilities had been compromised more than a year after the associated CVE was published."

I'm glad we're now running much more up to date versions of everything.