1333079 - Update Python dependencies, Jan edition

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P2)

Description

[Ed Morley [:emorley]]

Other than the Django 1.10 migration, we last updated deps in October, so time to do so again.

https://requires.io/github/mozilla/treeherder/requirements/?branch=master

vagrant ~/treeherder $ pip list -o
amqp (1.4.9) - Latest: 2.1.4 [wheel]
beautifulsoup4 (4.5.1) - Latest: 4.5.3 [wheel]
billiard (3.3.0.23) - Latest: 3.5.0.2 [sdist]
celery (3.1.24) - Latest: 4.0.2 [wheel]
django-cors-headers (1.2.2) - Latest: 2.0.0 [wheel]
django-environ (0.4.0) - Latest: 0.4.1 [wheel]
django-extensions (1.7.4) - Latest: 1.7.5 [wheel]
django-filter (0.15.3) - Latest: 1.0.1 [wheel]
django-rest-swagger (2.1.0) - Latest: 2.1.1 [wheel]
elasticsearch (2.4.0) - Latest: 5.1.0 [wheel]
elasticsearch-dsl (2.1.0) - Latest: 5.1.0 [wheel]
flake8 (2.5.4) - Latest: 3.2.1 [wheel]
isort (4.2.2) - Latest: 4.2.5 [wheel]
kombu (3.0.37) - Latest: 4.0.2 [wheel]
mccabe (0.4.0) - Latest: 0.6.0 [wheel]
mohawk (0.3.3) - Latest: 0.3.4 [wheel]
newrelic (2.72.1.53) - Latest: 2.78.0.57 [sdist]
pep8 (1.5.7) - Latest: 1.7.0 [wheel]
py (1.4.31) - Latest: 1.4.32 [wheel]
pyflakes (1.0.0) - Latest: 1.5.0 [wheel]
pytest (3.0.3) - Latest: 3.0.6 [wheel]
pytest-django (3.0.0) - Latest: 3.1.2 [wheel]
python-dateutil (2.5.3) - Latest: 2.6.0 [wheel]
pytz (2016.7) - Latest: 2016.10 [wheel]
requests (2.11.1) - Latest: 2.12.5 [wheel]
simplejson (3.9.0) - Latest: 3.10.0 [sdist]
urllib3 (1.18) - Latest: 1.20 [wheel]
waitress (1.0.0) - Latest: 1.0.1 [wheel]
WebOb (1.6.2) - Latest: 1.7.1 [wheel]
WebTest (2.0.23) - Latest: 2.0.24 [wheel]
Werkzeug (0.11.11) - Latest: 0.11.15 [wheel]
whitenoise (3.2.2) - Latest: 3.2.3 [wheel]

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:update-common-deps > mozilla:master

Comment 2

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:update-dev-deps > mozilla:master

Comment 3

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/1880eefdb4d5f0711f8be235a77736a1f2113ed8
Bug 1333079 - Update py from 1.4.31 to 1.4.32

https://github.com/pytest-dev/py/blob/master/CHANGELOG
https://github.com/pytest-dev/py/compare/1.4.31...1.4.32

https://github.com/mozilla/treeherder/commit/f26d4427ef046e8c37c01b281fa9d6b9d1d6c255
Bug 1333079 - Update pytest from 3.0.3 to 3.0.6

http://pytest.org/latest/changelog.html
https://github.com/pytest-dev/pytest/compare/3.0.3...3.0.6

https://github.com/mozilla/treeherder/commit/2040febae997922f369affd24797a6a8bd9de4e5
Bug 1333079 - Update pytest-django from 3.0.0 to 3.1.2

https://pytest-django.readthedocs.io/en/latest/changelog.html
https://github.com/pytest-dev/pytest-django/compare/3.0.0...3.1.2

https://github.com/mozilla/treeherder/commit/c447911c076fb184600b46248fafc53c91b0d4d0
Bug 1333079 - Update django-extensions from 1.7.4 to 1.7.5

https://github.com/django-extensions/django-extensions/blob/master/CHANGELOG.md
https://github.com/django-extensions/django-extensions/compare/1.7.4...1.7.5

https://github.com/mozilla/treeherder/commit/026471c63132f03e7c11b1c31dcc6c4c9fd965ad
Bug 1333079 - Update Werkzeug from 0.11.11 to 0.11.15

https://github.com/pallets/werkzeug/blob/master/CHANGES
https://github.com/mitsuhiko/werkzeug/compare/0.11.11...0.11.15

https://github.com/mozilla/treeherder/commit/e27bebdb86f84931c626a6447aa6230e8642aa25
Bug 1333079 - Update beautifulsoup4 from 4.5.1 to 4.5.3

https://groups.google.com/forum/#!forum/beautifulsoup

https://github.com/mozilla/treeherder/commit/d347a632739c9455ffe0417aea745461daa060c2
Bug 1333079 - Update waitress from 1.0.0 to 1.0.1

http://docs.pylonsproject.org/projects/waitress/en/latest/#change-history
https://github.com/Pylons/waitress/compare/v1.0.0...v1.0.1

https://github.com/mozilla/treeherder/commit/06be75f27eccc8215057c63e666d984c8c120836
Bug 1333079 - Update WebOb from 1.6.2 to 1.7.1

https://webob.readthedocs.io/en/latest/changes.html
https://github.com/Pylons/webob/compare/1.6.2...026d162ebde8ad4bf90a60a79fff420c4400e758

https://github.com/mozilla/treeherder/commit/733cc7e739085c4ad04e620626f28537d6ec9264
Bug 1333079 - Update WebTest from 2.0.23 to 2.0.24

https://github.com/Pylons/webtest/blob/master/CHANGELOG.rst
https://github.com/Pylons/webtest/compare/2.0.23...2.0.24

Comment 4

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/06b0f6d9768524084bec2c5ca00d830a6eb2eddf
Bug 1333079 - Update django-cors-headers from 1.2.2 to 2.0.0

https://github.com/ottoyiu/django-cors-headers/blob/master/HISTORY.rst
https://github.com/ottoyiu/django-cors-headers/compare/1.2.2...2.0.0

https://github.com/mozilla/treeherder/commit/cc5c17f2bca0fff887d5193b42183be54a519a14
Bug 1333079 - Update newrelic from 2.72.1.53 to 2.78.0.57

https://docs.newrelic.com/docs/release-notes/agent-release-notes/python-release-notes
https://github.com/edmorley/newrelic-python-agent/compare/v2.72.1.53...v2.78.0.57

https://github.com/mozilla/treeherder/commit/3a74046c106a4cf9ff5b1e465428785cbd7409f7
Bug 1333079 - Update django-rest-swagger from 2.1.0 to 2.1.1

https://github.com/marcgibbons/django-rest-swagger/blob/master/CHANGELOG.md
https://github.com/marcgibbons/django-rest-swagger/compare/2.1.0...2.1.1

https://github.com/mozilla/treeherder/commit/d73ac87385503478842ddf6e2e1dd84a5412cbd7
Bug 1333079 - Update simplejson from 3.9.0 to 3.10.0

https://github.com/simplejson/simplejson/blob/master/CHANGES.txt
https://github.com/simplejson/simplejson/compare/v3.9.0...v3.10.0

https://github.com/mozilla/treeherder/commit/9f0a748c78236e26e05dfa70642731107f473517
Bug 1333079 - Update urllib3 from 1.18 to 1.20

https://github.com/shazow/urllib3/blob/master/CHANGES.rst
https://github.com/shazow/urllib3/compare/1.18...1.20

https://github.com/mozilla/treeherder/commit/aafed02ca030cf4365f8e4f5b9526c53ad77d6e4
Bug 1333079 - Update python-dateutil from 2.5.3 to 2.6.0

https://github.com/dateutil/dateutil/blob/master/NEWS
https://github.com/dateutil/dateutil/compare/2.5.3...2.6.0

https://github.com/mozilla/treeherder/commit/7fbba012cd65f37a1d656f316faacc158d1a75c6
Bug 1333079 - Update mohawk from 0.3.3 to 0.3.4

https://mohawk.readthedocs.io/en/latest/#changelog
https://github.com/kumar303/mohawk/compare/0.3.3...0.3.4

https://github.com/mozilla/treeherder/commit/3fbbc7b1db90762468fada7117ebe3866266d8c6
Bug 1333079 - Update pytz from 2016.7 to 2016.10

https://launchpad.net/pytz/+announcements

https://github.com/mozilla/treeherder/commit/75b8aae3d68d22098e019971702939ac7ab242e5
Bug 1333079 - Update django-environ from 0.4.0 to 0.4.1

https://github.com/joke2k/django-environ#changelog
https://github.com/joke2k/django-environ/compare/v0.4...v0.4.1

https://github.com/mozilla/treeherder/commit/252913fa4764d3527048ab9324f5126b5c5d4091
Bug 1333079 - Update whitenoise from 3.2.2 to 3.2.3

http://whitenoise.evans.io/en/stable/changelog.html
https://github.com/evansd/whitenoise/compare/v3.2.2...v3.2.3

https://github.com/mozilla/treeherder/commit/05b6685241e4ae0b4c42a5564e899c85beedb08d
Bug 1333079 - Update requests from 2.11.1 to 2.13.0

https://github.com/kennethreitz/requests/blob/master/HISTORY.rst
https://github.com/kennethreitz/requests/compare/v2.11.1...v2.13.0

Comment 5

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:update-celery-kombu > mozilla:master

Comment 6

[Ed Morley [:emorley]]

> Bug 1333079 - Update django-cors-headers from 1.2.2 to 2.0.0
> 
> https://github.com/ottoyiu/django-cors-headers/blob/master/HISTORY.rst
> https://github.com/ottoyiu/django-cors-headers/compare/1.2.2...2.0.0

The new release no longer creates the unnecessary `corsheaders_corsmodel` table (https://github.com/ottoyiu/django-cors-headers/commit/cb431687a79899102b3ca05a558a03cbc8c5c384), so it's not present in a clean vagrant env.

I've dropped the old unused table from dev/stage/prod using:

DROP TABLE `treeherder`.`corsheaders_corsmodel`;

Comment 7

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/f09694e3ccb60393ea171e2923a0633186049124
Bug 1333079 - Update to newer celery, kombu, py-amqp and billiard

Updating in one go, since the celery 4 release only supports the newer
versions of its dependencies and vice versa.

Of note, this fixes the unhelpful connection error messages shown when
in fact there was an authentication problem.

Changes:

http://celery.readthedocs.io/en/stable/whatsnew-4.0.html
http://docs.celeryproject.org/en/stable/changelog.html
https://github.com/celery/celery/compare/v3.1.24...v4.0.2

http://docs.celeryproject.org/projects/kombu/en/stable/changelog.html
https://github.com/celery/kombu/compare/v3.0.37...v4.0.2

http://amqp.readthedocs.io/en/stable/changelog.html
https://github.com/celery/py-amqp/compare/v1.4.9...v2.1.4

https://github.com/celery/billiard/blob/master/CHANGES.txt
https://github.com/celery/billiard/compare/v3.3.0.23...v3.5.0.2

https://github.com/mozilla/treeherder/commit/1db3e2baf46c5df426dc234657d94e9585668456
Bug 1333079 - Switch to new style Celery setting names

Converted using:

celery upgrade settings treeherder/config/settings.py --django
celery upgrade settings tests/settings.py --django

As suggested by:
http://celery.readthedocs.io/en/latest/whatsnew-4.0.html#lowercase-setting-names

(I manually reverted the env variable name back to simplify deployment,
we can always clean these up later.)

The Django integration in celery.py has been cleaned up by following:
https://celery.readthedocs.io/en/latest/django/first-steps-with-django.html

Comment 8

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/8fae27955af6a5bd73c4a70a1fadb2208e21ff0e
Revert "Bug 1333079 - Switch to new style Celery setting names"

This reverts commit 1db3e2baf46c5df426dc234657d94e9585668456.

https://github.com/mozilla/treeherder/commit/5cfd338801c6b51c26d49fd68ea7135b40876b66
Revert "Bug 1333079 - Update to newer celery, kombu, py-amqp and billiard"

This reverts commit f09694e3ccb60393ea171e2923a0633186049124.

Comment 9

[Ed Morley [:emorley]]

I've reverted the celery/kombu/py-amqp change since it was causing a pile up (30,000 tasks) on the `default` queue, rather than jobs being sent to the queue they should have been. Will have to investigate.

Comment 10

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #9)
> I've reverted the celery/kombu/py-amqp change since it was causing a pile up
> (30,000 tasks) on the `default` queue, rather than jobs being sent to the
> queue they should have been. Will have to investigate.

I've filed bug 1337717 to tackle this again in the future.

For now, we're all done here :-)