846349 - Update outdated packages

Status: RESOLVED WONTFIX

(Marketplace Graveyard :: Code Quality, defect, P2)

Description

[Rob Hudson [:robhudson]]

This is output from pip-tools `pip-review` command. We may or may not want to upgrade many of these but I thought it'd be worth a review from time to time...

Django==1.5 is available (you have 1.4.5)
Jinja2==2.6 is available (you have 2.5.5)
M2Crypto==0.21.1 is available (you have 0.20.2)
PIL==1.1.6 is available (you have 1.1.7)
PyBrowserID==0.9.1 is available (you have 0.6.0)
SQLAlchemy==0.7.10 is available (you have 0.7.5)
Sphinx==1.1.3 is available (you have 0.6.3)
bleach==1.2.1 is available (you have 1.1.5)
celery==3.0.15 is available (you have 2.5.1)
chardet==2.1.1 is available (you have 1.0.1)
django-celery==3.0.11 is available (you have 2.2.4)
django-statsd-mozilla==0.3.8.5 is available (you have 0.3.8)
django-storages==1.1.6 is available (you have 1.1.4)
django-tastypie==0.9.12 is available (you have 0.9.11)
docutils==0.10 is available (you have 0.7)
easy-thumbnails==1.2 is available (you have 1.1)
elasticutils==0.6 is available (you have 0.5)
feedparser==5.1.3 is available (you have 5.0.1)
gunicorn==0.17.2 is available (you have 0.15.0)
html5lib==0.95 is available (you have 0.90)
httplib2==0.7.7 is available (you have 0.7.6)
kombu==2.5.6 is available (you have 2.1.2)
lxml==3.1.0 is available (you have 2.2.6)
metlog-py==0.10.0 is available (you have 0.9.10)
mimeparse==0.1.4 is available (you have 0.1.3)
newrelic==1.10.2.38 is available (you have 1.5.0.103)
pyquery==1.2.4 is available (you have 0.4)
python-dateutil==2.1 is available (you have 1.5)
pytz==2012j is available (you have 2010e)
raven==3.1.16 is available (you have 2.0.7.1)
rdflib==3.2.3 is available (you have 3.0.0)
recaptcha-client==1.0.6 is available (you have 1.0.5)
receipts==0.2.5 is available (you have 0.2.4.1)
requests==1.1.0 is available (you have 0.14.0)
simplejson==3.1.0 is available (you have 2.3.2)
suds==0.4 is available (you have 0.3.9)

Comment 1

[Wil Clouser [:clouserw]]

This bug is ripe for splitting up.  If you upgrade one or two, file a new bug and assign it to yourself.

Comment 2

[Mathieu Pillard [:mat]]

The number of outdated apps have now doubled. I'll be trying to eliminate a few... I already stumbled upon https://github.com/gawel/pyquery/issues/6 and https://github.com/gawel/pyquery/issues/40 , which prevent us to update pyquery.

Comment 3

[Mathieu Pillard [:mat]]

Took a first stab at some that looked easy / straightforward / harmless : https://github.com/mozilla/zamboni/pull/990

Comment 4

[Mathieu Pillard [:mat]]

According to discussion in the PR, https://github.com/kennethreitz/requests/issues/749#issuecomment-19284753 prevents us from upgrading to requests 1.2.1 or higher.

Comment 5

[Mathieu Pillard [:mat]]

More outdated packages fixed in https://github.com/mozilla/zamboni/commit/00b40589773dbe9edeff197fc5cf001806536f81

There is still a lot of work to do...

Comment 6

[Rob Hudson [:robhudson]]

Updated a few outdated celery packages (& dependencies):
https://github.com/mozilla/zamboni/commit/89e8cfa

Comment 7

[Mathieu Agopian [:magopian]]

Updated most of the outdated packages: PR opened https://github.com/mozilla/zamboni/pull/1538

Extended report is here: https://gist.github.com/magopian/d4b6d497d0a9a3b67362

Comment 8

[Mathieu Agopian [:magopian]]

PR 1538 merged in https://github.com/mozilla/zamboni/commit/c78a34dbb63aae48a0b3d8bdeceacc931362de05

Comment 9

[Kevin Ngo [:ngoke]]

Some failing tests related to Bleach https://ci-addons.allizom.org/job/marketplace/5535/

Comment 10

[Mathieu Agopian [:magopian]]

I'm pretty sure those failures happen because jenkins doesn't have the up to date requirements (especially bleach). Is it possible? How are we supposed to fix that?

I ran a few failing tests locally, and they pass without a glitch (and fail with bleach 1.1.5 instead of 1.2.2).

Comment 11

[Mathieu Agopian [:magopian]]

With updated requirements, the whole mkt jenkins build now fails: https://ci-addons.allizom.org/job/marketplace/5542/console

This is because there's no STATIC_URL in the jenkins settings, and django-browserid (the updated version) requires one.

@mat thus downgraded django-browserid back to 0.8 (the previous used version).

Comment 12

[Wil Clouser [:clouserw]]

it looks like STATIC_URL is just "" according to https://github.com/mozilla/zamboni/blob/master/scripts/run_mkt_tests.sh#L86 .  You can change that file and build.sh to adjust how jenkins runs, afaik.

Comment 13

[Wil Clouser [:clouserw]]

After running pip-review I'm not sure we'll ever be able to close this bug.  Perhaps we should review/update the packages quarterly or something?  Anyone have a good upgrade strategy?

Comment 14

[Mathieu Pillard [:mat]]

We were discussing this with magopian on IRC, I think we need a much shorter review period. Something like every month minimum. And we need better tools than just barebone pip-review (I know magopian has some he needs to clean up and share). 

But I think it's doable, it's just that the first steps are very painful because it hasn't been done in a long time. I also think it's valuable, especially because a lot of these packages don't get backported security fixes, this means the best way to stay secure is to stay on top of the updates.

Comment 15

[Mathieu Agopian [:magopian]]

check https://requires.io (thanks Rob for the tip ;)

Comment 16

[Mathieu Agopian [:magopian]]

Here's a report from https://requires.io/github/magopian/zamboni/requirements/?branch=master

The more requirements we can switch from a github link to a PyPI requirement, the more accurate the report!

What's missing is a view of what blocks upgrading a specific package, and the perceived "risk".

Comment 18

[Dethe Elza [:dethe]]

Closing this because it is an eternal issue, but in lieu of the bug we should have a policy about periodically updating dependencies.

Comment 19

[Mathieu Pillard [:mat]]

Re-opening because we haven't decided on a policy and it has gotten worse. https://requires.io/ is a good start to get an overview.

Comment 20

[Mathieu Pillard [:mat]]

Some easy ones taken care of in https://github.com/mozilla/zamboni/commit/113432198f4e28560a3a74e6f66ce5b7663e50c0

Comment 21

[Mathieu Pillard [:mat]]

More: https://github.com/mozilla/zamboni/commit/8761c9500c1c6290c6a75b5159863bb2801bb00c

Comment 22

[Rob Hudson [:robhudson]]

ES lib updates:
https://github.com/mozilla/zamboni/commit/29f36da

Comment 23

[David Durst [:ddurst]]

Well, it was fixed to a point. But this is really just ongoing maintenance.