1181879 - Update to Django v1.8

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P3)

Description

[Ed Morley [:emorley]]

Bug 1181836 will update us to Django 1.7.9.

At some point later we'll need to/want to update to Django 1.8, but that will presumably be more involved.

https://docs.djangoproject.com/en/1.8/howto/upgrade-version/
https://docs.djangoproject.com/en/1.8/internals/deprecation/

https://docs.djangoproject.com/en/1.8/releases/1.8/
https://docs.djangoproject.com/en/1.8/releases/1.8.1/
https://docs.djangoproject.com/en/1.8/releases/1.8.2/
https://docs.djangoproject.com/en/1.8/releases/1.8.3/

Comment 1

[Ed Morley [:emorley]]

Running the tests locally with Django 1.8.3 actually passes all tests but one, I was expecting this to be worse :-)

tests/model/sql/test_models.py:59:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

/usr/lib/python2.7/contextlib.py:17: in __enter__
    return self.gen.next()
tests/model/sql/test_models.py:32: in assert_num_queries
    _old_debug_cursor = connection.use_debug_cursor
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _


self = <django.db.DefaultConnectionProxy object at 0xb6b59fec>, item = 'use_debug_cursor'

    def __getattr__(self, item):
>       return getattr(connections[DEFAULT_DB_ALIAS], item)
E       AttributeError: 'DatabaseWrapper' object has no attribute 'use_debug_cursor'

Comment 2

[William Lachance (:wlach)]

Django 1.8's new UUIDField would be nice to use with Perfherder once we use the ORM for that (bug 1192976), as it would allow us to use the signature's uuid as a primary key. 
https://github.com/mozilla/treeherder/pull/870#discussion-diff-36965310R554

(there are probably other places in the Treeherder reference schema where it would be good to do this)

I wouldn't say this "blocks" bug 1192976, but maybe we should just do this now, since we no doubt will want to upgrade eventually and it might not be that much work.

Comment 3

[William Lachance (:wlach)]

(In reply to Ed Morley [:emorley] from comment #1)
> Running the tests locally with Django 1.8.3 actually passes all tests but
> one, I was expecting this to be worse :-)
> 
> tests/model/sql/test_models.py:59:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> 
> /usr/lib/python2.7/contextlib.py:17: in __enter__
>     return self.gen.next()
> tests/model/sql/test_models.py:32: in assert_num_queries
>     _old_debug_cursor = connection.use_debug_cursor
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> 
> 
> self = <django.db.DefaultConnectionProxy object at 0xb6b59fec>, item =
> 'use_debug_cursor'
> 
>     def __getattr__(self, item):
> >       return getattr(connections[DEFAULT_DB_ALIAS], item)
> E       AttributeError: 'DatabaseWrapper' object has no attribute
> 'use_debug_cursor'

Apparently use_debug_cursor has been changed to force_debug_cursor in 1.8. Changing that causes these tests to pass.

However, I noticed some additional issues:

* Some weird problems in test_buildapi, looks like some kind of weird interaction between new relic and our unit testing environment. Probably not that hard to fix.
* Problems creating initial database on vagrant up (possibly fixable by using actual migrations to create the reference data, which I believe :mdoglio is working on)

Comment 4

[William Lachance (:wlach)]

Attachment: Patch to fix connection cursor issue

Just posting this in case someone finds it helpful.

Comment 5

[Ed Morley [:emorley]]

(In reply to William Lachance (:wlach) from comment #3)
> * Some weird problems in test_buildapi, looks like some kind of weird
> interaction between new relic and our unit testing environment. Probably not
> that hard to fix.

Do you have some log output for this? Happy to take a glance at it.

Comment 6

[Ed Morley [:emorley]]

Was it similar to https://pastebin.mozilla.org/8843129 ? If so, this isn't Django 1.8 specific, but some strange interaction between New Relic and some of our tests, that is only seen when people re-provision after New Relic was added to requirements/common.txt.

Comment 7

[William Lachance (:wlach)]

(In reply to Ed Morley [:emorley] from comment #6)
> Was it similar to https://pastebin.mozilla.org/8843129 ? If so, this isn't
> Django 1.8 specific, but some strange interaction between New Relic and some
> of our tests, that is only seen when people re-provision after New Relic was
> added to requirements/common.txt.

Yes, it was similar to that IIRC. :)

Comment 8

[Mauro Doglio [:mdoglio]]

I'm pretty sure we also need to check that are custom django commands are still working (see https://docs.djangoproject.com/en/1.8/howto/custom-management-commands/#custom-commands-options)

Comment 9

[Mauro Doglio [:mdoglio]]

Let's upgrade to https://docs.djangoproject.com/en/1.8/releases/1.8.4/ directly

Comment 10

[Mauro Doglio [:mdoglio]]

Attachment: PR 886

Comment 11

[William Lachance (:wlach)]

Comment on attachment 8649850 [details] [review]
PR 886

lgtm, I assume you've tested the management commands, vagrant provisioning, etc?

Comment 12

[Mauro Doglio [:mdoglio]]

I actually found some issues with migrations that (for reasons I ignore) didn't show up in my previous tests. I'm digging into it

Comment 13

[Mauro Doglio [:mdoglio]]

All the issues solved, the branch has been sitting on staging for 1/2 hour now, I'm merging into master

Comment 14

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/1ab14030ce38d1952a9a3564551cb746411b8709
Bug 1181879 - upgrade django to 1.8.4

Comment 15

[Ed Morley [:emorley]]

I just tried to rollback a prod push (something broke that was working on stage; though unrelated to this bug) and it looks like downgrading is not possible:
http://treeherderadm.private.scl3.mozilla.com/chief/treeherder.prod/logs/66ee095257552ba2c7b830eba408cb474094cd7b.1440102141

How would one work around this for future reference?
I'd always thought it was possible to downgrade - if not, updating django is more risky than I thought.