1261725 - Tox tests are slow with coverage

Status: RESOLVED WONTFIX

(Release Engineering Graveyard :: Applications: Balrog (backend), defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

While doing some development I noticed that the tests seem to run slower under tox than the old Makefile/nose call. And that there seems to be migrate output whenever you get a test failures. I don't recall the latter, so I'm making a tenuous connection between the two. 

Could be completely bogus, but can't the DB be created directly with the latest schema, and do one migration test elsewhere (we're probably doing this already).

Comment 1

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #0)
> While doing some development I noticed that the tests seem to run slower
> under tox than the old Makefile/nose call. And that there seems to be
> migrate output whenever you get a test failures. I don't recall the latter,
> so I'm making a tenuous connection between the two. 
> 
> Could be completely bogus, but can't the DB be created directly with the
> latest schema, and do one migration test elsewhere (we're probably doing
> this already).

I looked into this a bit and it looks like enabling coverage slows things down, especially when doing so with pytest in tox. A few quick tests I ran:
py.test (no tox): 27s
py.test (tox): 28s
py.test+cov (no tox): 32s
py.test+cov (tox): 47s
nose (no tox): 23s
nose (tox): 23s
nose+cov (no tox): 29s
nose+cov (tox): 29s

In all of the above cases, we're using migrate create the db (I verified by making the admin CSRF test fail and checking the log output). So, py.test is clearly slower than nose, and it seems to have particular issues with coverage inside of tox. I haven't dug into this any further.

We do have a schema vs. model test (https://github.com/mozilla/balrog/blob/master/auslib/test/test_db.py#L1775), so I think it's fine to move the other ones away from migrating their db, though it's not clear how much that would help here.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Thanks for digging, the numbers are very helpful. I think this is just 'suck it up Nick' --> WONTFIX.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

https://pypi.python.org/pypi/detox lets you run multiple test environments in parallel. It's nice win for me but once we drop py26 it'll evaporate.