1339532 - make backend tests faster

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P3)

Description

[[e-mail] (:bhearsum)]

The Balrog backend tests have gotten quite a bit slower as we've added more and more to them. We'll soon have close to 1,000 tests, and even on a fast machine they can take up to 5 minutes to run. We already parallelize them with pytest-xdist, but we probably do a LOT of extra work for each test because we rebuild the database in setUp(). That can probably be done once per class instead of once per test. We'd still need to populate the database per test to avoid tests stomping on one another, but that part probably isn't very expensive in comparison.

Catlee suggested that pytest fixtures (http://doc.pytest.org/en/latest/fixture.html) would probably make this pretty easy.

Comment 1

[[e-mail] (:bhearsum)]

Olasode, are you still planning to look at this?

Comment 2

[Olasode Abdul-Mumeen]

Yes, I have been on it for A while but I have not been able to see any change in the speed of the tests with my implementations. I think I will probably raise a PR so you can take a look at what I have.

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Olasode Abdul-Mumeen from comment #2)
> Yes, I have been on it for A while but I have not been able to see any
> change in the speed of the tests with my implementations. I think I will
> probably raise a PR so you can take a look at what I have.

Are you still looking at this, Olasode?

Comment 4

[Olasode Abdul-Mumeen]

Hi Ben,
I am sorry I didn't reach out earlier. My schedule leaves no enough time to work on this. I think I will have to release it and pick up another bug in coming weeks when my schedule clears up.
Thanks.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Olasode Abdul-Mumeen from comment #4)
> Hi Ben,
> I am sorry I didn't reach out earlier. My schedule leaves no enough time to
> work on this. I think I will have to release it and pick up another bug in
> coming weeks when my schedule clears up.
> Thanks.

That's OK, the prototyping you already did is a great proof of concept - thank you for that!

Comment 6

[[e-mail] (:bhearsum)]

Note to the next person that looks at this: https://github.com/mozilla/balrog/pull/333 is a good starting point.

Comment 7

[Rachael]

Hi Ben
Noticed that when running the test files, docker building  has to happen in each instance of running the tests.Is there a possibility of having the build only once, cause i think the builds also contribute to the long periods the tests takes to run?

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Rachael from comment #7)
> Hi Ben
> Noticed that when running the test files, docker building  has to happen in
> each instance of running the tests.Is there a possibility of having the
> build only once, cause i think the builds also contribute to the long
> periods the tests takes to run?

I'm surprised if the Docker build isn't fully cached (unless something big is changed) -- if you have some output to share that might be helpful in figuring out why. If there's any issues with the Docker part, we should fix that. But we should also speed up the tests themselves as much as possible (through fixtures or other means).

Comment 9

[Rachael]

(In reply to Ben Hearsum (:bhearsum) from comment #8)
> (In reply to Rachael from comment #7)
> > Hi Ben
> > Noticed that when running the test files, docker building  has to happen in
> > each instance of running the tests.Is there a possibility of having the
> > build only once, cause i think the builds also contribute to the long
> > periods the tests takes to run?
> 
> I'm surprised if the Docker build isn't fully cached (unless something big
> is changed) -- if you have some output to share that might be helpful in
> figuring out why. If there's any issues with the Docker part, we should fix
> that. But we should also speed up the tests themselves as much as possible
> (through fixtures or other means).

Docker issues sorted the tests use cached images unless something changes thats when they have to be copied over

Comment 10

[[e-mail] (:bhearsum)]

We haven't been able to make any progress on this in the PR. It's not clear to me that fixtures (at least they way they're being used in the two PRs that were tried for this bug) are going to help.

I truly appreciate your efforts here Rachael, but because this is a low priority bug, and the path forward isn't clear, I'm not going to be able to mentor this any further - so I'm unassigning it.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

https://github.com/mozilla/balrog/pull/709 was another approach.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

catlee implemented fixtures in https://github.com/mozilla/balrog/pull/840 and the speed the tests up considerably.