1305516 - design and implement system test

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Once we finish the bulk of the coding, documentation and testing of the new collector named Antenna, we'll need to set it up on some infrastructure and do a system test.

This issue covers figuring out how we will system test Antenna.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Tagging this with QA-wanted and cc:ing Matt.

Matt: You expressed some interest in doing a system test for the new collector and also a load test. How do you want to go forward with this?

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Matt and I talked about this. We threw details into an etherpad here:

https://public.etherpad-mozilla.org/p/antenna-v1-systemtest

Next steps
==========

1. (willkg) We should have these tests live in the Antenna repository. We can toss
   in a README that says "yo, we ran these once adn they worked on my machine then!"
   We'll put the system tests in tests/systemtests/ and if we write load tests
   we can put them in tests/loadtests/.

2. (willkg) Put them in tests/, move the unit tests down a directory to tests/unittests/
   and fix circle accordingly.

3. (mbrandt) Submit a raw crash to Antenna.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I talked with Matt about this last week. No work has been done on this other than what I did to get the Antenna repository ready for it.

I talked with Miles last week about how to structure the system test such that it can do its work and verify the outcomes are correct. That requires permissions to things I don't have access to. Ultimately, we want the system test to be able to run against -dev and -stage environments so as to verify correctness. We're tossing around making it part of the deployment pipeline such that failures would block pushes to -prod.

This needs more work. I'm going to take it and work on it next week. Matt indicated interest in pair programming on this, so I'll set up some meetings for that.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm working on this now. I didn't end up pair programming with Matt on it for various reasons one of which being that I'm kind of figuring it out as I go along and that's harder to pair program on for a short stint.

The repository for it so far is here:

https://github.com/willkg/jansky

It currently has a cli run mode because that's easier to implement at the moment. After I finish that, I'll add a web run mode.

Then we need to set it up in the -dev environment with access to the same things Antenna -dev has access to.

I figure I've got another week of work to do. That's not wasted effort, though. As work progresses, I'm testing it against my local docker environment and the -dev environment and this will probably surface some bugs in the Antenna code which I'll fix. This gives us a system test we can connect with CI on -dev so that it runs whenever we make changes to the code base.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I nixed jansky and moved it all over to the Antenna repository.

It consists of a py.test suite with a run_tests.sh script that sets up a virtual env, installs requirements, runs the test suite and then reports results via stdout and stderr.

Miles and I talked about how we want to run this in the -dev environment. After he's done with some big issues with Socorro, we'll figure out the details. But it'll be something like "something will launch an ec2 node with the code in the -dev environment, run the run_tests.sh script, capture the stdout and stderr and report that".

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

The work for this is done. Getting it working in the -dev environment is covered in bug #1323464.

Tests are located here: https://github.com/mozilla/antenna/tree/master/tests/systemtest

Marking as FIXED.