1882199 - Verify that systemtests are using the intended backend

Status: RESOLVED FIXED

(Socorro :: Antenna, task)

Description

[Daniel Thorn [:relud]]

responding to :willkg in https://github.com/mozilla-services/antenna/pull/974#pullrequestreview-1901929506:

The only issue I have is that looking at the CI output--there's no way to verify that the environment configuration changes actually cause antenna to be configured with pubsub in the second systemtest run. If we typo'd the CRASHMOVER_CRASHPUBLISH_CLASS variable, we'd see the same output, but it wouldn't be using pubsub. I'm not sure how to urge pytest into spitting something else to output short of writing a pytest plugin. I'll think about that and if I think of anything, we can do a followup PR.

We need to verify that when we run systemtests we are sure they ran with gcs/s3/sqs/pubsub in a way that is visible to someone manually reviewing CI logs.

Here's a few ways I can think of to do it:

• add a pytest flag like in ingestion-edge load tests to control what queue/storage backend to check during tests.
• edit the docker-compose.yml to remove the backends that aren't in use, so that we can see what services docker compose brings up, and expect the tests to fail due to unreachable services if incorrectly configured.
  • if executing this option, make sure to bring all the services down first, then remove unused, then let the setup and systemtest commands bring up dependencies, and use git to revert any local changes afterward
• write the backends used to a file, and cat that file after running systemtests (so that it isn't hidden by pytest)

:willkg feel free to suggest other options, or recommend one that you would prefer here.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I like the idea of a pytest flag as long as it's clear from the test output what the setup was.

I like the second idea, but I'd adjust it to be less complicated by running docker compose down ... like this:

      - run:
          name: Run systemtest with pubsub
          command: |
            echo 'CRASHMOVER_CRASHPUBLISH_CLASS=antenna.ext.pubsub.crashpublish.PubSubCrashPublish' >> my.env
            docker compose run --rm ci shell ./bin/run_setup.sh
            # bring up backing services and ci-web
            docker compose up --detach --wait --wait-timeout=10 ci-web
            # shut down backing services we don't want running
            docker compose down sqs
            docker compose run --rm ci shell bash -c 'cd systemtest && NGINX_TESTS=0 POST_CHECK=1 HOST=http://ci-web:8000 pytest -vv'
            sed '$d' -i my.env # remove config on last line

I think I don't like writing things to a file and then verifying, but that is a way to get additional information out without pytest interfering.

I wonder if writing "indicator tests" would help. Something like this:

@pytest.mark.skipif( some test which is true if pubsub is not enabled )
def test_pubsub_enabled():
    pass

Then we can see what was set up based on the test output and what got skipped. Then we'd see this in the output:

SKIPPED [1] test_pubsub_enabled:100: pubsub is not enabled

It's a little goofy and we'd have to make sure the wording makes sense, but I think these would be straightforward to write and don't require anything fancy.

Maybe doing a combination of docker compose down ... and pytest options or indicator tests would be good?

Comment 2

[Daniel Thorn [:relud]]

Attachment: [mozilla-services/antenna] bug-1882199: Verify that systemtests are using the intended backend (#979)

okay, here's an option: parameterize the fixture and skip when the param doesn't match the configured backend, showing the tested backend as part of the test name

Comment 3

[Daniel Thorn [:relud]]

bonus: we know this works because it exposed that only sqs tests were running (fixed now)

Comment 4

[Daniel Thorn [:relud]]

relud merged PR [mozilla-services/antenna]: bug-1882199: Verify that systemtests are using the intended backend (#979) in 121b574.