1909870 - Implement new set of system tests

Status: NEW

(Tecken :: General, task, P2)

Description

[Sven Marnach :sven]

The current "system tests" for Tecken test interesting aspects of the public API, but they are not very systematic or complete. The codebase is also a bit hard to extend, since the tests are written as a set of interacting standalone scripts that invoke each other.

To get a more complete, systematic and maintainable set of system tests, we would like to start from scratch using pytest as the test framework. pytest provides a lot of flexibility when running the tests, and has built-in result reporting, so we can focus on writing the actual tests.

We plan to rename the old systemtests directory in the Tecken repository to smoketests, or alternatively to split them out to a separate repository.

We will need a bit of infrastructure to get going:

• A wrapper script to conveniently run the tests.
• A Tecken client implementation supporting Tecken's complete public API, including retry behaviour for 500s and 429s.
• Example symbol files and zip archives stored in a publicly available place, e.g. a GCS bucket, so we can test the upload-by-download functionality with a sufficiently big file.
• A way to generate fake symbol files and zip archives on the fly.

Comment 1

[Sven Marnach :sven]

Attachment: [mozilla-services/tecken] bug-1909870: Rename old system tests to smoke tests. (#2972)

Comment 2

[Sven Marnach :sven]

Attachment: [mozilla-it/webservices-infra] bug-1909870: Add a GCS storage bucket for the Tecken system tests. (#2468)

Comment 3

[Sven Marnach :sven]

smarnach merged PR [mozilla-services/tecken]: bug-1909870: Rename old system tests to smoke tests. (#2972) in 6e3dc91.

Comment 4

[Sven Marnach :sven]

smarnach merged PR [mozilla-it/webservices-infra]: bug-1909870: Add a GCS storage bucket for the Tecken system tests. (#2468) in d8b0922.

Comment 5

[Sven Marnach :sven]

Attachment: [mozilla-services/tecken] bug-1909870: Implement a new system test suite. (#2974)

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

smarnach merged PR [mozilla-services/tecken]: bug-1909870: Implement a new system test suite. (#2974) in 41803b8.

Comment 7

[Sven Marnach :sven]

Attachment: [mozilla-it/webservices-infra] bug-1909870: Expire system test uploads after a week. (#2654)

Comment 8

[Sven Marnach :sven]

smarnach merged PR [mozilla-it/webservices-infra]: bug-1909870: Expire system test uploads after a week. (#2654) in 7eb6707.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

Everything up to this point was deployed to prod in bug #1922112 just now. Tagging Sven to verify.

Comment 10

[Sven Marnach :sven]

The changes so far either affected the new GCS environment or the new systemtest directory in the tecken repo. Neither affect the AWS deployment, so there's nothing to verify.