1618942 - [tests] write system test

Status: RESOLVED FIXED

(Tecken :: General, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

I've been using scripts in the tecken-loadtests repository as a system test to make sure critical things are working when we push to stage and prod. The scripts in that repository are defunct at this point. I wrote up bug #1618685 to fix them, but I think instead of doing that, it'd be a better use of time to write a new system test in the tecken repository.

Specifically, a system test needs to meet the following requirements:

1. be able to test stage and prod environments
2. test uploading symbols
3. test upload-by-download
4. test download
5. test symbolication

This bug covers that work.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this to work on now.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

If we end up using taskcluster, then we'll need to use the taskcluster library. Minor details in bug #1593805.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm tackling the symbolication api first. I need:

1. a script that takes a processed crash and generates a stack for symbolication api from it
2. a script that takes a stack, runs it through the symbolication api (version 4 and 5), and "verifies" it in some way

Maybe instead of generating stacks, we generate test cases? It's just that test cases would be environment-specific since symbol storage is environment specific. I don't think I want to maintain stacks for stage and prod. I can't maintain stacks for local dev environments since who knows what they'll have symbols-files-wise.

Given that, maybe verification is something like, "yes, I got back a symbolication response that has the right structure and has some data in it".

I think that's a good mvp. Symbolication is non-destructive, so it can operate on any of the environments.

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/tecken/pull/2015

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2015: bug 1618942: add system test for symbolicate api and downloading sym files

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2015: "bug 1618942: add system test for symbolicate api and downloading sym files" in 77b2345.

Things that still need to be done:

1. write an upload test and an upload-by-download test
2. fix download test to use an optional auth token to download private symbols
3. verify the symbolicate api v4 and v5 response schemas--are they correct? are they complete?

I need to do the first 2. Item 3 can be pushed off to a new bug and deferred until I need it.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2016: bug 1618942: add upload system tests

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2016: "bug 1618942: add upload system tests" in 6ee6231.

I'm going to push off items 2 and 3 to new bugs. I want to be done with this and move on with my life and the myriad of things that are blocked on this.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

Marking this as FIXED.