1316407 - Create a suite of e2e tests for the API

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Matt Brandt [:mbrandt]]

Tracker bug for the API end-to-end smoketests that we would like to create. 

These will likely live in the socorro/e2e-tests directory and utilize the Python requests library. Tests exercise workflows that touch endpoints that do not require authentication.

Comment 1

[Matt Brandt [:mbrandt]]

Adrian/Peter, can you weigh in on which endpoints you'd like tested? If would be helpful if you could identify specific requests with thier params.

Comment 2

[[DEACTIVATED] Adrian Gaudebert]

I believe this is the list of all public endpoints that we use in Socorro and do not plan on deprecating:

* Bugs
* Platforms
* ProcessedCrash
* ProductBuildTypes
* ProductVersions
* RawCrash
* SignatureFirstDate
* SignaturesByBugs
* NewSignatures
* SuperSearch
* SuperSearchFields
* UploadedSymbols (?)

Of them all, here are the most important ones, in decreasing importance order:

* SuperSearch
* SuperSearchFields
* ProductVersions
* RawCrash / ProcessedCrash
* Platforms
* SignatureFirstDate
* Bugs / SignaturesByBugs

Comment 3

[[DEACTIVATED] Adrian Gaudebert]

It's difficult to come up with a clear plan on how to test them. We cannot really rely on the data output, since data can change all the time. Especially with SuperSearch, testing specific features can be difficult. 

Maybe at first we just want to do sanity checks? Just verify that for very generic queries, data gets returned? 

Also, maybe we want to take some time to sit together and hack a plan while we're all in Hawaii?

Comment 4

[Peter Bengtsson [:peterbe]]

What Adrian said. However, I don't think you should anything other than his second list of 8 endpoints. 
I suggest you make a loop where you do something like:

1) Query ProductVersions of all active versions, select out the distinct product names and write down the featured versions. 
(So you end up with something like {'Thunderbird': ['49.0', '50.0b', '51.0a1'], ...}

2) Make a SuperSearch search for each product with a list of versions, make sure there's always a facet by version. Don't bother looking at the numbers. Also, write down the first UUID for each of these searches. So you'll have len(uuids) == len(products)

3) SuperSearchFields should just return non-zero items

4) Query RawCrash and ProcessedCrash with the uuids stored from the SuperSearch. For each uuid, when you query the ProcessedCrash write down the signature and keep that for later. There should now be len(signatures)==len(products)

5) I think you should also query /api/UnredactedCrashes?crash_id=:uuid and expect a 403 Forbidden. 

6) Platforms should just return non-empty results. 

7) For each signature saved, query SignatureFirstDate and there should be a date returned. No need to check the value. 

8) Again, use each signature to query Bugs. Write down all bug IDs found. Then query SignaturesByBugs with all those bug IDs. 

I'd be happy to help some more.

Comment 5

[Matt Brandt [:mbrandt]]

:peterbe has shown interest in writing these tests using pytest and the requests api.

A couple of requirements that will ensure that works with our current testing workflows, ensure the work is compatible with the following pytest packages:
* pytest-base-url
* pytest-xdist

Comment 6

[Peter Bengtsson [:peterbe]]

(In reply to Matt Brandt [:mbrandt] from comment #5)
> :peterbe has shown interest in writing these tests using pytest and the
> requests api.
> 
> A couple of requirements that will ensure that works with our current
> testing workflows, ensure the work is compatible with the following pytest
> packages:
> * pytest-base-url
> * pytest-xdist

I don't know what that means. At all :)

What I would, otherwise, do is write a method that looks like this:

def test_public_api_navigation(self):
    response = requests.get(self.base_url_or_something + '/api/ProductVersions/')
    ...

Comment 7

[Matt Brandt [:mbrandt]]

The variation that you'll need to introduce is relatively small, basically:

def test_public_api_navigation(self, base_url):
    response = requests.get(base_url + '/api/ProductVersions/')
    ...

Comment 8

[Peter Bengtsson [:peterbe]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3617

Comment 9

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/d78bbccdc972ebe37db928233e841095c0a0faa4
fixes bug 1316407 - Create a suite of e2e tests for the API (#3617)

* fixes bug 1316407 - Create a suite of e2e tests for the API

* adding @pytest.mark.nondestructive

* Adjust assert

Tweaking this will give us a better assertion error message when it fails.