1331726 - [e2e-tests] test_public_api_navigation is sporadically failing

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Matt Brandt [:mbrandt]]

test_public_api_navigation sporadically fails on both the stage and proud test jobs.

https://github.com/mozilla/socorro/blob/master/e2e-tests/tests/test_api.py#L88

>    e2e-tests/tests/test_api.py:88: in test_public_api_navigation
>        assert len(first_dates) == len(signatures)
>    E   assert 4 == 5
>    E    +  where 4 = len({'InvalidArrayIndex_CRASH | nsTArray_Impl<T>::ElementAt | nsCommandManager::CommandStatusChanged': '2016-09-26T11:49:1....so@0x4c859': '2016-01-14T16:14:02.753060+00:00', 'nsMsgDBView::OnDeleteCompleted': '2012-06-25T07:52:54.144540+00:00'})
>    E    +  and   5 = len({'18e696b6-17ec-4761-b806-cb1bb2170110': 'nsMsgDBView::OnDeleteCompleted', '8f423b83-0170-48b4-b6d5-939242170111': 'In...-a8ed-47b4-a105-d00c62170110': 'libstagefright.so@0x4c859', 'c4f81c33-90c2-48d0-93de-b1d572170110': 'OOM | small', ...})

Comment 1

[Matt Brandt [:mbrandt]]

I've added a skip while we investigate the cause of the failure.

https://github.com/mozilla/socorro/pull/3652

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Adrian says that this is checking for something that may or may not be true, but there's no good way to know what the exact result should be. He suggests we change it to >= 1.

Further, he pontificated about whether this check is important to the rest of the tests. We might want to break this big API test into multiple tests.

I'll look into both of those today.

Comment 3

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/21582a4de4d942d2eed9db1af4f0dac1a40f38a7
Fixes bug 1331726 - fix assertion in test_public_api_navigation (#3660)

This fixes the assertion that we should have at least one signature first date
for each signature--that turns out to not necessarily be the case. I changed it
to make sure it's > 0. Hopefully we don't eventually find ourselves in a
situation where it's 0, though I suspect that's possible.

While doing that, I changed some variable names to make the code easier to read
and I added comments to summarize what's about to happen so it's easier to
verify the test is doing what it should be doing.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Going back to comment #2, we can't really break up the big test because it's carrying information it discovers forward to later steps. Breaking it up wouldn't be helpful.