[e2e-tests] test_public_api_navigation is sporadically failing

RESOLVED FIXED

Status

RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: mbrandt, Assigned: willkg)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [e2e-tests], URL)

(Reporter)

Description

2 years ago
test_public_api_navigation sporadically fails on both the stage and proud test jobs.

https://github.com/mozilla/socorro/blob/master/e2e-tests/tests/test_api.py#L88

>    e2e-tests/tests/test_api.py:88: in test_public_api_navigation
>        assert len(first_dates) == len(signatures)
>    E   assert 4 == 5
>    E    +  where 4 = len({'InvalidArrayIndex_CRASH | nsTArray_Impl<T>::ElementAt | nsCommandManager::CommandStatusChanged': '2016-09-26T11:49:1....so@0x4c859': '2016-01-14T16:14:02.753060+00:00', 'nsMsgDBView::OnDeleteCompleted': '2012-06-25T07:52:54.144540+00:00'})
>    E    +  and   5 = len({'18e696b6-17ec-4761-b806-cb1bb2170110': 'nsMsgDBView::OnDeleteCompleted', '8f423b83-0170-48b4-b6d5-939242170111': 'In...-a8ed-47b4-a105-d00c62170110': 'libstagefright.so@0x4c859', 'c4f81c33-90c2-48d0-93de-b1d572170110': 'OOM | small', ...})
(Reporter)

Comment 1

2 years ago
I've added a skip while we investigate the cause of the failure.

https://github.com/mozilla/socorro/pull/3652
Adrian says that this is checking for something that may or may not be true, but there's no good way to know what the exact result should be. He suggests we change it to >= 1.

Further, he pontificated about whether this check is important to the rest of the tests. We might want to break this big API test into multiple tests.

I'll look into both of those today.

Comment 3

2 years ago
Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/21582a4de4d942d2eed9db1af4f0dac1a40f38a7
Fixes bug 1331726 - fix assertion in test_public_api_navigation (#3660)

This fixes the assertion that we should have at least one signature first date
for each signature--that turns out to not necessarily be the case. I changed it
to make sure it's > 0. Hopefully we don't eventually find ourselves in a
situation where it's 0, though I suspect that's possible.

While doing that, I changed some variable names to make the code easier to read
and I added comments to summarize what's about to happen so it's easier to
verify the test is doing what it should be doing.

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Going back to comment #2, we can't really break up the big test because it's carrying information it discovers forward to later steps. Breaking it up wouldn't be helpful.
You need to log in before you can comment on or make changes to this bug.