1218451 - Intermittent test failures on stage for newsletter form submission

Status: RESOLVED WONTFIX

(www.mozilla.org :: Pages & Content, defect)

Description

[Alex Gibson [:agibson]]

We recently added a couple of Selenium tests for newsletter submission to bedrock:

https://github.com/mozilla/bedrock/blob/master/tests/functional/test_contribute.py#L36

https://github.com/mozilla/bedrock/blob/master/tests/functional/test_newsletter.py#L22

Since their creation we've been seeing some intermittent test failures for these two tests when running against stage:

https://webqa-ci.mozilla.com/job/bedrock.stage/16/
https://webqa-ci.mozilla.com/job/bedrock.stage/25/
https://webqa-ci.mozilla.com/job/bedrock.stage/27/

Looking at the screenshots in the HTML reports, it appears as if these failures could be genuine. Instead of the expected success message, the user sees an error returned from the server reading:

"We are sorry, but there was a problem with our system. Please try again later!"

Comment 1

[Jessilyn Davis]

Thanks agibson!

2 Questions for you:

1) Are these still happening? 

2) What's the timezone of the alert?

The 3 timeouts reported above look like:

Oct 22, 2015 12:03:05 PM
Oct 23, 2015 12:48:46 PM
Oct 24, 2015 2:51:00 AM


I'm going to compare this with queries/filters we have run in our ExactTarget account to see if something there is causing timeouts to Basket (a la Bug 1179496), causing it to fail.


Upon first glance, the October Firefox + You newsletter launched across the 8 languages Oct 22-24:


EN (LARGEST) 10/22/2015 6:43 AM  PT
ID 10/22/2015 7:06 PM PT
RU 10/22/2015 10:11 PM PT
DE 10/23/2015 1:09 AM PT
FR 10/23/2015 1:36 AM PT
PT 10/23/2015 3:36 AM PT
PL 10/24/2015 1:08 AM PT
ES 10/24/2015 3:08 AM PT


If there is significant traffic to our email preference center after an email send (which is likely) - I *think* Master_Subscriber table is locked when someone has updated their preferences, which is the same table that's queried when someone subscribes. Pmac, can you confirm?


Also - it's great to have these tests! (Case in point this bug in finding things that are not working well.) However, I want to make sure we're not burning email sends (and skewing live data engagement results, as well as padding our subscriber tables with test data) for these automated tests. Can you tell me more about how these tests work?

Comment 2

[Alex Gibson [:agibson]]

The tests currently run once per day, on both dev and stage. They are marked as destructive, so do not run against production. Each test sends submits with the email `[e-mail]`.

Example said test is here:

https://github.com/mozilla/bedrock/blob/master/tests/functional/test_newsletter.py#L23-L31

hope that helps.

Comment 3

[Alex Gibson [:agibson]]

Also, yes they are still happening although maybe slightly less frequently. This is the latest one:

https://webqa-ci.mozilla.com/job/bedrock.stage/40/

Comment 4

[Jessilyn Davis]

Thanks agibson! QQ: what's the timezone for the errors?

(Latest one for my records in this bug:
Oct 28, 2015 9:24:14 PM )

Comment 5

[Alex Gibson [:agibson]]

(In reply to Jessilyn Davis from comment #4)
> Thanks agibson! QQ: what's the timezone for the errors?
> 
> (Latest one for my records in this bug:
> Oct 28, 2015 9:24:14 PM )

Just checked with Stephen Donner, and their Jenkins instance's timezone set to America/Los_Angeles.

Comment 6

[Jessilyn Davis]

Cool - it doesn't seem to be correlated with the email newsletter send.

agibson, how often is this happening?  

Ben, does this look like any of these timeouts were happening during FUEL queries/automations?


Trying to get a sense of what's causing it + the impact to assess urgency. We're going to be doing a technical audit of our email platform system this quarter, and it would be good to add this use-case to the list of issues to investigate fixing/preventing.

Comment 7

[Alex Gibson [:agibson]]

> agibson, how often is this happening?

Currently seeing these tests fail every couple of days. Only seems to be failing against stage, dev seems fine.

Comment 8

[Jessilyn Davis]

Stage & Prod use the same Production ExactTarget account. Dev uses a sandbox ET account. So that makes sense.

Are there tests running on prod too?

Good to know it's not urgent/happening all the time - but it would be good to find the pattern and see what is causing it.

Comment 9

[Alex Gibson [:agibson]]

> Are there tests running on prod too?

The tests aren't running on prod as they are marked as destructive (i.e. they submit form data). We could probably schedule some tests to run there too if it helps, although I suspect it will be the same as stage seeing as they use the same ET account.

Comment 10

[Ben Niolet]

Jessilyn, 

These failures shouldn't have anything to do with FUEL since those automations run every three hours and the staging errors are happening every few days. Also we're not seeing any failures on Prod and I would expect that if FUEL was driving this.

Comment 11

[Jessilyn Davis]

Solid. Thanks Ben! That was what I figured, but wanted to be sure/get it in the bug.

Comment 12

[Alex Gibson [:agibson]]

(In reply to Ben Niolet from comment #10)
> Jessilyn, 
> 
> These failures shouldn't have anything to do with FUEL since those
> automations run every three hours and the staging errors are happening every
> few days. Also we're not seeing any failures on Prod and I would expect that
> if FUEL was driving this.

Please note, the tests aren't running against Prod (see Comment 9). They also don't run continuously, but on average once/twice per day. 

(I'm assuming when you say failures, you mean the test failures and not something you're seeing in Exact Target.)

Comment 13

[Paul [:pmac] McLanahan]

This may be a manifestation of some of the timeout errors we were seeing during and after the PHX1 datacenter exit. Are these still happening regularly? There are quite a few things that could be going on, but it may need to happen more regularly to track it down :(

Comment 14

[Alex Gibson [:agibson]]

Last failure we saw was on 9 Nov: https://webqa-ci.mozilla.com/job/bedrock.stage/86/

I'll keep an eye out for any further occurrences, and not here if/when the happen. Thanks pmac!

Comment 15

[Alex Gibson [:agibson]]

Sorry, that should have read "and note here if/when they happen"

Comment 16

[Alex Gibson [:agibson]]

Got another one today: https://webqa-ci.mozilla.com/job/bedrock.stage/110/

Comment 17

[Paul [:pmac] McLanahan]

I've taken a deeper look and I still have no clue :( Is it possible to have these retry once or twice before reporting failure? I'll continue looking however.

Comment 18

[Alex Gibson [:agibson]]

(In reply to Paul [:pmac] McLanahan from comment #17)
> I've taken a deeper look and I still have no clue :( Is it possible to have
> these retry once or twice before reporting failure? I'll continue looking
> however.

I'll talk to Dave Hunt to see if this is something we can accommodate.

Comment 19

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The problem with automatically retrying is that it paves over a real failure. If the noise of the intermittent failure is too high, then I would either suggest removing the test or skipping it for the affected environment(s). If there's more information we can provide to help determine the cause of this failure, let me know.

Comment 20

[Paul [:pmac] McLanahan]

My opinion on this one is that there are so many things that might cause basket to respond with an error that a failing test isn't really informative. If it fails 3 times in a row then there's a real problem that we need to know about.

Comment 21

[Alex Gibson [:agibson]]

While basket failures are one thing, I think these tests are still important to catch breakages in our front-end code. Perhaps we would be better to limit these tests to run only against dev, where we don't seem to see these failures.

Comment 22

[Alex Gibson [:agibson]]

Attachment: GitHub pull request to skip these tests on stage env

Comment 23

[Alex Gibson [:agibson]]

Before we resort to disabling these tests on stage we're going to try and rerun if a failure occurs.

Comment 24

[Alex Gibson [:agibson]]

Attachment: GitHub pull request

Comment 25

[Alex Gibson [:agibson]]

I'm going to close this bug as wontfix, as I think it has likely run its course for now.

To recap on the outcome here:

We added a marker (@pytest.mark.flaky(reruns=1)) to basket-dependent tests, which tells Jenkins to rerun the test a maximum of 1 time should the test fail. Since then we haven't really seen any alerts to indicate that the form fails twice in succession, which seems to fall under acceptable criteria based on the discussion here.

I should note that the intermittent failures themselves have not been fixed, we're just suppressing our alerting for these failures.

Thanks all!