636173 - Getting Service Unavailable during test runs doing searches

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[David Burns :automatedtester]

While doing an empty search(litmus 13847) and a unicode search(litmus 13697) we are getting service unavailable.

Our automated tests took screenshots of the failures

http://qa-selenium.mv.mozilla.com:8080/job/input%20stage/349/artifact/input.stage.mozilla.com_1298477413.png 

http://qa-selenium.mv.mozilla.com:8080/job/input%20stage/349/artifact/input.stage.mozilla.com_1298477448.png

Comment 1

[Fred Wenzel [:wenzel]]

CCing cshields: QA seems to be getting this kind of error relatively frequently lately. As you can see from the styling of the error message, this is not an error directly in the app, but thrown by Zeus. Any idea when Zeus would bring up this kind of page, and what could cause this in our case?

Do you guys have any graphs on the load of the staging boxes? Perhaps something's going on on one of the affected servers that takes it down for a limited amount of time. This particular build was kicked off at 8am[1], so perhaps some cron job ran.

[1] http://qa-selenium.mv.mozilla.com:8080/job/input%20stage/349/

Comment 2

[Fred Wenzel [:wenzel]]

Punting this to 3.4 for now. It's not an error in the app so shouldn't delay this milestone.

Comment 3

[David Burns :automatedtester]

http://qa-selenium.mv.mozilla.com:8080/job/input%20stage/396/ 7 tests failing because of this...

Comment 4

[Matt Brandt [:mbrandt]]

+1 Our automated tests are failing 3 out of 4 runs due to this.

Comment 5

[David Burns :automatedtester]

Its more 1 out of 2. Of the last 10 runs 6 have hit this issue.

http://qa-selenium.mv.mozilla.com:8080/job/input%20stage/476/
http://qa-selenium.mv.mozilla.com:8080/job/input%20stage/473/
http://qa-selenium.mv.mozilla.com:8080/job/input%20stage/472/
http://qa-selenium.mv.mozilla.com:8080/job/input%20stage/470/
http://qa-selenium.mv.mozilla.com:8080/job/input%20stage/467/
http://qa-selenium.mv.mozilla.com:8080/job/input%20stage/466/

Comment 6

[David Burns :automatedtester]

We are hoping to have a Test Day against this staging server on Tuesday and would be great if we can try stabilize this before hand

Comment 7

[Corey Shields [:cshields]]

Is this related to 644302??

Comment 8

[David Burns :automatedtester]

Think they might be related

Comment 9

[Fred Wenzel [:wenzel]]

(In reply to comment #7)
> Is this related to 644302??

I am working on making some SQL for IT to run that'd reduce the weight of this query, in both prod and stage.

Comment 10

[Corey Shields [:cshields]]

marking this dependent upon 644302, which is pending verification

Comment 11

[Corey Shields [:cshields]]

So is this fixed now?

Comment 12

[David Burns :automatedtester]

We have had over 12 builds on our CI that havent encountered this problem.

Comment 13

[Stephen Donner [:stephend] Not actively reading bugmail]

Marking as verified FIXED based on comment 12, and http://qa-selenium.mv.mozilla.com:8080/job/input%20stage/buildTimeTrend