695105 - Mozillians dev/trunk staging server suffers from timeouts/Zeus "Service Unavailable" error pages

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Zip file of the Selenium test report for Mozillians

Our Mozillians prod and stage Selenium-automation runs don't fail like our mozillians.trunk one [1] does; it's really, really unreliable (we hit a Zeus "Service Unavailable" page quite a bit).

Attached is an HTML report from our Jenkins (download the folder and open index.htm).

[1] http://qa-selenium.mv.mozilla.com:8080/view/Mozillians/job/mozillians.trunk/ -- should be on Mozilla-MV, if you're remote.

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Screenshot of jegged build history

Comment 2

[Jason Thomas [:jason]]

How often do these tests run on dev environment? I believe that these 'Service Unavailable' occur due to the backends not responding quick enough after a restart/reload of the apache httpd or wsgi daemon which continuously occurs on dev.

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Jason Thomas [:jason] from comment #2)
> How often do these tests run on dev environment? I believe that these
> 'Service Unavailable' occur due to the backends not responding quick enough
> after a restart/reload of the apache httpd or wsgi daemon which continuously
> occurs on dev.

We have a cron set for:

*/30 09-18 * * 1-5

If we can do either graceful restarts (I believe AMO just "touch"es the dir/files, making Apache not restart as often, etc.), or work around when it's updating, we can change the cron to do so.  Or, perhaps a post-commit trigger would be even better?

Comment 4

[Jason Thomas [:jason]]

On the generic cluster we currently touch the wsgi file every time the update runs (every 5 mintutes), which would cause wsgi daemon to reload on a new request.

I have updated the script to only touch the wsgi file for mozillians-dev when the code updates.

Hopefully this will help with the issue.

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Jason Thomas [:jason] from comment #4)
> On the generic cluster we currently touch the wsgi file every time the
> update runs (every 5 mintutes), which would cause wsgi daemon to reload on a
> new request.
> 
> I have updated the script to only touch the wsgi file for mozillians-dev
> when the code updates.
> 
> Hopefully this will help with the issue.

This seems a lot better now, Jason; thanks.  Do you know if Affiliates does the same thing, too?  If so, can you look at a fix for that?

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; thanks.