522086 - Service Unavailable on https://addons.mozilla.org/en-US/developers/dashboard

Status: RESOLVED WORKSFORME

(mozilla.org Graveyard :: Server Operations, task)

Description

[Alice0775 White]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2b1pre) Gecko/20091013 Firefox/3.5.1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2b1pre) Gecko/20091013 Namoroka/3.6b1pre ID:20091013045938

Go URL and logged in, 
The following error pages are displayed after I was kept waiting in considerably long time.
"Service Unavailable
The service is temporarily unavailable. Please try again later."

Add-on Developer Hub page is displayed when I perform several times reloading.

Reproducible: Sometimes

Steps to Reproduce:
1.Start Namoroka with new profile
2.Go "Add-on Developer Hub page" https://addons.mozilla.org/en-US/developers/dashboard
3.Log in


Expected Results:  
Add-on Developer Hub page is not displayed.The following pages are displayed

Service Unavailable
The service is temporarily unavailable. Please try again later.

Add-on Developer Hub page should be displayed.

Comment 1

[Wil Clouser [:clouserw]]

Looks like there was connectivity problems with mysql around 10:45am?  Do we know why and is it fixed?  The above URL is working for me now.

Comment 2

[Dave Miller [:justdave]]

There were no pages from nagios about databases, and no indications in the logs of anything funny on the master server (which is what the developer section would hit) during that time frame. Munin shows a small (+1 or so) spike in load average around 10:45am, and a small increase in slow queries, so what I can see looks like usage went up a little around that time.  The one thing that might have happened is if the increase in slow queries caused connections to get held longer and max clients got hit.  That metric isn't being tracked by munin (it really ought to be - I filed a bug to make sure we remember to get that set up).

Comment 3

[Dave Miller [:justdave]]

OK, I was mistaken, that metric is being tracked by munin, and we didn't hit it.

So there's basically no record of what went wrong, and nothing we can do to figure it out at this point.