520850 - [AMO] Preview.amo is down

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wil Clouser [:clouserw]]

preview.amo has been down for a couple hours.  All it shows is this message:

No suitable nodes are available to serve your request.

Comment 1

[Michael Morgan [:morgamic]]

Is someone looking at this?

Comment 2

[matthew zeier [:mrz]]

Was failing health check on:

Node 10.2.83.54:81: Node pm-app-amo24 has failed - Timeout while waiting for response for https://10.2.80.44:50003/en-US/firefox/statistics/addon/2848/format:rss/key:1c935f183c04ff8f2da80afb03132f74

I don't know how/where it was getting that URL - the check is on /en-US/firefox/.  I turned debugging on on the monitor and it came back on its own.

Comment 3

[chizu]

It's still dead. Zeus gives up on it before requests finish, but they do finish if you connect to apache directly.

Comment 4

[chizu]

A slave database that preview couldn't connect to was added to the config. oremj added in the required firewall rule and it's working now.

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #4)
> A slave database that preview couldn't connect to was added to the config.
> oremj added in the required firewall rule and it's working now.

Thanks; this is indeed working again.

Comment 6

[raymond [:retornam] (inactive)]

http://screencast.com/t/b9qW4sbpQZ

preview is down again

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #6)
> http://screencast.com/t/b9qW4sbpQZ
> 
> preview is down again

It's back up...

Comment 8

[chizu]

(In reply to comment #6)
> http://screencast.com/t/b9qW4sbpQZ
> 
> preview is down again

Looked around at some logs, and I think this was just bad luck with some apache restarts and the zeus test. Please reopen if it's still happening at times.

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; thanks, Trevor (we'll probably just file a new bug).