909790 - Paas connection issues

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[Andy McKay]

My stackato instance keeps giving back:

https://metaplace.paas.allizom.org/
An error has occurred: {"code":"ECONNREFUSED","errno":"ECONNREFUSED","syscall":"connect"}

Intermittently.

Looking at the stackato console at:

http://api.paas.allizom.org/console/users/amckay%40mozilla.com/apps/metaplace/

It says its started, but also says its down.

This is an instance that I've had running for a while and just ignored.

Comment 1

[Paul [:pmac] McLanahan]

http://newrelic-irc-notify.paas.allizom.org/ is doing the same thing. 

A side note is that it shouldn't be responding at all as the console reports it as down. I've tried shutting it down again from the CLI to no avail. I can start a new instance, but then 2 are running.

Comment 2

[Olivier Yiptong [:oyiptong]]

http://json-ip.paas.allizom.org/ is doing the same thing, additionnally, here's the output from command line tools:

$ stackato stats json-ip
json-ip
+----------+-------------+----------------+--------------+---------------+
| Instance | CPU (Cores) | Memory (limit) | Disk (limit) | Uptime        |
+----------+-------------+----------------+--------------+---------------+
| 0        | 0.0% (2)    | 110.0M (512M)  | 33.3M (2G)   | 2d:1h:27m:24s |
+----------+-------------+----------------+--------------+---------------+
$ stackato instances json-ip

+-------+-------+--------------------+
| Index | State | Start Time         |
+-------+-------+--------------------+
| 0     | DOWN  | 08/27/2013 10:58AM |
+-------+-------+--------------------+
$ stackato instances json-ip

+-------+-------+--------------------+
| Index | State | Start Time         |
+-------+-------+--------------------+
| 0     | DOWN  | 08/27/2013 10:58AM |
+-------+-------+--------------------+
$ stackato instances json-ip

+-------+---------+--------------------+
| Index | State   | Start Time         |
+-------+---------+--------------------+
| 0     | RUNNING | 08/25/2013 09:31AM |
+-------+---------+--------------------+
$ stackato stats json-ip
json-ip
+----------+-------------+----------------+--------------+---------------+
| Instance | CPU (Cores) | Memory (limit) | Disk (limit) | Uptime        |
+----------+-------------+----------------+--------------+---------------+
| 0        | 0.0% (2)    | 110.0M (512M)  | 33.3M (2G)   | 2d:1h:27m:54s |
+----------+-------------+----------------+--------------+---------------+

Comment 3

[Chris Turra [:cturra]]

last week i made a change to the stackato router to support websockets (see bug 907407). i am suspicious about these being related to that work.

Comment 4

[Chris Turra [:cturra]]

i have restarted the `router2g` service, which appears to have resolved these issues. we are also working with activestate to review any of the logs to see if we can track down the root cause. can i have each of you test that your applications are functioning correctly again?

Comment 5

[Andy McKay]

Working for my instances.

Comment 6

[Andy McKay]

This is occurring again today.

Comment 7

[Chris Turra [:cturra]]

(In reply to Andy McKay [:andym] from comment #6)
> This is occurring again today.

i have restarted the router2g process again on the stackato cloud controller and it looks to have sorted itself. i am going to follow up with activestate on this to see if we can track down the root cause.

Comment 8

[Chuck Harmston [:chuck]]

A slightly-different symptom that may warrant a new bug: http://flue.paas.allizom.org/ is now 404ing on all requests.

Comment 9

[Chris Turra [:cturra]]

:chuck - the flue application doesn't appear to be running. this would be result of the work i did last night on the dea nodes. see the following email thread on that:

  https://mail.mozilla.org/pipermail/paas-users/2013-August/000073.html


you can solve this by restarting your application.

Comment 10

[Chris Turra [:cturra]]

after seeing a number of bugs like this one get opened after i made the router2g changes last week, i have reverted back to the default router in stackato 2.6. it appears to have cleared up these issues for others and i suspect addresses this bug also. you can see more details in my comments on bug 910931. 

marking this bug as r/fixed. please do reopen however if these connection issues persist.