Status

Infrastructure & Operations
WebOps: IT-Managed Tools
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: Andy McKay, Assigned: cturra)

Tracking

Details

(Reporter)

Description

5 years ago
My stackato instance keeps giving back:

https://metaplace.paas.allizom.org/
An error has occurred: {"code":"ECONNREFUSED","errno":"ECONNREFUSED","syscall":"connect"}

Intermittently.

Looking at the stackato console at:

http://api.paas.allizom.org/console/users/amckay%40mozilla.com/apps/metaplace/

It says its started, but also says its down.

This is an instance that I've had running for a while and just ignored.
Assignee: server-ops → server-ops-webops
Component: Server Operations → WebOps: IT-Managed Tools
Product: mozilla.org → Infrastructure & Operations
QA Contact: shyam → nmaul
http://newrelic-irc-notify.paas.allizom.org/ is doing the same thing. 

A side note is that it shouldn't be responding at all as the console reports it as down. I've tried shutting it down again from the CLI to no avail. I can start a new instance, but then 2 are running.
http://json-ip.paas.allizom.org/ is doing the same thing, additionnally, here's the output from command line tools:

$ stackato stats json-ip
json-ip
+----------+-------------+----------------+--------------+---------------+
| Instance | CPU (Cores) | Memory (limit) | Disk (limit) | Uptime        |
+----------+-------------+----------------+--------------+---------------+
| 0        | 0.0% (2)    | 110.0M (512M)  | 33.3M (2G)   | 2d:1h:27m:24s |
+----------+-------------+----------------+--------------+---------------+
$ stackato instances json-ip

+-------+-------+--------------------+
| Index | State | Start Time         |
+-------+-------+--------------------+
| 0     | DOWN  | 08/27/2013 10:58AM |
+-------+-------+--------------------+
$ stackato instances json-ip

+-------+-------+--------------------+
| Index | State | Start Time         |
+-------+-------+--------------------+
| 0     | DOWN  | 08/27/2013 10:58AM |
+-------+-------+--------------------+
$ stackato instances json-ip

+-------+---------+--------------------+
| Index | State   | Start Time         |
+-------+---------+--------------------+
| 0     | RUNNING | 08/25/2013 09:31AM |
+-------+---------+--------------------+
$ stackato stats json-ip
json-ip
+----------+-------------+----------------+--------------+---------------+
| Instance | CPU (Cores) | Memory (limit) | Disk (limit) | Uptime        |
+----------+-------------+----------------+--------------+---------------+
| 0        | 0.0% (2)    | 110.0M (512M)  | 33.3M (2G)   | 2d:1h:27m:54s |
+----------+-------------+----------------+--------------+---------------+
(Assignee)

Comment 3

5 years ago
last week i made a change to the stackato router to support websockets (see bug 907407). i am suspicious about these being related to that work.
(Assignee)

Comment 4

5 years ago
i have restarted the `router2g` service, which appears to have resolved these issues. we are also working with activestate to review any of the logs to see if we can track down the root cause. can i have each of you test that your applications are functioning correctly again?
Flags: needinfo?
(Reporter)

Comment 5

5 years ago
Working for my instances.
(Reporter)

Comment 6

5 years ago
This is occurring again today.
Flags: needinfo?
(Assignee)

Comment 7

5 years ago
(In reply to Andy McKay [:andym] from comment #6)
> This is occurring again today.

i have restarted the router2g process again on the stackato cloud controller and it looks to have sorted itself. i am going to follow up with activestate on this to see if we can track down the root cause.
A slightly-different symptom that may warrant a new bug: http://flue.paas.allizom.org/ is now 404ing on all requests.
(Assignee)

Comment 9

5 years ago
:chuck - the flue application doesn't appear to be running. this would be result of the work i did last night on the dea nodes. see the following email thread on that:

  https://mail.mozilla.org/pipermail/paas-users/2013-August/000073.html


you can solve this by restarting your application.
(Assignee)

Comment 10

5 years ago
after seeing a number of bugs like this one get opened after i made the router2g changes last week, i have reverted back to the default router in stackato 2.6. it appears to have cleared up these issues for others and i suspect addresses this bug also. you can see more details in my comments on bug 910931. 

marking this bug as r/fixed. please do reopen however if these connection issues persist.
Assignee: server-ops-webops → cturra
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.