[stackato] Error 503: Unexpected response from service gateway. Please contact your cloud administrator.

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: IT-Managed Tools
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: giorgos, Assigned: cturra)

Tracking

Details

(Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/159] )

(Reporter)

Description

4 years ago
I get the following error from stackato

Error 503: Unexpected response from service gateway. Please contact your cloud administrator.

fairly often recently. Usually deleting and re-creating the application does the trick. Here is the log from the last time

"""
Deleting application [mozillians] ... OK
Using manifest file "stackato.yml"
Framework:       python
Runtime:         ActivePython 2.7
Application Url: mozillians.paas.allizom.org
Creating Application [mozillians] in [https://api.paas.allizom.org] ... OK
Binding Service mozillians-media to mozillians ... Error 503: Unexpected response from service gateway. Please contact your cloud administrator.
"""
(Assignee)

Comment 1

4 years ago
what service are you seeing this with most commonly? that will help me track this down.
Flags: needinfo?(giorgos)
(Reporter)

Comment 2

4 years ago
In 'mozillians' app I've three services:
 1. mysql
 2. memcache
 3. filesystem

I think I've seen this error when creating or binding filesystem (mostly).
Flags: needinfo?(giorgos)
(Assignee)

Comment 3

4 years ago
thnx :giorgos - i think i may have an idea of what's going on here, but am not entirely sure how to resolve. i am going to grab this bug and spend a little more time on it. if i can't make any more progress on it, i will escalate to activestate support.

the 503 error you're seeing appears to only be with with filesystem service. on the stackato services node, i see some "gateway timeout" errors for *only* this service. what's odd is that the filesystem gateway is the service node itself :O

more details to follow once i have had more time to play with this.
Assignee: server-ops-webops → cturra
(Reporter)

Comment 4

4 years ago
Steps to reproduce this (with another project):

1. Git clone https://github.com/glogiotatidis/
2. Checkout branch stackato
3. stackato push


Every push fails while creating the rabbitmq service

Creating new service [gloremo-memcached] ... OK
Binding Service gloremo-memcached to gloremo ... OK
Creating new service [gloremo-mysql] ... OK
Binding Service gloremo-mysql to gloremo ... OK
Creating new service [gloremo-rabbitmq] ... Error 503: Unexpected response from service gateway. Please contact your cloud administrator. 


Removing rabbitmq fixes the problem.
(Assignee)

Comment 5

4 years ago
looks like there was an issue with the rabbitmq service on the stackato dev cluster. here are what the logs said from the services node:

[2014-03-25 09:26:11.418136] rabbit_gateway - pid=5658 tid=9ea1 fid=76b9  DEBUG -- Provision request for label=rabbitmq-2.4 plan=free
[2014-03-25 09:26:11.418375] rabbit_gateway - pid=5658 tid=9ea1 fid=76b9  DEBUG -- [RMQaaS-Provisioner] Attempting to provision instance (request={:label=>"rabbitmq-2.4", :name=>"rabbitmq-1d9ec", :email=>"cturra@mozilla.com", :plan=>"free"})
[2014-03-25 09:26:11.418540] rabbit_gateway - pid=5658 tid=9ea1 fid=76b9  DEBUG -- Pick best nodes from: [{"available_capacity"=>198, "capacity_unit"=>1, "id"=>"rabbit_node_free_1", "plan"=>"free", "time"=>1395764765}]
[2014-03-25 09:26:11.438240] rabbit_gateway - pid=5658 tid=9ea1 fid=76b9  DEBUG -- [RMQaaS-Provisioner] Provisioning on rabbit_node_free_1



i have resolved by restarting the rabbit service and everything appears to be functioning once again.

$ stackato create-service rabbitmq
Creating new service [rabbitmq-4afaf] ... OK

$ stackato delete-service rabbitmq-4afaf
Deleting service [rabbitmq-4afaf] ... OK
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/159]
(Assignee)

Comment 6

4 years ago
based on my tests in comment 5, i am marking this bug as r/fixed. if this is something you're still continuing to see, please feel free to reopen.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.