949425 - [stackato] Error 503: Unexpected response from service gateway. Please contact your cloud administrator.

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[Giorgos Logiotatidis [:giorgos]]

I get the following error from stackato

Error 503: Unexpected response from service gateway. Please contact your cloud administrator.

fairly often recently. Usually deleting and re-creating the application does the trick. Here is the log from the last time

"""
Deleting application [mozillians] ... OK
Using manifest file "stackato.yml"
Framework:       python
Runtime:         ActivePython 2.7
Application Url: mozillians.paas.allizom.org
Creating Application [mozillians] in [https://api.paas.allizom.org] ... OK
Binding Service mozillians-media to mozillians ... Error 503: Unexpected response from service gateway. Please contact your cloud administrator.
"""

Comment 1

[Chris Turra [:cturra]]

what service are you seeing this with most commonly? that will help me track this down.

Comment 2

[Giorgos Logiotatidis [:giorgos]]

In 'mozillians' app I've three services:
 1. mysql
 2. memcache
 3. filesystem

I think I've seen this error when creating or binding filesystem (mostly).

Comment 3

[Chris Turra [:cturra]]

thnx :giorgos - i think i may have an idea of what's going on here, but am not entirely sure how to resolve. i am going to grab this bug and spend a little more time on it. if i can't make any more progress on it, i will escalate to activestate support.

the 503 error you're seeing appears to only be with with filesystem service. on the stackato services node, i see some "gateway timeout" errors for *only* this service. what's odd is that the filesystem gateway is the service node itself :O

more details to follow once i have had more time to play with this.

Comment 4

[Giorgos Logiotatidis [:giorgos]]

Steps to reproduce this (with another project):

1. Git clone https://github.com/glogiotatidis/
2. Checkout branch stackato
3. stackato push


Every push fails while creating the rabbitmq service

Creating new service [gloremo-memcached] ... OK
Binding Service gloremo-memcached to gloremo ... OK
Creating new service [gloremo-mysql] ... OK
Binding Service gloremo-mysql to gloremo ... OK
Creating new service [gloremo-rabbitmq] ... Error 503: Unexpected response from service gateway. Please contact your cloud administrator. 


Removing rabbitmq fixes the problem.

Comment 5

[Chris Turra [:cturra]]

looks like there was an issue with the rabbitmq service on the stackato dev cluster. here are what the logs said from the services node:

[2014-03-25 09:26:11.418136] rabbit_gateway - pid=5658 tid=9ea1 fid=76b9  DEBUG -- Provision request for label=rabbitmq-2.4 plan=free
[2014-03-25 09:26:11.418375] rabbit_gateway - pid=5658 tid=9ea1 fid=76b9  DEBUG -- [RMQaaS-Provisioner] Attempting to provision instance (request={:label=>"rabbitmq-2.4", :name=>"rabbitmq-1d9ec", :email=>"[e-mail]", :plan=>"free"})
[2014-03-25 09:26:11.418540] rabbit_gateway - pid=5658 tid=9ea1 fid=76b9  DEBUG -- Pick best nodes from: [{"available_capacity"=>198, "capacity_unit"=>1, "id"=>"rabbit_node_free_1", "plan"=>"free", "time"=>1395764765}]
[2014-03-25 09:26:11.438240] rabbit_gateway - pid=5658 tid=9ea1 fid=76b9  DEBUG -- [RMQaaS-Provisioner] Provisioning on rabbit_node_free_1



i have resolved by restarting the rabbit service and everything appears to be functioning once again.

$ stackato create-service rabbitmq
Creating new service [rabbitmq-4afaf] ... OK

$ stackato delete-service rabbitmq-4afaf
Deleting service [rabbitmq-4afaf] ... OK

Comment 6

[Chris Turra [:cturra]]

based on my tests in comment 5, i am marking this bug as r/fixed. if this is something you're still continuing to see, please feel free to reopen.