1236021 - brasstack is throwing 502

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Ludovic Hirlimann [:Usul]]

load https://brasstacks.mozilla.com/orangefactor/ see that it complains about
 502: Bad Gateway loading https://brasstacks.mozilla.com/api/count?startday=2015-12-24&endday=2015-12-31&tree=trunk 

of course https://mana.mozilla.org/wiki/display/websites/brasstacks.mozilla.com#brasstacks.mozilla.com-AdminContacts is empty :(

Comment 1

[Marlena]

logon to server notice process for app not running on brasstacks1.dmz.sjc1.mozilla.com.  Restarted it 

[[e-mail]3 ~]# ps -ef  | grep -i orangefactor
root     32508 32481  0 13:54 pts/0    00:00:00 grep -i orangefactor
[[e-mail]3 ~]# /etc/init.d/orangefactor stop; /etc/init.d/orangefactor start; /etc/init.d/nginx reload
stopping orangefactor                                      [FAILED]
starting orangefactorspawn-fcgi: child spawned successfully: PID: 32524
                                                           [  OK  ]
Reloading nginx:                                           [  OK  ]
[[e-mail]3 ~]# ps -ef  | grep -i orangefactor
webtools 32524     1  1 13:55 ?        00:00:06 /home/webtools/apps/orangefactor/bin/python /home/webtools/apps/orangefactor/src/orangefactor/server/woo_server.fcgi
root     32568 32481  0 14:01 pts/0    00:00:00 grep -i orangefactor
[[e-mail]3 ~]# 


Site confirmed working now from Moc

Comment 2

[:Atoll]

Site's repaired, woot. Ashish, do we want to close a monitoring gap for comment 1?

Comment 3

[Ashish Vijayaram [:ashish]]

(In reply to Richard Soderberg [:atoll] from comment #2)
> Site's repaired, woot. Ashish, do we want to close a monitoring gap for
> comment 1?

Host monitoring was removed in Bug 1231001.

Comment 4

[:Atoll]

While I'm glad this was an easy fix, I'm troubled in light of 1231001#c2 which basically asks for all monitoring to be removed.

:emorley, should we instruct MOC to refuse all participation on this server, or should we add a basic process check for the obvious fix above, or what's the correct plan forward here? It's an IT-managed server, so we're expected to manage it, and shutting off all monitoring wholesale seems guaranteed to make IT look bad for no good reason :(

Comment 5

[Ed Morley [:emorley]]

I don't believe it should be IT managed; it's effectively an a-team project server running an EOL webapp that we hope to replace with equivalent Treeherder functionality (and decom the box) in the next few months.

Comment 6

[:Atoll]

Ah, interesting. Okay! We will keep that in mind when people report breakage in the future. I assume it's safe to try and perform the basic repair steps from comment 1 as desired, without any particular harm to it?

Comment 7

[Ed Morley [:emorley]]

Yup :-)