Orangefactor down - reports 502

RESOLVED FIXED

Status

Tree Management
OrangeFactor
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: Tomcat, Assigned: emorley)

Tracking

Details

(Reporter)

Description

2 years ago
when loading orangefactor i get:

 502: Bad Gateway loading https://brasstacks.mozilla.com/api/count?startday=2016-01-12&endday=2016-01-19&tree=trunk
(Reporter)

Comment 1

2 years ago
ed, nigel mentioned that this might be something for you
Flags: needinfo?(emorley)
Summary: Orangefactor reports 502 → Orangefactor down - reports 502
(Assignee)

Updated

2 years ago
Duplicate of this bug: 1240674
(Assignee)

Comment 3

2 years ago
Yeah it does this every now and again, I'll give it a kick.
The whole stack is pretty ancient, it's still running python 2.6 etc, and a lot of the configs need some TLC.
Assignee: nobody → emorley
Flags: needinfo?(emorley)
(Assignee)

Comment 4

2 years ago
[root@brasstacks1.dmz.scl3 ~]# /etc/init.d/orangefactor stop; /etc/init.d/nginx reload; /etc/init.d/orangefactor start
stopping orangefactor                                      [FAILED]
Reloading nginx:                                           [  OK  ]
starting orangefactorspawn-fcgi: child spawned successfully: PID: 2925
                                                           [  OK  ]
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Comment 5

2 years ago
The old process did not exist, so I was not able to see what caused the excessive memory usage.
Probably using supervisor could help improving its availability?
(Assignee)

Comment 7

2 years ago
I've also pip installed the various requests[security] extras to stop the SSL warnings (since it's running Python 2.6) a la:
https://urllib3.readthedocs.org/en/latest/security.html#pyopenssl

(In reply to Xidorn Quan [:xidorn] (UTC+10) from comment #6)
> Probably using supervisor could help improving its availability?

Yeah likely, though in previous instances the process had hung rather than dieing entirely iirc. I also didn't set the box up, and most of the existing config is undocumented, so I'm a little wary about changing too much without documenting first, which would take longer than it's probably worth.

I'd much prefer to just decommission the whole box, which we'll do once we have OrangeFactor v2 using Treeherder's API directly.
(Assignee)

Comment 8

2 years ago
https://hg.mozilla.org/automation/orangefactor/rev/b16494043881
https://hg.mozilla.org/automation/orangefactor/rev/effe61195b1d
You need to log in before you can comment on or make changes to this bug.