922092 - Orangefactor API returning 500 ISE errors again

Status: RESOLVED WORKSFORME

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Ed Morley [:emorley]]

http://brasstacks.mozilla.com/orangefactor/api/count?startday=2013-09-23&endday=2013-09-30&tree=trunk

-> internal server error

Comment 1

[Ed Morley [:emorley]]

[[e-mail]3 ~]# /etc/init.d/orangefactor stop; /etc/init.d/orangefactor st art
stopping orangefactor                                      [  OK  ]
starting orangefactorspawn-fcgi: child spawned successfully: PID: 11484
                                                           [  OK  ]

http://brasstacks.mozilla.com/orangefactor/api/count?startday=2013-09-23&endday=2013-09-30&tree=trunk 
-> works

Comment 2

[Ed Morley [:emorley]]

Broken again:
18:02:33.756 GET http://brasstacks.mozilla.com/orangefactor/api/count [HTTP/1.1 500 Internal Server Error 1111ms]

Comment 3

[Mark Côté [:mcote]]

Restarted again but we should leave this open to investigate.

Here's an error I've seen a few times in the nginx error logs. Not sure what it means exactly, and it looks cut off.

2013/09/30 10:12:18 [error] 21909#0: *29507488 FastCGI sent in stderr: "Traceback (most recent call last):
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 237, in process
    return self.handle()
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 228, in handle
    return self._delegate(fn, self.fvars, args)
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 409, in _delegate
    return handle_class(cls)
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 385, in handle_class
    return tocall(*args)
  File "/home/webtools/apps/orangefactor/src/orangefactor/server/handlers.py", line 169, in GET
    results = json.dumps(self._GET(params, body))
  File "/home/webtools/apps/orangefactor/src/orangefactor/server/handlers.py", line 443, in _GET
    return BugData._GET(self, params, True)
  File "/home/webtools/apps/orangefactor/src/orangefactor/server/handlers.py", line 382, in _GET
    orange_bugs = bzcache.get_bugs(list(buglist))
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/bzcache-0.1.0-py2.6.egg/bzcache/bzcache.py", line 86, in get_bugs
    doc_type=[self.doc_type])
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/mozautoeslib-0.1.1-py2.6.egg/mozautoeslib/eslib.py", line 186, in query
    doc_types=self.doc_type)
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/pyes-0.15.0-py2.6.egg/pyes/es.py", line 793, in count
    return self._query_call("_count", query, indexes, doc_types, **query_params)
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/pyes-0.15.0-py2.6.egg/pyes/es.py", line 246, in _query_call
    response = self._send_request('GET', path, body, querystring_args)
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/pyes-0.15.0-py2.6.egg/pyes/es.py", line 218, in _send_request
    raise_if_error(response.status, decoded)
  File "/home/

Comment 4

[Ed Morley [:emorley]]

Just had to restart again.

Comment 5

[Mark Côté [:mcote]]

I don't think we've had an outage since comment #4, so closing this WORKSFORME since we didn't ever actually figure it out afaik.