909086 - Orangefactor API giving 500 ISEs

Status: RESOLVED WORKSFORME

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Ed Morley [:emorley]]

Visit http://brasstacks.mozilla.com/orangefactor/ -> stuck on "loading"

In webconsole:
http://brasstacks.mozilla.com/orangefactor/api/count?startday=2013-08-18&endday=2013-08-25&tree=trunk
-> HTTP/1.1 500 Internal Server Error

On brasstacks, the following didn't help:
$ /etc/init.d/nginx reload
$ /etc/init.d/orangefactor stop; /etc/init.d/orangefactor start

/var/log/nginx/error.log contains:
{
2013/08/25 02:08:04 [error] 10624#0: *27681442 FastCGI sent in stderr: "Traceback (most recent call last):
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 237, in process
    return self.handle()
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 228, in handle
    return self._delegate(fn, self.fvars, args)
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 409, in _delegate
    return handle_class(cls)
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 385, in handle_class
    return tocall(*args)
  File "/home/webtools/apps/orangefactor/src/orangefactor/server/handlers.py", line 169, in GET
    results = json.dumps(self._GET(params, body))
  File "/home/webtools/apps/orangefactor/src/orangefactor/server/handlers.py", line 443, in _GET
    return BugData._GET(self, params, True)
  File "/home/webtools/apps/orangefactor/src/orangefactor/server/handlers.py", line 351, in _GET
    tbpldata = self.getPushLogData(args)
  File "/home/webtools/apps/orangefactor/src/orangefactor/server/handlers.py", line 233, in getPushLogData
    result = eslib.query(es_args, exclude={'os': exclude_tbpl_os})
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/mozautoeslib-0.1.1-py2.6.egg/mozautoeslib/eslib.py", line 215, in query
    raise Exception("Key ['hits']['hits'] not found in response data")
Exception: Key ['hits']['hits'] not found in response data" while reading upstream, client: 10.22.74.209, server: brasstacks1.dmz.scl3.mozilla.com, request: "GET /orangefactor/api/count?startday=2013-08-18&endday=2013-08-25&tree=trunk HTTP/1.1", upstream: "fastcgi://127.0.0.1:9500", host: "brasstacks.mozilla.com", referrer: "http://brasstacks.mozilla.com/orangefactor/"

2013/08/25 02:08:38 [error] 10624#0: *27681450 FastCGI sent in stderr: "Traceback (most recent call last):
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 237, in process
    return self.handle()
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 228, in handle
    return self._delegate(fn, self.fvars, args)
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 409, in _delegate
    return handle_class(cls)
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/web/application.py", line 385, in handle_class
    return tocall(*args)
  File "/home/webtools/apps/orangefactor/src/orangefactor/server/handlers.py", line 169, in GET
    results = json.dumps(self._GET(params, body))
  File "/home/webtools/apps/orangefactor/src/orangefactor/server/handlers.py", line 443, in _GET
    return BugData._GET(self, params, True)
  File "/home/webtools/apps/orangefactor/src/orangefactor/server/handlers.py", line 382, in _GET
    orange_bugs = bzcache.get_bugs(list(buglist))
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/bzcache-0.1.0-py2.6.egg/bzcache/bzcache.py", line 114, in get_bugs
    False)
  File "/home/webtools/apps/orangefactor/lib/python2.6/site-packages/bzcache-0.1.0-py2.6.egg/bzcache/bzcache.py", line 140, in add_or_update_bug
    self.log('%s - exception while processing bug %s' % (date, id))
UnboundLocalError: local variable 'id' referenced before assignment" while reading upstream, client: 10.22.74.209, server: brasstacks1.dmz.scl3.mozilla.com, request: "GET /orangefactor/api/count?startday=2013-08-18&endday=2013-08-25&tree=trunk HTTP/1.1", upstream: "fastcgi://127.0.0.1:9500", host: "brasstacks.mozilla.com", referrer: "http://brasstacks.mozilla.com/orangefactor/"
}

Comment 1

[Mark Côté [:mcote]]

Wfm. I think you hitting this during the Bugzilla downtime (as discussed in email a couple weeks ago. :)

Comment 2

[Ed Morley [:emorley]]

This wasn't during the downtime - I could file this bug... :-)

(It also was not working still as of late last night/this morning).

Also, we should fail gracefully...

Comment 3

[Ed Morley [:emorley]]

[17:11:39.279] GET http://brasstacks.mozilla.com/orangefactor/api/bybug?startday=2013-08-19&endday=2013-08-26&tree=trunk [HTTP/1.1 500 Internal Server Error 63862ms]

Comment 4

[Ed Morley [:emorley]]

(That's from just now)

Comment 5

[Ed Morley [:emorley]]

Hmm seems to be intermittent (worked momentarily the next try).

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Various Mozilla sites have been slow/wonky all morning.

Comment 7

[Mark Côté [:mcote]]

Filed bug 909368 for adding error messages to the UI when AJAX calls fail.

I just tried that request, and it worked for me.  It did, however, take longer than expected, so I think something is indeed not working correctly.

Comment 8

[Ed Morley [:emorley]]

Thank you :-)

Comment 9

[Ed Morley [:emorley]]

Fine now