791132 - Staging has quite a few 500 - Internal Server Error

Status: VERIFIED DUPLICATE of bug 791889

(Socorro :: Webapp, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

There are, currently, 11 failures -- most 500 -- internal server error; the full report is here (must be either in-office or on the Mozilla-MV VPN):

http://qa-selenium.mv.mozilla.com:8080/view/Socorro/job/socorro.stage/1040/HTML_Report/?

Some example URLs:

https://crash-stats.allizom.org/query/query?product=Firefox&version=Firefox%3A17.0a2&platform=windows&range_value=1&range_unit=weeks&date=09%2F13%2F2012+22%3A57%3A55&query_search=signature&query_type=contains&query=&reason=&build_id=&process_type=plugin&hang_type=any&plugin_field=filename&plugin_query_type=exact&plugin_query=&do_query=1

https://crash-stats.allizom.org/query/query?product=Firefox&version=ALL%3AALL&range_value=1&range_unit=weeks&date=09%2F13%2F2012+22%3A58%3A23&query_search=signature&query_type=contains&query=&reason=&build_id=None&process_type=any&hang_type=any&do_query=1

https://crash-stats.allizom.org/query/query?product=Camino&version=ALL%3AALL&range_value=3&range_unit=days&date=09%2F13%2F2012+22%3A57%3A28&query_search=signature&query_type=contains&query=&reason=&build_id=&process_type=any&hang_type=any&do_query=1

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Large (sorry!) window:

Failed > Console Output  #600 	Sep 13, 2012 5:12:28 AM	 
Success > Console Output  #599 	Sep 12, 2012 8:46:17 PM

Comment 2

[Schalk Neethling [:espressive]]

In the Kohana logs on stage I see a lot of the following errors related to Bugzilla. From the above URLS, specifically the following one exhibits this:

https://crash-stats.allizom.org/query/query?product=Firefox&version=ALL%3AALL&range_value=1&range_unit=weeks&date=09%2F13%2F2012+22%3A58%3A23&query_search=signature&query_type=contains&query=&reason=&build_id=None&process_type=any&hang_type=any&do_query=1

error: Problem connecting to Bugzilla: The requested URL returned error: 500

For the other two sample URLs the following is the error in the Kohana logs:

http://socorro-mware-stage-zlb.webapp.phx1.mozilla.com/bpapi/search/signatures/products/Firefox/build_ids/None/in/signature/search_mode/contains/to/2012-09-13%2022%3A58%3A23/from/2012-09-06%2022%3A58%3A23/report_type/any/report_process/any/result_number/100/ which was HTTP status 500

And the following then is the actual error thrown by the middleware:

Sep 14 15:46:21 Socorro Web Services (pid 926): 2012-09-14 15:46:21,137 DEBUG - Dummy-2 - DataAPIService __init__
Sep 14 15:46:21 Socorro Web Services (pid 926): 2012-09-14 15:46:21,138 DEBUG - Dummy-2 - Search service __init__
Sep 14 15:46:21 Socorro Web Services (pid 926): 2012-09-14 15:46:21,138 ERROR - Dummy-2 - Caught Error: <type 'exceptions.TypeError'>
Sep 14 15:46:21 Socorro Web Services (pid 926): 2012-09-14 15:46:21,138 ERROR - Dummy-2 - 'NoneType' object is not iterable
Sep 14 15:46:21 Socorro Web Services (pid 926): 2012-09-14 15:46:21,139 ERROR - Dummy-2 - trace back follows:
Sep 14 15:46:21 Socorro Web Services (pid 926): 2012-09-14 15:46:21,139 ERROR - Dummy-2 - Traceback (most recent call last):
Sep 14 15:46:21 Socorro Web Services (pid 926): 2012-09-14 15:46:21,139 ERROR - Dummy-2 - File "/data/socorro/application/socorro/webapi/webapiService.py", line 64, in GET#012    result = self.get(*args)
Sep 14 15:46:21 Socorro Web Services (pid 926): 2012-09-14 15:46:21,139 ERROR - Dummy-2 - File "/data/socorro/application/socorro/middleware/search_service.py", line 40, in get#012    params["terms"] = self.decode_special_chars(params.get("terms"))
Sep 14 15:46:21 Socorro Web Services (pid 926): 2012-09-14 15:46:21,140 ERROR - Dummy-2 - File "/data/socorro/application/socorro/middleware/service.py", line 146, in decode_special_chars#012    value = [x.replace("%2F", "/") for x in value]
Sep 14 15:46:21 Socorro Web Services (pid 926): 2012-09-14 15:46:21,140 ERROR - Dummy-2 - TypeError: 'NoneType' object is not iterable

Comment 3

[Schalk Neethling [:espressive]]

Actually, I take it back, the Bugzilla errors in the logs is not related to any of the above sample URLs, they are all caused by the middleware 500 error and the resulting cause as shown above.

I will have to hunt down where the bugzilla one's came from but, I suspect it is on TCBS.

Comment 4

[Schalk Neethling [:espressive]]

I believe we can safely ignore the Bugzilla connection errors I mentioned earlier. I navigated around the entire site and specifically everywhere we hit Bugzilla and I could not trigger the error once and all data and status info regarding bugs, load just fine.

Comment 6

[Matt Brandt [:mbrandt]]

Verified duplicate -- good eye stephend. Thanks for catching/filing (and even duping) the bug.