611356 - b.m.o. report.cgi returns HTTP 500 error

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Mark Côté [:mcote]]

Since sometime yesterday, the /count method on bzAPI hasn't been working.  At the moment, it returns an HTTP 500 Internal Server Error.  /bug, with the same parameters, works fine.

Example:

https://api-dev.bugzilla.mozilla.org/latest/count?resolution=FIXED&changed_field=resolution&changed_field_to=FIXED&changed_before=Now&changed_after=2010-10-12T16%3A36%3A12Z&field0-0-0=cf_blocking_20&type0-0-0=notsubstring&value0-0-0=final&field0-1-0=cf_blocking_20&type0-1-0=notsubstring&value0-1-0=beta&field0-2-0=assigned_to&type0-2-0=equals&value0-2-0=mcote%40mozilla.com

--- 
error: 1
http_code: 500
message: "HTTP Error: 500 Internal Server Error"


https://api-dev.bugzilla.mozilla.org/latest/bug?resolution=FIXED&changed_field=resolution&changed_field_to=FIXED&changed_before=Now&changed_after=2010-10-12T16%3A36%3A12Z&field0-0-0=cf_blocking_20&type0-0-0=notsubstring&value0-0-0=final&field0-1-0=cf_blocking_20&type0-1-0=notsubstring&value0-1-0=beta&field0-2-0=assigned_to&type0-2-0=equals&value0-2-0=mcote%40mozilla.com

--- 
bugs: 
  - 
    [...]
    status: RESOLVED
    summary: No default button selection in Profile Manager
    [...]
  - 
    [...]
    status: RESOLVED
    summary: ProfileManager needs to be localizable
    [...]

Comment 1

[Shawn Wilsher :sdwilsh]

The API has been getting hit pretty hard, so it's been disabled intermittently.  It's up again right now AFAIK.

Comment 2

[Mark Côté [:mcote]]

Not for me at the moment.  Super simple query:

https://api-dev.bugzilla.mozilla.org/latest/count?resolution=FIXED

--- 
error: 1
http_code: 500
message: "HTTP Error: 500 Internal Server Error"

Comment 3

[Bob Clary [:bc] (inactive)]

I have two sets of virtual machines making calls to BzApi. One set from the phoenix colo and one set from locally. Can you tell if a particular set of IP addresses is particularly egregious ?

Comment 4

[Bob Clary [:bc] (inactive)]

mcote tells me the two ips that were blocked were in phx and sjc. I assume the sjc one is jmaher's and the phx one is mine.

justdave: how badly did I hit BzApi to get cut off ?

Comment 5

[Gervase Markham [:gerv]]

The BzAPI machine now has Munin running so hopefully sysadmin-type people can tell what it needs. My unskilled analysis suggests it has plenty of CPU and disk and network, but needs more memory.

Those who have access to the machine can do:

ssh api-dev -L 5050:localhost:80

and access http://localhost:5050/munin

to see the graphs.

Gerv

Comment 6

[Gervase Markham [:gerv]]

In regard to this particular bug, it's Bugzilla which is broken, not the BzAPI:

https://bugzilla.mozilla.org/report.cgi?x_axis_field=&y_axis_field=bug_status&z_axis_field=&query_format=report-table&short_desc_type=allwordssubstr&short_desc=&longdesc_type=allwordssubstr&longdesc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&resolution=FIXED&emailassigned_to1=1&emailtype1=exact&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailtype2=exact&email2=&bug_id_type=anyexact&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&format=table&action=wrap&field0-0-0=noop&type0-0-0=noop&value0-0-0=

or, in short form

https://bugzilla.mozilla.org/report.cgi?y_axis_field=bug_status&resolution=FIXED&format=table&action=wrap

=>

500 Server Error

Gerv

Comment 7

[Reed Loden [:reed]]

Moving this over to server-ops@ to get the data from the error log... If it turns out to be an actual Bugzilla issue, can move back here or to Bugzilla product for fixing.

Comment 8

[Dave Miller [:justdave]]

The IPs which got blocked from bzapi got unblocked, and were not at fault.  The problem turned out to be a direct link from the release notes for Firefox 4.0b7 to a query that searched for bugs whose state changed to FIXED during a range of dates (which is a fairly intensive query - and lots of people were actually clicking on it).

As for the error this generates:

[Fri Nov 12 00:22:20 2010] [error] DBD::mysql::db selectall_arrayref failed: Table 'bugs.cf_blocking_thunderbird33' doesn't exist

Seems someone created a new custom field and it didn't work...

Comment 9

[Dave Miller [:justdave]]

This should be resolved now.  We ended up writing a command-line script to finish the job that kept timing out in the GUI for the custom field creation.

Comment 10

[Axel Hecht [:Pike]]

Confirmed that my uses of the bzapi work again now, too.

Comment 11

[Mark Côté [:mcote]]

Yup, works for me now too.