747016 - report.cgi is causing load-death of the webheads

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, defect)

Description

[:glob ✱]

report.cgi is causing load-death of the webheads, requiring several reboots.

httpd access logs are showing:

[warn] Use of uninitialized value in string eq at /data/www/bugzilla.mozilla.org/report.cgi line 157. [#1334830057.11594]

the arecibo report doesn't exist, so the arecibo error reporting is the most likely cause here.

so we need to fix two things here, the bug in the error management, and the uninit warning itself.

Comment 1

[:glob ✱]

it looks like the uninit error is triggered for each row for some queries; this is resulting in a massive volume of arecibo reports which is killing the server.

i'll push a change which does two things; first fix the undef warning, second nice(19) the forked process which sends the report to arecibo.

Comment 2

[:glob ✱]

Committing to: bzr+ssh://[e-mail]/bmo/4.0/
modified report.cgi
modified Bugzilla/Arecibo.pm
Committed revision 8147.

Committing to: bzr+ssh://[e-mail]/bmo/4.0/
modified report.cgi
Committed revision 8148.

Committing to: bzr+ssh://[e-mail]/bmo/4.2/
modified Bugzilla/Arecibo.pm
Committed revision 8131.

Comment 3

[Ashish Vijayaram [:ashish]]

(In reply to Byron Jones ‹:glob› from comment #1)
> it looks like the uninit error is triggered for each row for some queries;
> this is resulting in a massive volume of arecibo reports which is killing
> the server.
> 
Explains why there were load alerts from arecibo too. I didn't realise they're correlated because I didn't find any related reports on arecibo.