767271 - detect and cancel long running mod_perl processes

Status: NEW

(Bugzilla :: Bugzilla-General, defect)

Description

[:glob ✱]

every now and then we see *extremely* long httpd worker processes on bmo (today: 394838 seconds for a post to xmlrpc.cgi).

we should deploy something which kills very long running working processes (> 8 hours ?)

Comment 1

[Shyam Mani [:fox2mike]]

I'm not really sure killing apache workers works that well (i've seen it kill httpd altogether sometimes).

How about looking at something like http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild 

I'm also CC'ing some webops folks for their input.

Comment 2

[:glob ✱]

maxrequestsperchild doesn't appear to limit the duration a worker can spend servicing a single request.

we really need something like http://httpd.apache.org/docs/2.2/mod/core.html#timeout however that doesn't work when using mod_perl.

Comment 3

[Shyam Mani [:fox2mike]]

CC'ing Ashish.

Comment 4

[Philippe M. Chiasson (:gozer)]

Generally, since this is running in mod_perl, nothing prevents bugzilla's main handler from setting a timeout, say 30 seconds, and gracefully self-terminate with logging if it runs out of time.

Better solution than an external kill script, IMO.

Comment 5

[Philippe M. Chiasson (:gozer)]

Can we implement this ?

Comment 6

[:glob ✱]

dkl, this isn't an infrastructure issue - bugzilla's mod_perl handler should detect long running processes as per comment 4.

(In reply to Philippe M. Chiasson (:gozer) from comment #5)
> Can we implement this ?

sure; i'll take a look.