Last Comment Bug 370370 - On Windows, Bugzilla crashes when accessing editparams.cgi with mod_perl enabled
: On Windows, Bugzilla crashes when accessing editparams.cgi with mod_perl enabled
Status: RESOLVED WORKSFORME
:
Product: Bugzilla
Classification: Server Software
Component: Installation & Upgrading (show other bugs)
: 2.23.4
: x86 Windows 2000
: -- normal (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
: default-qa
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2007-02-14 02:45 PST by Yann
Modified: 2010-02-28 12:46 PST (History)
8 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
v1 (1.57 KB, patch)
2009-12-01 08:00 PST, Michael Thomas (Mockodin)
no flags Details | Diff | Splinter Review

Description Yann 2007-02-14 02:45:21 PST
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Build Identifier: 

I installed mod_perl on windows.

Windows 2000
Apache 2.2.4 from apachelougne
Activestate 5.8.8.820
Mod_perl 2.0.3 from apachelounge
Bugzilla 2.23.4

It seems to work fine under common situations, but every time I click on the "Parameters" link, it crashes the apache child module.

It works fine under CGI.

Reproducible: Always

Steps to Reproduce:
1.use mod_perl
2.click on "Parameters" link
3.the child apache process restarts
Actual Results:  
Apache error:
[notice] Parent: child process exited with status 3221225725 -- Restarting.
Comment 1 Yann 2007-02-14 09:14:45 PST
I tested all Bugzilla features I could think of in 2.23.4 (except for emails), and this is the only place where this issue occurs.
Comment 2 Max Kanat-Alexander 2007-02-14 14:04:01 PST
I can't reproduce this on Linux. Does it happen if you use the standard apache.org version of everything?
Comment 3 Yann 2007-02-15 01:54:05 PST
I will try this tomorrow and come back to you.
Comment 4 Yann 2007-02-16 11:51:31 PST
Damn, couldn't get that tested today, hope by first half of next week. 
Comment 5 Yann 2007-02-20 05:59:56 PST
(In reply to comment #2)
> I can't reproduce this on Linux. Does it happen if you use the standard
> apache.org version of everything?

OK, tested and same issue with standard Apache version of Apache and mod_perl.
Really strange that it crashes only on editparams.cgi ...

Comment 6 Yann 2007-02-25 09:24:18 PST
I checked bug 370010 (size limit for mod_perl), and went on the documentation page (http://search.cpan.org/~pgollucci/mod_perl-2.0.3/docs/api/Apache2/SizeLimit.pod#Synopsis).

According to this page: 

"Win32
    Under mod_perl 1, SizeLimit provided basic functionality by using Win32::API to access process memory information. This worked because there was only one mod_perl thread. With mod_perl 2, Win32 runs a true threaded MPM, which unfortunately means that we can't tell the size of each interpreter. Win32 support is disabled until a solution for this can be found."

So shouldn't mod_perl be not recommended for Win32 at all in the release notes for 3.0?
Comment 7 Jarmo Torvinen 2007-03-21 09:31:05 PDT
I can confirm this problem. The mod_perl is currently unusable for Windows Apache 2.2 installation using Apache MPM winnt (http://httpd.apache.org/docs/2.2/mod/mpm_winnt.html).

At least it does not crash for the moment.


Error message on apache startup:
--------------------------

C:\Program Files\Apache Software Foundation\Apache2.2\bin>httpd.exe
Syntax error on line 154 of C:/Program Files/Apache Software Foundation/Apache2.
2/conf/httpd.conf:
Apache2::SizeLimit at the moment works only with non-threaded MPMs at C:/Perl/si
te/lib/Apache2/SizeLimit.pm line 51.
BEGIN failed--compilation aborted at C:/Perl/site/lib/Apache2/SizeLimit.pm line 98.
Compilation failed in require at C:/Bugzilla/mod_perl.pl line 33.
BEGIN failed--compilation aborted at C:/Bugzilla/mod_perl.pl line 33.
Compilation failed in require at (eval 3) line 1.
Comment 8 Frédéric Buclin 2007-04-03 13:03:06 PDT
Max, looks like something which should be mentioned in the release notes if confirmed.
Comment 9 Max Kanat-Alexander 2007-04-03 14:55:09 PDT
Can you try it with Apache 2.0 and see if it works there? I've heard of various problems with Apache 2.2 and mod_perl 2.
Comment 10 Yann 2007-04-04 11:59:59 PDT
(In reply to comment #9)
> Can you try it with Apache 2.0 and see if it works there? I've heard of various
> problems with Apache 2.2 and mod_perl 2.

I wish I could, but Apache::SizeLimit got activated since I reported this bug, which prevents from running on Windows (it does not support Worker MPM).

I will try and test it by de-activating that line in mod_perl.cgi. But even if that works out, having no protection against RAM overflow sounds scary... any idea to work around that ?

Comment 11 Yann 2007-04-04 12:01:14 PDT
(In reply to comment #10)
> I wish I could, but Apache::SizeLimit got activated since I reported this bug,
> which prevents from running on Windows (it does not support Worker MPM).

I meant that basically Apache won't start because Apache::SizeLimit throws an error message that it cannot work on Worker MPM.
Comment 12 Yann 2007-04-17 03:09:18 PDT
(In reply to comment #9)
> Can you try it with Apache 2.0 and see if it works there? I've heard of various
> problems with Apache 2.2 and mod_perl 2.

Confirmed the issue is the same with Apache 2.0.
I isolated the issue in editparams.cgi to occur in the following code (does not know if it helps):


foreach my $panel (Bugzilla::Config::param_panels()) {
    eval("require Bugzilla::Config::$panel") || die $@;
    my @module_param_list = "Bugzilla::Config::${panel}"->get_param_list(1);
    my $item = { name => lc($panel),
                 current => ($current_panel eq lc($panel)) ? 1 : 0,
                 param_list => \@module_param_list,
                 sortkey => eval "\$Bugzilla::Config::${panel}::sortkey;"
               };
    push(@panels, $item);
    $current_module = $panel if ($current_panel eq lc($panel));
}
Comment 13 Max Kanat-Alexander 2007-05-08 13:36:59 PDT
Added to relnotes in bug 379777.
Comment 14 hardon 2009-04-14 15:02:10 PDT
Commenting Apache2::SizeLimit from mod_perl.pl makes it work somehow (its impossible(?) to track a threads memory usage so IMO its not fair to bail on this). Then I also ran into the editparams.cgi problem, made me extremely confused. By trial and error I discovered: There is a a weird limit on 859 chars per string during template generation, and multiline strings (typically html) exceed 859 chars. I have tried to find where this limit comes from, but no luck. The workaround is simple thou, don't use multiline html in templates (dependencygraph.html.tmpl, mta.html.tmpl etc.) but concat lines:
  mail_delivery_method => "Defines how email is sent, or if it is sent at all.<br> " _
"                           <ul> "_
"                             <li> "_

API monitoring gave me some sort of stack trace (last called is at bottom):
mod_perl.dll: modperl_callback
mod_perl.dll: modperl_handler_anon_get
perlXXX.dll: perl_hv_fetch
...
lots of calls, all internally in perlXXX.dll. Consumes lots of cpu and loads of I/O. Possibly infinite recursion and stack exhaustion. 
...
perlXXX.dll: perl_die

If I must guess I would say its perl's fault and not mod_perl, but who knows. Someone should probably report this bug to them:-)

Win XP
Apache 2.2.11 threaded MPM
Active Perl 5.8.9.825
mod_perl 2.0.4
Bugzilla 3.2.3
Comment 15 Max Kanat-Alexander 2009-04-14 15:35:39 PDT
hardon: Thank you for tracking that down!! Since you have the data, could you report it to ActiveState at: http://bugs.activestate.com/ ?
Comment 16 hardon 2009-04-15 10:21:32 PDT
It seems it has already have been fixed:-) I upgraded to ActivePerl 5.10.0.1004, and it seems to work fine there! Of maybe I was just lucky? Someone else should probably confirm before closing this bug.
Comment 17 Frédéric Buclin 2009-06-17 02:49:12 PDT
(In reply to comment #16)
> It seems it has already have been fixed:-) I upgraded to ActivePerl
> 5.10.0.1004, and it seems to work fine there! Of maybe I was just lucky?

Yann, are things better on mod_perl with Perl 5.10?
Comment 18 Michael Thomas (Mockodin) 2009-10-14 17:15:32 PDT
Still does not work with Apache::SizeLimit as it currently does not support a multi-threaded environment.

PerlDoc of Apache2::SizeLimit
At this time, Apache2::SizeLimit does not support use under threaded
MPMs.  This is because there is no efficient way to get the memory
usage of a thread, or make a thread exit cleanly.

At present the module itself is doing the following:

die "Apache2::SizeLimit at the moment works only with non-threaded MPMs"
        if Apache2::MPM->is_threaded();

I suggest updating mod_perl.pl with the following line:

use if $^O ne 'MSWin32', 'Apache2::SizeLimit';

If memory usage is a problem use of "MaxRequestsPerChild" in apache config can be used in its place.
Comment 19 Michael Thomas (Mockodin) 2009-10-14 17:16:30 PDT
re comment 17

with Apache::SizeLimit disabled I have encountered no issues under 5.10
Comment 20 Yann 2009-10-23 02:06:02 PDT
(In reply to comment #19)
> with Apache::SizeLimit disabled I have encountered no issues under 5.10

Nice for you. The problem if I remember properly is that b.m.o when started using mod_perl had memory usage issue (8GB+). Hence the introduction of this Apache::SizeLimit parameter.

I would like to use mod_perl on Windows too, but "MaxRequestsPerChild" cannot guarantee memory hog won't happen afaik. This is an issue for production environment unfortunately.

Still let us know how it works over time (number of hits/users before memory jumps to high levels).
Comment 21 Michael Thomas (Mockodin) 2009-12-01 08:00:03 PST
Created attachment 415394 [details] [diff] [review]
v1

(In reply to comment #20)
Will do Yann. I totally agree on the "cannot guarantee memory hog" comment. It is not a guarantee but with a period of proper monitoring or some good upfront load testing it should be possible to tune things to a reasonable level.

Posting this patch simply because this does allow mod_perl to work. Granted a it does require extra tuning and monitoring. But I think its a more reasonable statement in the notes than saying mod_perl doesn't work under win32.

Out of curiosity what would your concurrent request rate be by chance? A dedicated server or server cluster with 16-32GB of memory (fairly cheap nowadays) should be able to handle quite a lot. Granted cheap is relative..
Comment 22 Max Kanat-Alexander 2009-12-01 16:19:13 PST
Comment on attachment 415394 [details] [diff] [review]
v1

This patch belongs on another bug--it doesn't address the issue described in the Summary of this bug.
Comment 23 Michael Thomas (Mockodin) 2009-12-01 16:54:49 PST
(In reply to comment #22)
> (From update of attachment 415394 [details] [diff] [review])
> This patch belongs on another bug--it doesn't address the issue described in
> the Summary of this bug.

True

Yann is comment 0 still occurring? Under windows + mod_perl + bugzilla 3.5.x I'm not seeing the issue.
Comment 24 Michael Thomas (Mockodin) 2009-12-01 20:00:41 PST
Comment on attachment 415394 [details] [diff] [review]
v1

Attachment 415394 [details] [diff] moved to bug 532318
Comment 25 Max Kanat-Alexander 2010-02-03 17:01:44 PST
Marking WORKSFORME based on mockodin's comment.

Note You need to log in before you can comment on or make changes to this bug.