633061 - Require Apache2::SizeLimit 0.96 for proper operation on Linux

Status: RESOLVED FIXED

(Bugzilla :: Installation & Upgrading, defect)

Description

[Max Kanat-Alexander]

Currently, Apache::SizeLimit determines the wrong "unshared" size for Apache processes on Linux. I just submitted a patch that fixes this to the Apache::SizeLimit maintainer that he says that he will be integrating as part of 0.96. He says that 0.96 should be out tomorrow.

Once I applied this patch, I also noticed that 70MB was a *much* too large size for the set_max_unshared_size value. Apache quickly takes up all of my memory if I use 70MB for that value and run the webservice test suite. Upon experimentation, I've found that about 45MB is a good size. This lets httpd children live for a long time (about 80 requests, in my tests), but doesn't take up all my RAM when running the webservice QA tests.

Comment 1

[Max Kanat-Alexander]

Attachment: v1

Comment 2

[David Lawrence [:dkl]]

Still waiting on 0.96 to show up in CPAN. Is it there for you?

Comment 3

[Max Kanat-Alexander]

Ah, I don't think it's out, and it may not actually resolve this problem, as it turns out. There's a small debate happening about it.

Comment 4

[Max Kanat-Alexander]

0.96 rc1 is out, and it does have the fix in it so the final 0.96 should be out in a few days.

Comment 5

[Ozan Caglayan]

There's also another problem with all these SizeLimit stuff. mod_perl 2.0.4 ships 0.03 version of SizeLimit. When you install mod_perl and SizeLimit both, the SizeLimit one is not detected and the 0.03 is used. You have to manually remove it from mod_perl and everything works Ok.

But this is unmaintainable for distros. mod_perl just released 2.0.5 which includes SizeLimit 0.95. Using that version fixes all the problems.

Comment 6

[Max Kanat-Alexander]

(In reply to comment #5)
> There's also another problem with all these SizeLimit stuff. mod_perl 2.0.4
> ships 0.03 version of SizeLimit. When you install mod_perl and SizeLimit both,
> the SizeLimit one is not detected and the 0.03 is used. You have to manually
> remove it from mod_perl and everything works Ok.

  Yes, in normal Bugzilla installation situations, this is overcome by using install-module.pl.

> But this is unmaintainable for distros.

  I do understand that, it's a difficult situation we've run into before.

> mod_perl just released 2.0.5 which
> includes SizeLimit 0.95. Using that version fixes all the problems.

  0.95 does not have the fix required to make it actually work properly on Linux. 0.96 will have it.

Comment 7

[Max Kanat-Alexander]

I've added a link to the URL field that explains in high technical detail exactly how memory consumption works on Linux, so that anybody interested in the very technical details of this bug can understand them. (Also, so that we can easily find this link and send people to it in the future.)

Comment 8

[Frédéric Buclin]

I think it should block 4.2rc1 (to have the version bump before the RC stage). If mkanat disagrees, feel free to remove the flag.

Comment 9

[David Lawrence [:dkl]]

Unless I missing something, I still only see 0.95 in CPAN. Any idea when the author is going to publish?

dkl

Comment 10

[Frédéric Buclin]

I just looked on CPAN, and I realized that Apache2::SizeLimit is part of mod_perl. So isn't it enough to require mod_perl to have this module automatically installed?

Comment 11

[Frédéric Buclin]

Also, I don't think this should go into 4.0.x as some distros may not want to upgrade their mod_perl package just for Bugzilla (or for some other reasons), and this would prevent them from updating their Bugzilla package.

Comment 12

[Max Kanat-Alexander]

No, it's "dual-lifed" like core Perl modules are. It has a CPAN module and it's *also* part of mod_perl. The newer versions are CPAN modules, the older versions are part of mod_perl.

Comment 13

[miketosh]

The most recent Apache2::SizeLimit I can find available for download on CPAN is 0.95.  And that is part of the modperl 2.0.5, not a separate package.

http://search.cpan.org/search?query=Apache2%3A%3ASizeLimit&mode=all

Comment 14

[Max Kanat-Alexander]

Right. 0.96 isn't out yet.

Comment 15

[Frédéric Buclin]

Retargetting to 4.2 as we shouldn't bump the min version required on stable branches.

Comment 16

[Frédéric Buclin]

Not a blocker as 0.96 isn't out yet. We will take it when it's available, if this happens before 4.2 final.

Comment 17

[:glob ✱]

0.96 was released on 2011-12-21.

Comment 18

[Frédéric Buclin]

The doc says that reading /proc/self/smaps is expensive and that we should consider using Apache2::SizeLimit->set_check_interval(). I see it's not used by Bugzilla. May we should?

Comment 19

[Frédéric Buclin]

While you are on it, shouldn't you also bump the min version of mod_perl to 2.0.2 (released on October 20, 2005)? Its changelog says:

- Add support for the newer Smaps (/proc/self/statm) on Linux systems that support it to accurately count the amount of shared memory.

Also, there is a comment in mod_perl.pl which says:

    # $0 is broken under mod_perl before 2.0.2, so we have to set it
    # here explicitly or init_page's shutdownhtml code won't work right.

Comment 20

[Max Kanat-Alexander]

(In reply to Frédéric Buclin from comment #18)
> The doc says that reading /proc/self/smaps is expensive and that we should
> consider using Apache2::SizeLimit->set_check_interval(). I see it's not used
> by Bugzilla. May we should?

  It hasn't presented any measurable problem yet, and we need to kill large processes as quickly as possible. Otherwise they could sit around until they get called again.

(In reply to Frédéric Buclin from comment #19)
> While you are on it, shouldn't you also bump the min version of mod_perl to
> 2.0.2 (released on October 20, 2005)? Its changelog says:
> 
> - Add support for the newer Smaps (/proc/self/statm) on Linux systems that
> support it to accurately count the amount of shared memory.

  That's actually a note about Apache::SizeLimit, which is bundled with mod_perl. In other words, it's earlier release notes of this library. The release didn't actually work right with smaps, only the most recent Apache::SizeLimit does.

  So for now, there's not enough reason to change the min requirement.

Comment 21

[David Lawrence [:dkl]]

Comment on attachment 511268 [details] [diff] [review]
v1

Review of attachment 511268 [details] [diff] [review]:
-----------------------------------------------------------------

Works fine for me. install-module.pl pulls down the correct 0.96 version. After proper configuration
I am able to observe proper behavior in the logs when adjusting the max_unshared_size value where
Apache kills off children once the max has been passed. r=dkl

Comment 22

[Max Kanat-Alexander]

Sweet! We should include this in rc2 in case people have trouble installing it or other problems. (Installing mod_perl-related modules requires you to have mod_perl-devel installed on CentOS, which can confuse people.)

If it causes people too much trouble, we might have to back it out and just point major installs at it, for 4.2.

Comment 23

[Frédéric Buclin]

We will have to update the relnotes before releasing rc2 to mention the 0.93 -> 0.96 change. Committing it now to see what really still need patches/reviews.

Committing to: bzr+ssh://[e-mail]/bugzilla/trunk/
modified mod_perl.pl
modified Bugzilla/Install/Requirements.pm
Committed revision 8089.

Committing to: bzr+ssh://[e-mail]/bugzilla/4.2/
modified mod_perl.pl
modified Bugzilla/Install/Requirements.pm
Committed revision 8011.

Comment 24

[Frédéric Buclin]

Added to relnotes for 4.2rc2.