the default Apache2::SizeLimit limit on bugzilla is set to 70,000kb. wicked pointed out that this may be too small for bmo, and i think he's right. right now the webheads have about 6g of ram each, however we never use more than about 2.5g according to ganglia. we should see a performance boost if we increase the per-process sizelimit. i suggest increasing it to 150,000kb. justdave/dkl/ashish thoughts? note: the main apache error log will contain messages show when a process is terminated. on bugzilla-stage-tip it looks like: [Fri Dec 23 05:34:58 2011] (10699) Apache2::SizeLimit httpd process too big, exiting at SIZE=114472 KB SHARE=40504 KB UNSHARED=73968 REQUESTS=91 LIFETIME=5157 seconds most of the processes on stage-tip when killed are at about this size (110,000kb).
I agree that increasing the limit would be a good thing to try and see if it gives a boost in performance. We can monitor it closely and make sure nothing bad happens but with 6G of ram 150000k should be safe. dkl
i grabbed a copy of all the sizelimit kills from one prod webhead, and *every* request is failing after serving 2 requests, with an average lifetime of 0.1 seconds: count: 3,674,951 avg requests: 2.0 avg lifetime: 0.1 avg unshared: 382,745 [Sun Jan 1 04:02:04 2012] (4769) Apache2::SizeLimit httpd process too big, exiting at SIZE=450236 KB SHARE=80092 KB UNSHARED=370144 REQUESTS=2 LIFETIME=0 seconds unshared is the value that we need to pay attention to, however bug 633061 states that ASL calculates this incorrectly (it's too high). this isn't completely relevant, as when we upgrade the minimum version to 0.96, the default limit is dropped from 70,000 to 45,000. there's a big gap between the current limit of 70,000 and the average of 382,745; and i'm not sure we have enough ram on the phx webheads to increase the limit to avoid this situation. how about we: - bump up the current limit slowly and measure the impact on memory usage, requests per process, and process lifetime - use these measurements to determine if we need to add ram to the webheads (currently phx webheads have 6gb, and sjc webheads have 12gb)
The incremental approach works for me.
bump to 100,000 Committing to: bzr+ssh://firstname.lastname@example.org/bmo/4.0/ modified mod_perl.pl Committed revision 8002.
i'm going to wait for the ram in the webheads to be increased (bug 715425), and then increase the sizelimit to 400,000 and monitor closely.
Depends on: 715425
bump to 400,000 Committing to: bzr+ssh://email@example.com/bmo/4.0/ modified mod_perl.pl Committed revision 8060.
the memory usage increased significantly less after the bump to 400,000. it's now running with a limit of 1,600,000 and there's a large amount of unused memory still. Committing to: bzr+ssh://firstname.lastname@example.org/bmo/4.0/ modified mod_perl.pl Committed revision 8061. Committing to: bzr+ssh://email@example.com/bmo/4.2/ modified mod_perl.pl Committed revision 8048.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.