Closed
Bug 685552
Opened 13 years ago
Closed 13 years ago
Email auto-completion causes server to thrash
Categories
(Bugzilla :: Bugzilla-General, defect)
Tracking
()
RESOLVED
FIXED
Bugzilla 4.0
People
(Reporter: nathan.boy, Assigned: dkl)
References
Details
Attachments
(1 file, 1 obsolete file)
2.88 KB,
patch
|
mkanat
:
review+
|
Details | Diff | Splinter Review |
User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.157 Safari/535.1 Steps to reproduce: I run Bugzilla 4.0.2 using Apache Jetty's CGI Servlet. The server is a slice with 256 MB of RAM and an additional 500 MB swap. I produced this problem by viewing a reported bug in my web browser and clicking the "Edit" link next to the "Assigned To" field of the bug. I then typed in the email address of a user. Actual results: The server I use to host our Bugzilla has become unresponsive periodically ever since we upgraded to Bugzilla 4.0.2. Today I noticed that just before the server becomes unresponsive, a large number of jsonrpc.cgi calls are made. Running top, the most active server processes look like this: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2804 root 20 0 84964 16m 1704 D 24 6.6 0:07.11 jsonrpc.cgi 2813 root 20 0 84308 24m 1768 D 22 10.1 0:07.05 jsonrpc.cgi 2825 root 20 0 66916 17m 1692 D 19 7.3 0:04.09 jsonrpc.cgi 2801 root 20 0 81404 15m 1732 D 18 6.3 0:05.70 jsonrpc.cgi 2823 root 20 0 66916 17m 1692 D 16 7.3 0:03.89 jsonrpc.cgi 2807 root 20 0 84572 23m 1768 D 12 9.4 0:04.00 jsonrpc.cgi 2385 root 20 0 612m 27m 716 S 11 11.2 0:33.39 java 2816 root 20 0 84964 25m 1768 D 11 10.4 0:06.27 jsonrpc.cgi 2810 root 20 0 85888 21m 1724 D 10 8.9 0:04.77 jsonrpc.cgi 2798 root 20 0 92476 14m 1676 D 7 5.8 0:06.14 jsonrpc.cgi Our server requires a reboot when enough of these calls are made to cause the server to exhaust its swap and become unresponsive. It appears that a call to jsonrpc.cgi is made each time a key is pressed in the "Assigned To" field, presumably to aid in auto-completion. Expected results: Right now I am working around this by turning on the 'usemenuforusers' parameter so that emails can be chosen from a list rather than through email auto-completion. Ideally auto-complete would not make this many resource-intensive calls to the server, though.
Assignee | ||
Comment 1•13 years ago
|
||
I can confirm that this happens on my local test instance used for testing BMO changes. When I type out an email address almost entirely, it will fire off a call to jsonrpc.cgi for each letter almost and I see all of them running in my process list. If enough are fired, the site becomes unresponsive and I have to wait til the start to finish up on their own or manually kill them. The workaround so far is to just type a small amount of the email address and let the autocomplete drop down appear. Even though I can normally type the whole address before autocomplete returns even for a small result list. Is there a way to tweak the autocomplete settings to wait a larger amount of time before calling jsonrpc.cgi or even keep a count of how many it is fired so far and wait til the number gets below a certain threshold? I will try to experiment if I find some time. dkl
Comment 2•13 years ago
|
||
I don't know how many matches are returned, but couldn't the code call jsonrpc.cgi only once to get an initial list, and then filter the list based on new characters entered by the user? It would then only re-request a new list if a character is removed.
Assignee | ||
Comment 4•13 years ago
|
||
According to the javascript console, the autocomplete widget is cancelling any prior requests it has made each time a new letter is entered in the text field. For for example the string "dkl@moz" will fire off a jsonrpc.cgi request 4 times, but each subsequent request cancels the ones before it. So according to autocomplete it only has 1 request running at any given time. But on the apache side on my instance, I see each of the jsonrpc.cgi processes still running in the process list and apache defaults to killing them off after a specified amount of time. Apache is not recognizing that the request was cancelled by the client (similar to hitting the stop button in Firefox) and just lets them run til timeout. So if I type "dkl@mozilla.co", I will have 11 defunct jsonrpc.cgi requests running on my server. Setting a num of requests threshold in the autocomplete code in js/field.js won't do anything since autocomplete is already cancelling any outstanding requests anyway. LpSolits idea of sending only one request and then have the autocomplete code filter the full results based on additional keys type is a compelling solution but depending on what the user types initially, it could be a very large result list depending on the Bugzilla installation. Is there an Apache directive/setting that will make Apache properly kill off a process once it receives the cancel signal from the client instead of letting the processes run to completion or get killed off from some timeout setting? dkl
Comment 5•13 years ago
|
||
(In reply to David Lawrence [:dkl] from comment #1) > Is there a way to tweak the autocomplete settings to wait a larger amount of > time before calling jsonrpc.cgi Yes, however, this would introduce a perceptible delay in receiving results for every single autocomplete query. Also, you can't guess how quickly people are going to type vs. how fast somebody's Bugzilla server is going to be. In other words, not only would this slow things down from the user's perspective, it probably wouldn't even help. Nathan's problem is that his hardware will never be able to support autocomplete or any of Bugzilla's future AJAX functionality, under basically any realistic circumstances. If you are running Bugzilla under mod_cgi, every single request takes a very long time, even if it's going to return a tiny amount of data or do a tiny amount of work. That means that if every user produces several .cgi requests on every page or on every interaction, they will all live for a long time and consume quite a bit of RAM because they are all alive at once. Nathan is the only user who's ever reported to me that they're running Bugzilla with so little RAM. I'm impressed that previous versions of Bugzilla worked so well in that situation! :-) Probably what Nathan wants to do is disable every AJAX feature of Bugzilla (presently the automatic duplicate detection and the AJAX autocomplete). This would currently require some small amount of code hackery. (I'm also going to respond to dkl's other comment, here, which is also relevant.)
Comment 6•13 years ago
|
||
(In reply to David Lawrence [:dkl] from comment #4) > So if I type "dkl@mozilla.co", I will have 11 defunct jsonrpc.cgi requests > running on my server. How long are you waiting between typing each letter? Are you typing at full speed? The present delay is 50ms. It's "userAutoComp.queryDelay = 0.05;" in js/field.js. 50ms is barely perceptible, but 100ms would be perceptible, particularly on mod_perl installations. (Also, slowing down mod_cgi installs even more seems bad.) > LpSolits idea of sending only one request and then have the autocomplete > code filter the full results based on additional keys type is a compelling > solution Except that it wouldn't work, because of maxusermatches. The longer list is not actually a complete list, always. Also, what happens when the user hits backspace? What happens when they cut and paste? The complexity there becomes high very quickly. > Is there an Apache directive/setting that will make Apache properly kill off > a process once it receives the cancel signal from the client Your question assumes that there is such a thing as a "cancel signal". I suppose if the browser was nice, Apache would get a FIN from the browser, at the TCP level. And yeah, if we could detect that, we could...ah, no, actually not, come to think of it. This goes back to the historical problem of "what happens when the user closes the browser when process_bug.cgi is still running on a very long transaction, say for changing 10,000 bugs?" The fact that Apache forcibly times us out in that situation after several seconds of a closed connection is already bad; making it happen immediately would be even worse. Perhaps a better solution would be to set the queryDelay higher on mod_cgi installs, but I'm concerned that it would have to be abnormally high to not cause the problems that you're experiencing. (And there's basically no setting high enough that it would work on Nathan's system while still being actually useful to users--Nathan's users are better off with the old post-submit user-match suggestions.)
Can I disable this for now as it's practically impossible to add components now.
Assignee | ||
Comment 9•13 years ago
|
||
(In reply to Carl from comment #8) > Can I disable this for now as it's practically impossible to add components > now. Currently you will have to edit the template code in global/userselect.html.tmpl to turn it off as it is automatically enabled if you have jsonrpc support enabled. Perhaps we need a param for this to turn the feature on/off? dkl
Comment 10•13 years ago
|
||
For those who don't know what the above comment means, I edited this file: data/template/template/en/default/global/userselect.html.tmpl changed this: #line 85 "template/en/default/global/userselect.html.tmpl" if ($stash->get('id') && $stash->get(['feature_enabled', [ 'jsonrpc' ]])) { $output .= "\n <div id=\""; #line 83 "template/en/default/global/userselect.html.tmpl" to this: #line 85 "template/en/default/global/userselect.html.tmpl" if ($stash->get('id') && $stash->get(['feature_disabled', [ 'jsonrpc' ]])) { $output .= "\n <div id=\""; #line 83 "template/en/default/global/userselect.html.tmpl" Someone let me know if this is correct.
Assignee | ||
Comment 11•13 years ago
|
||
You should not be editing the files under data/template as those are the compiled versions of the files in template/. The file you need to update is template/en/default/global/userselect.html.tmpl. You have just commented out the whole section about autocomplete in that field and gotten the same result. When the files under template/ are changed, the versions under data/template are updated automatically. dkl
Comment 12•13 years ago
|
||
Ok found it, so what do I change this section to? [% IF id && feature_enabled('jsonrpc') %] <div id="[% id FILTER html %]_autocomplete" [% IF classes %] class="[% classes.join(' ') FILTER html %]" [% END %]> [% END %]
Assignee | ||
Comment 13•13 years ago
|
||
(In reply to Carl from comment #12) > Ok found it, so what do I change this section to? > > [% IF id && feature_enabled('jsonrpc') %] > <div id="[% id FILTER html %]_autocomplete" > [% IF classes %] class="[% classes.join(' ') FILTER html %]" [% END > %]> > [% END %] That part you can leave alone. The part you need to change is: [% IF id && feature_enabled('jsonrpc') %] <div id="[% id FILTER html %]_autocomplete_container"></div> </div> <script type="text/javascript"> if( typeof(YAHOO.bugzilla.userAutocomplete) !== 'undefined' && YAHOO.bugzilla.userAutocomplete != null){ YAHOO.bugzilla.userAutocomplete.init( "[% id FILTER js %]", "[% id FILTER js %]_autocomplete_container" [% IF multiple %], true[% END%]); } </script> [% END %] to: [% IF id && feature_enabled('jsonrpc') %] <div id="[% id FILTER html %]_autocomplete_container"></div> </div> <script type="text/javascript"> // if( typeof(YAHOO.bugzilla.userAutocomplete) !== 'undefined' // && YAHOO.bugzilla.userAutocomplete != null){ // YAHOO.bugzilla.userAutocomplete.init( "[% id FILTER js %]", // "[% id FILTER js %]_autocomplete_container" // [% IF multiple %], true[% END%]); // } </script> [% END %] dkl
Comment 14•13 years ago
|
||
Ah now it is much faster. Thank you very much.
Comment 15•13 years ago
|
||
Yeah, we should have a param somewhere around where usermatching is now. It should default to on, though.
Comment 16•13 years ago
|
||
Either that or we should just always turn this off for mod_cgi installs, but I'd feel that was pretty extreme and a lot of people wouldn't know the feature was there, then.
Assignee | ||
Comment 17•13 years ago
|
||
New patch that adds a param to turn off autocompetion for user fields. dkl
Attachment #564591 -
Flags: review?(mkanat)
Comment 18•13 years ago
|
||
Comment on attachment 564591 [details] [diff] [review] Patch to add param to disable user autocompletion (v1) >+ name => 'userautocompletion', Please use user_autocompletion to make it more readable. All new parameters should use underscores to make param names more readable.
Attachment #564591 -
Flags: review?(mkanat) → review-
Comment 19•13 years ago
|
||
Comment on attachment 564591 [details] [diff] [review] Patch to add param to disable user autocompletion (v1) >=== modified file 'docs/en/xml/administration.xml' >+ when entered. Another setting can enable user autocompletion for certain "Another setting" is too vague. Which parameter exactly? Either we name it explicitly, or we don't talk about it. >+ a few characters. Note that it is highly recommended to use mod_perl when s/highly//, because it works with mod_cgi, especially on small and moderately large installations. >=== modified file 'template/en/default/admin/params/usermatch.html.tmpl' >+ userautocompletion => "If this option is set, typing characters in a certain user " _ >+ "fields will display a list of matches that can be selected from.", "a certain user fields" is probably wrong (a + plural?). Also, we should be more explicit and say "in user fields".
Assignee | ||
Comment 20•13 years ago
|
||
Thanks LpSolit. Here is a new patch addressing your suggestions. dkl
Attachment #564591 -
Attachment is obsolete: true
Attachment #564635 -
Flags: review?(mkanat)
Comment 21•13 years ago
|
||
I actually prefer if it just gives me a list instead of using autocomplete. Then I can use the arrow keys and just select which one I want. Like how google search does it but without the background refreshing of the search results. Also, it should only start searching if there is a break of say 5 seconds after a letter is typed. Otherwise you run into the above problem.
Comment 22•13 years ago
|
||
Comment on attachment 564635 [details] [diff] [review] Patch to add param to disable user autocompletion (v2) Review of attachment 564635 [details] [diff] [review]: ----------------------------------------------------------------- All the below can be fixed on checkin. ::: Bugzilla/Config/UserMatch.pm @@ +47,4 @@ > }, > > { > + name => 'user_autocompletion', I think this will be confusing with the non-ajax autocomplete. Perhaps call it ajax_user_autocomplete or something? ::: docs/en/xml/administration.xml @@ +862,5 @@ > contains parameters for how user names can be queried and matched > + when entered. Another setting called 'user_autocompletion' enables certain > + user fields to display a list of matched user names as a drop down after typing > + a few characters. Note that it is recommended to use mod_perl when > + enabling 'user_autocompletion'. This new text seems like it should be a separate paragraph. ::: template/en/default/global/userselect.html.tmpl @@ +79,4 @@ > [% END %] > </select> > [% ELSE %] > + [% IF id && feature_enabled('jsonrpc') && Param('user_autocompletion') %] Check the param first--feature_enabled could actually take some small bit of time that we could otherwise save. @@ +94,4 @@ > [% IF size %] size="[% size FILTER html %]" [% END %] > [% IF id %] id="[% id FILTER html %]" [% END %] > > > + [% IF id && feature_enabled('jsonrpc') && Param('user_autocompletion') %] Same note there about the param.
Attachment #564635 -
Flags: review?(mkanat) → review+
Updated•13 years ago
|
Assignee: general → dkl
Status: NEW → ASSIGNED
Flags: approval4.2+
Flags: approval4.0+
Flags: approval+
Target Milestone: --- → Bugzilla 4.0
Assignee | ||
Comment 23•13 years ago
|
||
Changes made and committed trunk: Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bugzilla/trunk modified Bugzilla/Mailer.pm modified Bugzilla/Config/UserMatch.pm modified docs/en/xml/administration.xml modified template/en/default/admin/params/usermatch.html.tmpl modified template/en/default/global/userselect.html.tmpl Committed revision 7994. 4.2: Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bugzilla/4.2 modified Bugzilla/Config/UserMatch.pm modified docs/en/xml/administration.xml modified template/en/default/admin/params/usermatch.html.tmpl modified template/en/default/global/userselect.html.tmpl Committed revision 7950. 4.0: Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bugzilla/4.0 modified Bugzilla/Config/UserMatch.pm modified docs/en/xml/administration.xml modified template/en/default/admin/params/usermatch.html.tmpl modified template/en/default/global/userselect.html.tmpl Committed revision 7656.
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•