Last Comment Bug 685552 - Email auto-completion causes server to thrash
: Email auto-completion causes server to thrash
Status: RESOLVED FIXED
:
Product: Bugzilla
Classification: Server Software
Component: Bugzilla-General (show other bugs)
: 4.0.2
: All All
: -- normal (vote)
: Bugzilla 4.0
Assigned To: David Lawrence [:dkl]
: default-qa
Mentors:
: 690488 720445 834510 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-09-08 08:29 PDT by Nathan Boy
Modified: 2013-01-25 02:02 PST (History)
5 users (show)
mkanat: approval+
mkanat: approval4.2+
mkanat: approval4.0+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Patch to add param to disable user autocompletion (v1) (2.87 KB, patch)
2011-10-04 09:52 PDT, David Lawrence [:dkl]
LpSolit: review-
Details | Diff | Review
Patch to add param to disable user autocompletion (v2) (2.88 KB, patch)
2011-10-04 12:29 PDT, David Lawrence [:dkl]
mkanat: review+
Details | Diff | Review

Description Nathan Boy 2011-09-08 08:29:50 PDT
User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.157 Safari/535.1

Steps to reproduce:

I run Bugzilla 4.0.2 using Apache Jetty's CGI Servlet.  The server is a slice with 256 MB of RAM and an additional 500 MB swap.  I produced this problem by viewing a reported bug in my web browser and clicking the "Edit" link next to the "Assigned To" field of the bug.  I then typed in the email address of a user.


Actual results:

The server I use to host our Bugzilla has become unresponsive periodically ever since we upgraded to Bugzilla 4.0.2.  Today I noticed that just before the server becomes unresponsive, a large number of jsonrpc.cgi calls are made.  Running top, the most active server processes look like this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                    
 2804 root      20   0 84964  16m 1704 D   24  6.6   0:07.11 jsonrpc.cgi                                                
 2813 root      20   0 84308  24m 1768 D   22 10.1   0:07.05 jsonrpc.cgi                                                
 2825 root      20   0 66916  17m 1692 D   19  7.3   0:04.09 jsonrpc.cgi                                                
 2801 root      20   0 81404  15m 1732 D   18  6.3   0:05.70 jsonrpc.cgi                                                
 2823 root      20   0 66916  17m 1692 D   16  7.3   0:03.89 jsonrpc.cgi                                                
 2807 root      20   0 84572  23m 1768 D   12  9.4   0:04.00 jsonrpc.cgi                                                
 2385 root      20   0  612m  27m  716 S   11 11.2   0:33.39 java                                                       
 2816 root      20   0 84964  25m 1768 D   11 10.4   0:06.27 jsonrpc.cgi                                                
 2810 root      20   0 85888  21m 1724 D   10  8.9   0:04.77 jsonrpc.cgi                                                
 2798 root      20   0 92476  14m 1676 D    7  5.8   0:06.14 jsonrpc.cgi      

Our server requires a reboot when enough of these calls are made to cause the server to exhaust its swap and become unresponsive.  It appears that a call to jsonrpc.cgi is made each time a key is pressed in the "Assigned To" field, presumably to aid in auto-completion.


Expected results:

Right now I am working around this by turning on the 'usemenuforusers' parameter so that emails can be chosen from a list rather than through email auto-completion.  Ideally auto-complete would not make this many resource-intensive calls to the server, though.
Comment 1 David Lawrence [:dkl] 2011-09-08 08:58:23 PDT
I can confirm that this happens on my local test instance used for testing BMO changes. When I type out an email address almost entirely, it will fire off a call to jsonrpc.cgi for each letter almost and I see all of them running in my process list. If enough are fired, the site becomes unresponsive and I have to wait til the start to finish up on their own or manually kill them. The workaround so far is to just type a small amount of the email address and let the autocomplete drop down appear. Even though I can normally type the whole address before autocomplete returns even for a small result list.

Is there a way to tweak the autocomplete settings to wait a larger amount of time before calling jsonrpc.cgi or even keep a count of how many it is fired so far and wait til the number gets below a certain threshold? I will try to experiment if I find some time.

dkl
Comment 2 Frédéric Buclin 2011-09-08 10:02:06 PDT
I don't know how many matches are returned, but couldn't the code call jsonrpc.cgi only once to get an initial list, and then filter the list based on new characters entered by the user? It would then only re-request a new list if a character is removed.
Comment 3 David Lawrence [:dkl] 2011-09-08 10:14:57 PDT
Confirmed
Comment 4 David Lawrence [:dkl] 2011-09-09 08:46:00 PDT
According to the javascript console, the autocomplete widget is cancelling any prior requests it has made each time a new letter is entered in the text field. For for example the string "dkl@moz" will fire off a jsonrpc.cgi request 4 times, but each subsequent request cancels the ones before it. So according to autocomplete it only has 1 request running at any given time. 

But on the apache side on my instance, I see each of the jsonrpc.cgi processes still running in the process list and apache defaults to killing them off after a specified amount of time. Apache is not recognizing that the request was cancelled by the client (similar to hitting the stop button in Firefox) and just lets them run til timeout.

So if I type "dkl@mozilla.co", I will have 11 defunct jsonrpc.cgi requests running on my server. Setting a num of requests threshold in the autocomplete code in js/field.js won't do anything since autocomplete is already cancelling any outstanding requests anyway. 

LpSolits idea of sending only one request and then have the autocomplete code filter the full results based on additional keys type is a compelling solution but depending on what the user types initially, it could be a very large result list depending on the Bugzilla installation.

Is there an Apache directive/setting that will make Apache properly kill off a process once it receives the cancel signal from the client instead of letting the processes run to completion or get killed off from some timeout setting?

dkl
Comment 5 Max Kanat-Alexander 2011-09-12 17:18:28 PDT
(In reply to David Lawrence [:dkl] from comment #1)
> Is there a way to tweak the autocomplete settings to wait a larger amount of
> time before calling jsonrpc.cgi 

  Yes, however, this would introduce a perceptible delay in receiving results for every single autocomplete query. Also, you can't guess how quickly people are going to type vs. how fast somebody's Bugzilla server is going to be. In other words, not only would this slow things down from the user's perspective, it probably wouldn't even help.

  Nathan's problem is that his hardware will never be able to support autocomplete or any of Bugzilla's future AJAX functionality, under basically any realistic circumstances. If you are running Bugzilla under mod_cgi, every single request takes a very long time, even if it's going to return a tiny amount of data or do a tiny amount of work. That means that if every user produces several .cgi requests on every page or on every interaction, they will all live for a long time and consume quite a bit of RAM because they are all alive at once.

  Nathan is the only user who's ever reported to me that they're running Bugzilla with so little RAM. I'm impressed that previous versions of Bugzilla worked so well in that situation! :-) Probably what Nathan wants to do is disable every AJAX feature of Bugzilla (presently the automatic duplicate detection and the AJAX autocomplete). This would currently require some small amount of code hackery.

  (I'm also going to respond to dkl's other comment, here, which is also relevant.)
Comment 6 Max Kanat-Alexander 2011-09-12 17:29:55 PDT
(In reply to David Lawrence [:dkl] from comment #4)
> So if I type "dkl@mozilla.co", I will have 11 defunct jsonrpc.cgi requests
> running on my server. 

  How long are you waiting between typing each letter? Are you typing at full speed? The present delay is 50ms. It's "userAutoComp.queryDelay = 0.05;" in js/field.js. 50ms is barely perceptible, but 100ms would be perceptible, particularly on mod_perl installations. (Also, slowing down mod_cgi installs even more seems bad.)

> LpSolits idea of sending only one request and then have the autocomplete
> code filter the full results based on additional keys type is a compelling
> solution

  Except that it wouldn't work, because of maxusermatches. The longer list is not actually a complete list, always. Also, what happens when the user hits backspace? What happens when they cut and paste? The complexity there becomes high very quickly.

> Is there an Apache directive/setting that will make Apache properly kill off
> a process once it receives the cancel signal from the client

  Your question assumes that there is such a thing as a "cancel signal". I suppose if the browser was nice, Apache would get a FIN from the browser, at the TCP level. And yeah, if we could detect that, we could...ah, no, actually not, come to think of it.

  This goes back to the historical problem of "what happens when the user closes the browser when process_bug.cgi is still running on a very long transaction, say for changing 10,000 bugs?" The fact that Apache forcibly times us out in that situation after several seconds of a closed connection is already bad; making it happen immediately would be even worse.


  Perhaps a better solution would be to set the queryDelay higher on mod_cgi installs, but I'm concerned that it would have to be abnormally high to not cause the problems that you're experiencing. (And there's basically no setting high enough that it would work on Nathan's system while still being actually useful to users--Nathan's users are better off with the old post-submit user-match suggestions.)
Comment 7 Max Kanat-Alexander 2011-09-29 12:48:07 PDT
*** Bug 690488 has been marked as a duplicate of this bug. ***
Comment 8 Carl 2011-09-29 12:53:44 PDT
Can I disable this for now as it's practically impossible to add components now.
Comment 9 David Lawrence [:dkl] 2011-09-30 08:32:41 PDT
(In reply to Carl from comment #8)
> Can I disable this for now as it's practically impossible to add components
> now.

Currently you will have to edit the template code in global/userselect.html.tmpl to turn it off as it is automatically enabled if you have jsonrpc support enabled.

Perhaps we need a param for this to turn the feature on/off?

dkl
Comment 10 Carl 2011-09-30 13:10:09 PDT
For those who don't know what the above comment means, I edited this file:

data/template/template/en/default/global/userselect.html.tmpl

changed this:

#line 85 "template/en/default/global/userselect.html.tmpl"
    if ($stash->get('id') && $stash->get(['feature_enabled', [ 'jsonrpc' ]])) {
    $output .=  "\n    <div id=\"";
#line 83 "template/en/default/global/userselect.html.tmpl"

to this:

#line 85 "template/en/default/global/userselect.html.tmpl"
    if ($stash->get('id') && $stash->get(['feature_disabled', [ 'jsonrpc' ]])) {
    $output .=  "\n    <div id=\"";
#line 83 "template/en/default/global/userselect.html.tmpl"

Someone let me know if this is correct.
Comment 11 David Lawrence [:dkl] 2011-09-30 13:16:15 PDT
You should not be editing the files under data/template as those are the compiled versions of the files in template/. The file you need to update is template/en/default/global/userselect.html.tmpl. You have just commented out the whole section about autocomplete in that field and gotten the same result. When the files under template/ are changed, the versions under data/template are updated automatically.

dkl
Comment 12 Carl 2011-09-30 14:24:12 PDT
Ok found it, so what do I change this section to?

  [% IF id && feature_enabled('jsonrpc') %]
    <div id="[% id FILTER html %]_autocomplete" 
         [% IF classes %] class="[% classes.join(' ') FILTER html %]" [% END %]>
  [% END %]
Comment 13 David Lawrence [:dkl] 2011-09-30 14:28:33 PDT
(In reply to Carl from comment #12)
> Ok found it, so what do I change this section to?
> 
>   [% IF id && feature_enabled('jsonrpc') %]
>     <div id="[% id FILTER html %]_autocomplete" 
>          [% IF classes %] class="[% classes.join(' ') FILTER html %]" [% END
> %]>
>   [% END %]

That part you can leave alone.

The part you need to change is:

  [% IF id && feature_enabled('jsonrpc') %]
    <div id="[% id FILTER html %]_autocomplete_container"></div>
    </div>
    <script type="text/javascript">
      if( typeof(YAHOO.bugzilla.userAutocomplete) !== 'undefined' 
          && YAHOO.bugzilla.userAutocomplete != null){
        YAHOO.bugzilla.userAutocomplete.init( "[% id FILTER js %]", 
                    "[% id FILTER js %]_autocomplete_container"
                    [% IF multiple %], true[% END%]);        
      }
    </script>
  [% END %]

to:

  [% IF id && feature_enabled('jsonrpc') %]
    <div id="[% id FILTER html %]_autocomplete_container"></div>
    </div>
    <script type="text/javascript">
      // if( typeof(YAHOO.bugzilla.userAutocomplete) !== 'undefined' 
      //    && YAHOO.bugzilla.userAutocomplete != null){
      // YAHOO.bugzilla.userAutocomplete.init( "[% id FILTER js %]", 
      //              "[% id FILTER js %]_autocomplete_container"
      //               [% IF multiple %], true[% END%]);        
      // }
    </script>
  [% END %]

dkl
Comment 14 Carl 2011-09-30 17:51:41 PDT
Ah now it is much faster.  Thank you very much.
Comment 15 Max Kanat-Alexander 2011-10-03 22:15:25 PDT
Yeah, we should have a param somewhere around where usermatching is now. It should default to on, though.
Comment 16 Max Kanat-Alexander 2011-10-03 22:16:01 PDT
Either that or we should just always turn this off for mod_cgi installs, but I'd feel that was pretty extreme and a lot of people wouldn't know the feature was there, then.
Comment 17 David Lawrence [:dkl] 2011-10-04 09:52:30 PDT
Created attachment 564591 [details] [diff] [review]
Patch to add param to disable user autocompletion (v1)

New patch that adds a param to turn off autocompetion for user fields. 

dkl
Comment 18 Frédéric Buclin 2011-10-04 09:56:52 PDT
Comment on attachment 564591 [details] [diff] [review]
Patch to add param to disable user autocompletion (v1)

>+   name => 'userautocompletion', 

Please use user_autocompletion to make it more readable. All new parameters should use underscores to make param names more readable.
Comment 19 Frédéric Buclin 2011-10-04 10:04:10 PDT
Comment on attachment 564591 [details] [diff] [review]
Patch to add param to disable user autocompletion (v1)

>=== modified file 'docs/en/xml/administration.xml'

>+            when entered. Another setting can enable user autocompletion for certain

"Another setting" is too vague. Which parameter exactly? Either we name it explicitly, or we don't talk about it.


>+            a few characters. Note that it is highly recommended to use mod_perl when

s/highly//, because it works with mod_cgi, especially on small and moderately large installations.



>=== modified file 'template/en/default/admin/params/usermatch.html.tmpl'

>+  userautocompletion => "If this option is set, typing characters in a certain user " _
>+                        "fields will display a list of matches that can be selected from.", 

"a certain user fields" is probably wrong (a + plural?). Also, we should be more explicit and say "in user fields".
Comment 20 David Lawrence [:dkl] 2011-10-04 12:29:42 PDT
Created attachment 564635 [details] [diff] [review]
Patch to add param to disable user autocompletion (v2)

Thanks LpSolit. Here is a new patch addressing your suggestions.

dkl
Comment 21 Carl 2011-10-04 15:45:37 PDT
I actually prefer if it just gives me a list instead of using autocomplete.  Then I can use the arrow keys and just select which one I want.  Like how google search does it but without the background refreshing of the search results.

Also, it should only start searching if there is a break of say 5 seconds after a letter is typed.  Otherwise you run into the above problem.
Comment 22 Max Kanat-Alexander 2011-10-20 15:10:54 PDT
Comment on attachment 564635 [details] [diff] [review]
Patch to add param to disable user autocompletion (v2)

Review of attachment 564635 [details] [diff] [review]:
-----------------------------------------------------------------

All the below can be fixed on checkin.

::: Bugzilla/Config/UserMatch.pm
@@ +47,4 @@
>    },
>  
>    {
> +   name => 'user_autocompletion', 

I think this will be confusing with the non-ajax autocomplete. Perhaps call it ajax_user_autocomplete or something?

::: docs/en/xml/administration.xml
@@ +862,5 @@
>              contains parameters for how user names can be queried and matched
> +            when entered. Another setting called 'user_autocompletion' enables certain
> +            user fields to display a list of matched user names as a drop down after typing 
> +            a few characters. Note that it is recommended to use mod_perl when
> +            enabling 'user_autocompletion'.

This new text seems like it should be a separate paragraph.

::: template/en/default/global/userselect.html.tmpl
@@ +79,4 @@
>    [% END %]
>  </select>
>  [% ELSE %]
> +  [% IF id && feature_enabled('jsonrpc') && Param('user_autocompletion') %]

Check the param first--feature_enabled could actually take some small bit of time that we could otherwise save.

@@ +94,4 @@
>      [% IF size %] size="[% size FILTER html %]" [% END %]
>      [% IF id %] id="[% id FILTER html %]" [% END %]
>    >
> +  [% IF id && feature_enabled('jsonrpc') && Param('user_autocompletion') %]

Same note there about the param.
Comment 23 David Lawrence [:dkl] 2011-10-24 15:04:21 PDT
Changes made and committed

trunk:
Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bugzilla/trunk
modified Bugzilla/Mailer.pm
modified Bugzilla/Config/UserMatch.pm
modified docs/en/xml/administration.xml
modified template/en/default/admin/params/usermatch.html.tmpl
modified template/en/default/global/userselect.html.tmpl                                                                         Committed revision 7994.

4.2:
Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bugzilla/4.2
modified Bugzilla/Config/UserMatch.pm
modified docs/en/xml/administration.xml
modified template/en/default/admin/params/usermatch.html.tmpl
modified template/en/default/global/userselect.html.tmpl
Committed revision 7950.

4.0:
Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bugzilla/4.0
modified Bugzilla/Config/UserMatch.pm
modified docs/en/xml/administration.xml
modified template/en/default/admin/params/usermatch.html.tmpl
modified template/en/default/global/userselect.html.tmpl
Committed revision 7656.
Comment 24 Frédéric Buclin 2012-01-23 11:31:53 PST
*** Bug 720445 has been marked as a duplicate of this bug. ***
Comment 25 Frédéric Buclin 2013-01-25 02:02:58 PST
*** Bug 834510 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.