The default bug view has changed. See this FAQ.

Having a very large number of custom fields can make displaying show_bug.cgi slow

RESOLVED FIXED in Bugzilla 4.0

Status

()

Bugzilla
Creating/Changing Bugs
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: Wu Di, Assigned: Frédéric Buclin)

Tracking

({perf})

Bugzilla 4.0
Bug Flags:
approval +
approval4.0 +
blocking4.0.2 +
blocking4.0.1 -

Details

Attachments

(1 attachment, 2 obsolete attachments)

8.15 KB, patch
glob
: review+
Details | Diff | Splinter Review
(Reporter)

Description

6 years ago
User-Agent:       Mozilla/5.0 (Windows NT 5.1; rv:2.0b11) Gecko/20100101 Firefox/4.0b11
Build Identifier: 4.0

Many duplicated queries be executed makes Bugzilla slow when show a BUG...

Reproducible: Always

Steps to Reproduce:
1. Define many custom fields.
2. Add 'Field only appears when' visible limitation to some of them.
3. Enter a BUG, many take more than 5 secs.
Actual Results:  
Enter a BUG, many take more than 5 secs.

Expected Results:  
1~2 secs.
(Reporter)

Comment 1

6 years ago
Created attachment 513019 [details]
Query Log

Comment 2

6 years ago
  Thanks for the report. Could you tell me how many custom fields you have and how many of them have a "field only appears when" restriction? I need to reproduce this for myself.
OS: Linux → Windows CE
Summary: Many duplicated queries be executed makes Bugzilla slow when show a BUG... → Many custom fields with "Field only appears when" restrictions can make show_bug.cgi very slow
(Reporter)

Comment 3

6 years ago
87 custom fields we have.
Almost all of them have visibility restriction.

Comment 4

6 years ago
All right. And do you also have a lot of value restrictions? That is, "value A is only visible when field B is set to C"?
(Reporter)

Comment 5

6 years ago
Yes, all of them have.
(Reporter)

Comment 6

6 years ago
Created attachment 513063 [details]
query_log_without_restric

I've tried to delete all restrictions, it still very slow...

Comment 7

6 years ago
Are you running under mod_perl? Bugzilla is always slow unless you are running under mod_perl.
OS: Windows CE → Windows Server 2003
(Reporter)

Comment 8

6 years ago
Yes, I running it under mod_perl... but it's very fast here, only except show_bug.cgi ... And I have tried to run it under cgi mode a moment ago, it's still very slow...

See '110217 16:29:45' -> '110217 16:29:49' in Attachment 513063 [details] , only 5 queries but takes 4 secs., and 100% CPU occupation(mod_perl), I have tried these 5 queries in MySQL client, only 0.022m per query...
So, I guess there must have some loops there with too many custom fields?

Comment 9

6 years ago
Okay, we can look into the general case of this being slow.

Were you using a previous version of Bugzilla with this many custom fields? If you were, was Bugzilla slow on that old version as well?
OS: Windows Server 2003 → All
Summary: Many custom fields with "Field only appears when" restrictions can make show_bug.cgi very slow → Having a very large number of custom fields can make displaying show_bug.cgi slow
(Reporter)

Comment 10

6 years ago
I used 2.0.5, 3.0.6, 3.4.1, 3.4.2, 3.4.5, 3.5.3, 3.6.1, 3,6.2,
never be that slow in my memory... just in 4.0...
May be I could downgrade to older version to retest it someday...
OS: All → Windows Server 2003
(Reporter)

Comment 11

6 years ago
(In reply to comment #10)
> I used 2.0.5
3.0.5
(Assignee)

Comment 12

6 years ago
(In reply to comment #10)
> May be I could downgrade to older version to retest it someday...

Note that you cannot downgrade Bugzilla. You would have to test on a backup of an older Bugzilla.

The good news is that 87 custom fields is pretty uncommon.
Assignee: general → create-and-change
Component: Bugzilla-General → Creating/Changing Bugs
OS: Windows Server 2003 → All
Hardware: x86 → All
Version: unspecified → 4.0

Comment 13

6 years ago
Wu Di: Are all your custom fields single-select fields, or are they all types? How many of each type of field do you have? (This would help me with testing.)

Comment 14

6 years ago
Glob, do you want to investigate this when you have some time?
(Reporter)

Comment 15

6 years ago
(In reply to comment #13)
> Wu Di: Are all your custom fields single-select fields, or are they all types?
> How many of each type of field do you have? (This would help me with testing.)

Date/Time x 11
Drop Down x 24
Free Text x 41
Large Text Box x 11
Multiple-Selection Box x 5

(In reply to comment #14)
> Glob, do you want to investigate this when you have some time?

I will if I have some time.
(Reporter)

Comment 16

6 years ago
We import BUG information from a CVS file which export by customer, and export a formatted Excel to them, by using a extension made by us. So it'll be many custom fields needs to be added according to the format which we decided. So, good performance with many custom fields is very important for us. :P
(Reporter)

Comment 17

6 years ago
I deleted line: [% PROCESS "bug/field-label.html.tmpl" %]
in field.html.tmpl, problem seems be resolved (about 2 secs).
There are something wrong the field-label making...
(Reporter)

Comment 18

6 years ago
I deleted line: [% PROCESS "bug/field-help.none.tmpl" %]
in field-lable.html.tmpl, it fast enough (about 2-3 secs).
(Assignee)

Comment 19

6 years ago
(In reply to comment #18)
> I deleted line: [% PROCESS "bug/field-help.none.tmpl" %]
> in field-lable.html.tmpl, it fast enough (about 2-3 secs).

The reason is probably that field-label calls field-help for *each* label, which is totally useless (a single call from field.html.tmpl would be enough). The problem is twofold:

1) [% INCLUDE bug/field.html.tmpl %] from bug/edit.html.tmpl means that the data is cleared between each call to bug/field.html.tmpl (we need to use PROCESS to keep the data between calls), so there is no way to implicitly cache the content of %help_html (defined in field-help).

2) field-help loops over each field for each label. So if you have 90 custom fields, you add the ~60 hardcoded fields, and you get that the code is called (90+60)² times, i.e. 22,500 times. Remember that a single loop would be enough!!!

Confirming the bug as this is an obvious waste of ressource here, being significant or not.
Status: UNCONFIRMED → NEW
Ever confirmed: true
(Assignee)

Updated

6 years ago
Keywords: perf

Comment 20

6 years ago
Yeah, I realized the same thing when Wu Di explained this.

The first fix I'd try is [% RETURN IF field_help.defined %] (or whatever the variable is called) at the top of the template.

Comment 21

6 years ago
This may have a simple resolution that can easily be added to 4.0, so we should at least investigate this for 4.0. If the resolution requires significant refactoring, though, it will only go into 4.2.
Flags: blocking4.0.1+
Target Milestone: --- → Bugzilla 4.0
(Assignee)

Comment 22

6 years ago
(In reply to comment #20)
> The first fix I'd try is [% RETURN IF field_help.defined %] (or whatever the
> variable is called) at the top of the template.

This won't work as we use INCLUDE, not PROCESS. The hash is not persistent.
(Assignee)

Updated

6 years ago
Flags: blocking4.0.2+
Flags: blocking4.0.1-
Flags: blocking4.0.1+
(Assignee)

Comment 23

6 years ago
Created attachment 549610 [details] [diff] [review]
patch, v1

As calling bug/field-help.none.tmpl once per field label doesn't make sense, we load it only once as we do for global/field-descs.none.tmpl. NYTProf shows a real improvement with 15 custom fields (12ms vs 238ms spent in this template).
Assignee: create-and-change → LpSolit
Attachment #513019 - Attachment is obsolete: true
Attachment #513063 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #549610 - Flags: review?(glob)
Comment on attachment 549610 [details] [diff] [review]
patch, v1

Review of attachment 549610 [details] [diff] [review]:
-----------------------------------------------------------------

r=glob

note: this patch doesn't apply cleanly to 4.0
Attachment #549610 - Flags: review?(glob) → review+
(Assignee)

Comment 25

6 years ago
(In reply to comment #24)
> note: this patch doesn't apply cleanly to 4.0

Yeah, there is a small bitrot in Bugzilla/Template.pm due to an unrelated code change. I tested, and the patch works the same way. Thanks for the review.
Flags: approval4.0+
Flags: approval+
(Assignee)

Comment 26

6 years ago
If there are other useful improvements to do, we should file separate bugs for them (e.g. to decrease the time spent in bug/field-events.js.tmpl, which is also very time consuming).

Committing to: bzr+ssh://lpsolit%40gmail.com@bzr.mozilla.org/bugzilla/trunk/
modified Bugzilla/Template.pm
modified template/en/default/bug/field-help.none.tmpl
modified template/en/default/bug/field-label.html.tmpl
modified template/en/default/global/field-descs.none.tmpl
modified template/en/default/pages/fields.html.tmpl
Committed revision 7878.

Committing to: bzr+ssh://lpsolit%40gmail.com@bzr.mozilla.org/bugzilla/4.0/
modified Bugzilla/Template.pm
modified template/en/default/bug/field-help.none.tmpl
modified template/en/default/bug/field-label.html.tmpl
modified template/en/default/global/field-descs.none.tmpl
modified template/en/default/pages/fields.html.tmpl
Committed revision 7629.
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED

Comment 27

6 years ago
Comment on attachment 549610 [details] [diff] [review]
patch, v1

Review of attachment 549610 [details] [diff] [review]:
-----------------------------------------------------------------

::: template/en/default/bug/field-help.none.tmpl
@@ +25,2 @@
>  
> +[% RETURN UNLESS in_template_var %]

Could you simply remove all of the callers instead of adding this RETURN? There aren't that many field-help callers.

::: template/en/default/global/field-descs.none.tmpl
@@ +149,5 @@
>           IF !vars.field_descs.${bz_field.name}.defined %]
>      [% END %]
>    [% END %]
> +
> +  [% PROCESS "bug/field-help.none.tmpl" %]

Instead of adding this here, it might be better to load it in template_var when it is requested. This will add some overhead to every template_var call, as field-help is larger than all of our other stuff combined.
(Assignee)

Comment 28

6 years ago
(In reply to comment #27)
> > +[% RETURN UNLESS in_template_var %]
> 
> Could you simply remove all of the callers instead of adding this RETURN?
> There aren't that many field-help callers.

I already removed them. The reason I left RETURN here is to prevent future callers from calling this template, in case future reviewers or patchers are unaware of this change. It doesn't hurt.


> Instead of adding this here, it might be better to load it in template_var
> when it is requested. This will add some overhead to every template_var
> call, as field-help is larger than all of our other stuff combined.

Currently, template_var() explicitly calls field-descs.none.tmpl so it would need to be refactored, which I didn't want to do in 4.0 and 4.2 as they are closed for enhancements. Also, the most visited pages are bug reports and the search page, which all need this template. So the penalty seems pretty inexistent here. But I thought about that too. Maybe you could file a separate bug for this?

Comment 29

6 years ago
(In reply to comment #28)
> I already removed them. The reason I left RETURN here is to prevent future
> callers from calling this template, in case future reviewers or patchers are
> unaware of this change. It doesn't hurt.

  Okay, I suppose that's reasonable.

>. Also, the most visited pages are bug reports and
> the search page, which all need this template. So the penalty seems pretty
> inexistent here. But I thought about that too. Maybe you could file a
> separate bug for this?

  That sounds reasonable. I suppose let's actually see if there's any problem with it before we file a bug.
You need to log in before you can comment on or make changes to this bug.