24697 - Display search link for open bugs for each keyword in describekeywords.cgi

Status: RESOLVED FIXED

(Bugzilla :: Bugzilla-General, enhancement, P2)

Description

[leger]

- Go to the Keywords page: http://bugzilla.mozilla.org/describekeywords.cgi 
- Click on one of the links in the column at the right of the page which shows 
the bug count corresponding to the keyword. For instance, click on the link that 
shows the bug count for 'css1'. As of today, it shows a count of 121. 

Problem: The bug list shows all the bugs corresponding to the keyword regardless 
of their state. 

Expected result: The bug list should not show the bugs that have been resolved 
or fixed. The bug count should only show the opened bugs. 

Or maybe there should that column should be All Bugs and another one Open Bugs? 
what do you think?

Comment 1

[Terry Weissman]

I knew someone was going to whine about this sooner or later...

The performance will go way down if it has to actually go in and look at each
bug and determine if it is opened or not.  I should play with it someday and see
how much it will affect it, but I'm pretty sure it will end up slowing down the
page to the point of unusability.  Leaving this bug open until I do so.

Total number of bugs per keyword is easy, though, so that's what I did.

Comment 2

[Terry Weissman]

OK, actually, rereading this, I realize there is a compromise I can easily do:

- Display the total number of bugs; if you click on that, it will bring you to a
buglist showing all those bugs.
- Also display a link that says "Show open bugs" (or some such; that's too
wordy).  Clicking that link will bring you to a buglist of the open bugs with
that keyword.

The thing I probably can't do is display the *count* of open bugs with a given
keyword.

Comment 3

[Terry Weissman]

[e-mail] is the new owner of Bugzilla and Bonsai.  (For details,
see my posting in netscape.public.mozilla.webtools,
news://news.mozilla.org/38F5D90D.F40E8C1A%40geocast.com .)

Comment 4

[Jesse Ruderman]

It seems to me like the describekeywords.cgi only updates every few days, so I 
don't see why performance is an issue.

Comment 5

[Daniel Bratell]

To only see the number of open bugs in each category would be a great 
improvement and could also be used to collect statistics and easily find 
forgotten bugs. ("What, we shouldn't have any html4 bugs left?")

Comment 6

[leger]

Updating QA Contact

Comment 7

[Jesse Ruderman]

*** Bug 68222 has been marked as a duplicate of this bug. ***

Comment 8

[Matthew Tuck [:CodeMachine]]

2.14 Nomination:  Evaluate feasability.

Comment 9

[Dave Miller [:justdave]]

moving to real milestones...

Comment 10

[Matthew Tuck [:CodeMachine]]

*** Bug 74307 has been marked as a duplicate of this bug. ***

Comment 11

[Gervase Markham [:gerv]]

I think the right fix for this is for the numbers of open bugs to be collected 
nightly by collectstats.pl and for this to reference the latest numbers 
collected. That way, we can get the useful figures without a performance hit 
every time the page gets loaded.

Gerv

Comment 12

[Matthew Tuck [:CodeMachine]]

I'd be a little wary about this containing potentially out-of-date information. 
Maybe this could be cached somewhere.

Comment 13

[Gervase Markham [:gerv]]

24 hours is hardly out-of-date...

Gerv

Comment 14

[Matthew Tuck [:CodeMachine]]

I would have considered one day out of date to be out of date.  I can just see a
whole bunch of bug reports filed on this.

Comment 15

[Matthew Tuck [:CodeMachine]]

Erm, make that one second out of date.

Comment 16

[Gervase Markham [:gerv]]

We can just head the column:

"Bug counts (updated daily)".

<shrug> It's _so_ much better than the current numbers, which are confusing, 
depressing (because people think there are that many left to fix) and not a 
useful measure of anything.

Gerv

Comment 17

[Gervase Markham [:gerv]]

We can just head the column:

"Bug counts (updated daily)".

<shrug> It's _so_ much better than the current numbers, which are confusing, 
depressing (because people think there are that many left to fix) and not a 
useful measure of anything.

Gerv

Comment 18

[Jacob Steenhagen]

*** Bug 78042 has been marked as a duplicate of this bug. ***

Comment 19

[Dave Miller [:justdave]]

*** Bug 89244 has been marked as a duplicate of this bug. ***

Comment 20

[Gervase Markham [:gerv]]

OK, here's the plan. We add a new field to the keyworddefs table - "count". This
is updated nightly by collectstats.pl, and printed on the keywords list in a
column headed "Bug count (updated daily)".

Do we have to take into account the fact that some people may not be running
collectstats.pl? Should we have a config option such that, for small
installations, the figures could be real-time?

Gerv

Comment 21

[Zach Lipton [:zach]]

moving

Comment 22

[GavinS]

Attachment: version 1 patch for counting number of OPEN bugs linked to KW

[Nice new create attachment page - cool!]

Apologies if anyone else was working on this, but here is version 1 of a patch,
implementing Gerv's latest suggestion. (Against 2.14 I'm afraid -- that's as
up-to-date as I am!) Some points:

1) I used a mediumint type for the new DB field. Is that MySQL specific? Should
I use something else?

2) I've also kept the current count of ALL bugs linked to a KW. Should it still
be there? If so, should it be cached as well?

3) I haven't tried a checksetup.pl with a new DB install, just an update with
my current installation.

4) I think there should be a better way of getting the concept of 'OPEN' bugs
that just that just coding 'bug_status=blah...'. Should there be a
get_open_bug_url_param() routine?

5) Should any tables be locked when counting bugs?

Comment 23

[Gervase Markham [:gerv]]

I'm not sure if this is the best way to do this. I'd like to avoid putting stats
information in random tables. How about the following solution: the page
displays the total count, but there's also a query link which gives you the open
count for that keyword? It's normally only one or two keywords you are
interested in.

This will be more sane when you can do queries where you can just ask for a
count, but in the meantime a buglist will do fine.

Gerv

Comment 24

[Matthew Tuck [:CodeMachine]]

Before we go any further, let me ask - has anyone actually tried a query to see
if determining this in real time would actually be slow?

Comment 25

[Gervase Markham [:gerv]]

The only people who could do that would be myk and dawn, because such a query
would have to be run on b.m.o . And you'd have to multiply the time taken by the
number of keywords we have - currently 109.

I still think my link idea is the quickest fix.

Gerv

Comment 26

[Gervase Markham [:gerv]]

Attachment: Gerv's idea v.1

How about this?

Gerv

Comment 27

[Matthew Tuck [:CodeMachine]]

Yes, that's what I meant.  Myk, would you please be able to run and let us know
how long it takes.  The appropriate query (I think) is:

SELECT keyworddefs.name,keyworddefs. description, COUNT(keywords.bug_id) FROM
keyworddefs LEFT JOIN keywords ON keyworddefs.id = keywords.keywordid LEFT JOIN
bugs ON keywords.bug_id = bugs.bug_id AND bugs.resolution = '' GROUP BY
keywords.keywordid ORDER BY keyworddefs.name;

and compare it to:

SELECT keyworddefs.name, keyworddefs.description, COUNT(keywords.bug_id), FROM
keyworddefs LEFT JOIN keywords ON keyworddefs.id = keywords.keywordid GROUP BY
keywords.keywordid ORDER BY keyworddefs.name;

Comment 28

[GavinS]

Having the count of open bugs is a good thing, and I think we should be able to
show it somehow...

Hopefully, the SQL will be quick enough to include the counts directly.

If not, and you don't want to put stats info into the keyworddefs table:

1) What about calling the info 'cached data for performance reasons', instead of
'stats info' ?

2) What about having a link on the normal describekeywords page to 'describe
keywords with open-bug counts (slower)'

3) Eventually, would this page be able to use the stats generated for bug#109034 ?

Comment 29

[Myk Melez [:myk] [@mykmelez]]

The first query takes 11.36 seconds to run on the "bugs" database on b.m.o.
tonight, and the second query takes 6.63 seconds to run.

Comment 30

[Gervase Markham [:gerv]]

So, having the counts on the page is out, then :-) 

Shall we go with my link patch?

Gerv

Comment 31

[Jesse Ruderman]

That's 11 seconds each day, right?  That doesn't sound like too much, 
especially if the query is run at some time when the load on bugzilla is low 
(not right after a database sync).

By the way, if a search for open bugs takes twice as long as a search for all 
bugs, I think something's wrong with the query or the way the database is set 
up.  It seems to me that searches for open bugs should be optimized, because 
most searches are for open bugs.

Comment 32

[Gervase Markham [:gerv]]

Oh, that's 11 seconds for the whole lot? Oops.

Well, then we could probably do it live - if it's only an extra 5 seconds on the
current time. Or it could be a param to the CGI.

Gerv

Comment 33

[Matthew Tuck [:CodeMachine]]

There are a couple of places where we only do those counts only upon a param, so
it doesn't seem like a bad idea.  However, I would still like to show the total
param alongside the open param.  This will probably be possible after bug
#106589 is done.

As for speeding up these queries, I welcome your suggestion as to how to improve
the query or database schema without sacrificing database normality.

I'm not entirely comfortable with an extra 5 seconds straight off, I sure wish
we could return results from MySQL incrementally.

Comment 34

[Dave Miller [:justdave]]

We are currently trying to wrap up Bugzilla 2.16.  We are now close enough to
release time that anything that wasn't already ranked at P1 isn't going to make
the cut.  Thus this is being retargetted at 2.18.  If you strongly disagree with
this retargetting, please comment, however, be aware that we only have about 2
weeks left to review and test anything at this point, and we intend to devote
this time to the remaining bugs that were designated as release blockers.

Comment 35

[Matthew Tuck [:CodeMachine]]

My above comment above implementation of open vs. total was a bit cryptic, my
intention is that when resolution is NULL when there is none, you will be able
to do COUNT(*) and COUNT(resolution).

Comment 36

[Aleksey Nogin]

Currently http://bugzilla.mozilla.org/describekeywords.cgi does not show any
counts at all :-(

Comment 37

[Gervase Markham [:gerv]]

Yes; it's been disabled because it was putting too heavy a load on the server.

Gerv

Comment 38

[Matthew Tuck [:CodeMachine]]

Heavy load as in holding locks, or heavy load as in lots of disk thrashing?  If
the former, why not use shadow instead?

Comment 39

[Gervase Markham [:gerv]]

You'd need to ask Myk - he knows more.

Gerv

Comment 40

[Myk Melez [:myk] [@mykmelez]]

>Heavy load as in holding locks, or heavy load as in lots of disk thrashing?  If
>the former, why not use shadow instead?

Holding locks, but it did that (blocking other queries) on the shadow database
as well.

Comment 41

[David Lawrence [:dkl]]

Attachment: Revised patch for CVS tip

Comment 42

[Jouni Heikniemi]

Comment on attachment 109837 [details] [diff] [review]
Revised patch for CVS tip

Bitrotten.

Comment 43

[Dave Miller [:justdave]]

enhancements without current patches are being pushed to 2.20

Comment 44

[timeless]

Comment on attachment 109837 [details] [diff] [review]
Revised patch for CVS tip

+	 <th align="left">Open Bugs</th>
+	 <a
href="buglist.cgi?bug_status=NEW&bug_status=VERIFIED&bug_status=ASSIGNED&bug_st
atus=REOPENED&keywords=[% keyword.name FILTER url_quote %]">[% keyword.bugcount
%]</a>

verified bugs?? what kind of query is this. ignoring bitrot, this isn't right.
it also doesn't match the query that triggers the count

&resolution=--- is all you need.

Comment 45

[Dave Miller [:justdave]]

Bugzilla 2.20 feature set is now frozen as of 15 Sept 2004.  Anything flagged
enhancement that hasn't already landed is being pushed out.  If this bug is
otherwise ready to land, we'll handle it on a case-by-case basis, please set the
blocking2.20 flag to '?' if you think it qualifies.

Comment 46

[Dave Miller [:justdave]]

Reassigning bugs that I'm not actively working on to the default component owner
in order to try to make some sanity out of my personal buglist.  This doesn't
mean the bug isn't being dealt with, just that I'm not the one doing it.  If you
are dealing with this bug, please assign it to yourself.

Comment 47

[David D. Kilzer (ddk)]

Attachment: Revised again for CVS tip

This patch renames the current "Bugs" column to "Total Bugs" and adds an "Open Bugs" column with a "Search" link that searches for the open bugs containing the keyword.  It seems to be a good compromise considering the discussion on the bug about gathering more statistics or calculating them in real-time.

The "Search" link is implemented via the suggestion from Comment #44.

If you'd like to try it out, see bugzilla.opendarwin.org which implemented it in a custom template file:

http://bugzilla.opendarwin.org/describekeywords.cgi
http://bugzilla.opendarwin.org/show_bug.cgi?id=7083

Comment 48

[Vlad Dascalu]

Comment on attachment 210761 [details] [diff] [review]
Revised again for CVS tip

+    <td align="center">

Not sure about the alignment; it looks good in the screenshot you provided. Sadly we don't have an UI owner right now in order to request an UI review.

-        <th align="left">[% terms.Bugs %]</th>
+        <th align="left">Open [% terms.Bugs %]</th>

This impacts localization. The template version should probably be increased at the top, but we haven't done that in a while, and localizers use diff nowadays anyway.

+          Search</a>
+      [% ELSE %]
+        none

In case you have 0 open bugs, you could display either "Search" or "none". That's because keyword.bugcount refers to the total count. So the UI is inconsistent.

However it partially makes sense, since if you see "1" for the total bug count, someone could "search" within that for open bugs; if you see "0" for the total bug count, it doesn't make much sense to "search" within that.

Comment 49

[David D. Kilzer (ddk)]

Attachment: Patch v2

You gave the previous patch (Attachment 210761 [details] [diff]) a review+, although the comments lead me to believe it should have been a review-, so I'm uploading a new patch.  Changes:

- Updated template version number from 1.0 to 1.1.
- Changed "none" to "Search" with no link when the Total Bugs column returns 0 for the bug count.
- I like "Search" centered, so I'm leaving it (although it would be better to use CSS rather than align="center", but I think that's probably a different bug).

Comment 50

[Vlad Dascalu]

Comment on attachment 212433 [details] [diff] [review]
Patch v2

The previous comments were minor issues, that's why I gave review+ on the previous patch.

I like the new version as well (it increments the template version). I'm not sure about the non-clickable "Search" word; that's mostly an UI issue, probably the one doing the approval flag will let us know which one to do.

Either patch is fine with me (personally, I'd take the UI from the first one and the versioning from this one :) )

Comment 51

[Vlad Dascalu]

Comment on attachment 210761 [details] [diff] [review]
Revised again for CVS tip

I'm obsoleting this one and obsolete the second one, since the template version is wrong. It should be 2.0 - the minor version should be incremented only when HTML is modified, but it doesn't affect translators, and the major version should be for l10n-breakable changed.

Comment 52

[Vlad Dascalu]

Comment on attachment 210761 [details] [diff] [review]
Revised again for CVS tip

But since all (nearly all) templates are still 1.0, it doesn't make much sense to increment to 2.0 anyway, so this is the one that should go.

Comment 53

[Colin Ogilvie [:cso]]

Checking in template/en/default/reports/keywords.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/keywords.html.tmpl,v  <--  keywords.html.tmpl
new revision: 1.7; previous revision: 1.6
done