Closed Bug 25693 Opened 25 years ago Closed 24 years ago

request for a table for duplicates

Categories

(Bugzilla :: Bugzilla-General, enhancement, P3)

Product:
Bugzilla
Component:
Bugzilla-General
Platform:
x86
Windows NT
Type:
enhancement

Tracking

()

Status:
VERIFIED FIXED
Milestone:
Bugzilla 2.12

People

(Reporter: cbegle, Assigned: Chris.Yeh)

References

Details

Attachments

(3 files)

ZIP of patch
5.37 KB, application/zip
Details
Updated patch; three files as a tar.
20.00 KB, application/octet-stream
Details
bugfix patch
1.16 KB, patch
Details | Diff | Splinter Review 
It would be nice if we could query for bugs which had the most duplicates to
make it easy to generate the "Most Frequently Filed bugs".

I think a table that ahd the bug number and a tally for the # of bugs that are 
duplicates of that bug would be fine (ie not have to manage a list of bugs that 
are duplicates of the bug)... If we want to see details fo the duplicates, that 
info is stored in the comment.
Doing some kind of half-assed tables to create duplicates will only cause more
problems and confusion.

I completely agree that the way duplicates are handled by Bugzilla is woefully
inadequate, and that reports like this should at least be feasable.
Status: NEW → ASSIGNED
Summary: request for a table for duplicates → request for a table for duplicates
Bug 26053, "[RFE] Ability to limit search to bugs with DUPs", if implemented, 
may provide the basis for a simple, not quite perfect, but potentially
good enough solution for this problem.  If a field were added to each record
that simply counted how many bugs had been made a DUP of a given bug as
they come in, it would allow not only searching for bugs with DUPs, but,
with the addition of a column for the bug list ( "#DUP"?), ordering the list
by number of duplicates. 

There would be at least two inaccuracies. Sometimes bugs are marked as DUPs
incorrectly (mistaken resolution or typo), which would inflate the number
recorded over reality. Also, sometimes trees of DUPs form, with some of the
bugs marked as DUPs of the bug that will get fixed having DUPs themselves,
which would cause under-recording of the true number of DUPs in all.

But for the purposes of finding the most frequently reported bugs, these
inaccuracies may not matter greatly.

Any solution that would properly handle those cases would probably require
a complete revamp of the way DUPs are handled, and would probably be a real
pain to get to work for the existing records.

*** Bug 26053 has been marked as a duplicate of this bug. ***
I'm actually not scared of the work to handle existing records.  All I have to
do is write some code that searches for bugs that are closed with a resolution
of DUPLICATE, and find the last chunk of text in such bugs that says "*** This
bug has been marked as a duplicate of <something> ***".  Given that information,
it can go populate the database properly.

I'm just not sure how to represent this all in the database.  I suppose I could
just add two fields: one is the bug number that a given bug is a duplicate of,
and one is the count you propose of how many duplicates a bug has.  Given those,
I can do reasonable sanity checking and stuff, and aleviate the problems of
typos and bugs that are REOPENED.

Hmm.
You are right that it would be easier to start from the "bug that is a DUP"
end than the "bug that has DUPs" end, and doing so would provide more 
potential benefits. FWIW, bug 26053 started out as a proposal for a 
quick-n-dirty way of identifying the bugs that have DUPs, but I'd rather 
see a solution that addresses more problems than that.

How much additional work would it be to flatten those trees of DUPs for
the purposes of getting a better count? Could this count be done without 
actually changing which bug a leaf bug is a DUP of from a branch bug to the 
root bug? Or should those bugs be changed to match the bug that the branch 
bug is made a DUP as that is done? And should this be a new bug report?

I think it would be possible to maintain the count as being the size of the
total tree, not just the number of direct duplicates.

I don't see anything worthy of another bug report...
To get the other half you'd need to have a "at most x bugs" capability (no bug
report that I'm aware of), as well as the ability to specify the columns in the
URL (bug #12284).

Instead, if you want to order this within product categories, then you're
looking at general summary reports (bug #12282).
tara@tequilarista.org is the new owner of Bugzilla and Bonsai.  (For details,
see my posting in netscape.public.mozilla.webtools,
news://news.mozilla.org/38F5D90D.F40E8C1A%40geocast.com .)
Assignee: terry → tara
Status: ASSIGNED → NEW
is bug 38850 a dup or a blocker of this bug?
Bug 38850 is one possible solution to the problem discussed here. It recommends 
simply storing the bug number a bug is a dupe of, as a field.

Then, all the intelligence could be put in the script which walks the database 
to generate the most-frequent-bugs list. 

In outline, it would search the entire thing, keeping a running total:

Bug 34567   3
Bug 34590   6
...

Then, it would go through every bug on that list and, if it itself were a dupe, 
move the number, i.e. if 34590 were marked a dupe of 34567, the above would 
become:

Bug 34567   9
...

At the end of this procedure, you'd have the info you need - which bugs were 
most duped, and how many times, even indirectly. And it hardly requires any 
Bugzilla changes AFAICS.

Gerv
Tara: What are the chances of this happening any time soon? If none, I'll go 
away and do most-frequent-bugs another way...

Gerv
*** Bug 38850 has been marked as a duplicate of this bug. ***
I'm about to attach a zip of a duplicates.cgi (the most-frequent-bugs list) and 
a diff of changes to a reasonably recent CVS version which implements 
much-improved duplicate handling in Bugzilla. 

It would be very cool if this patch could be tested on Landfill or evaluated by 
other people, for as-soon-as-possible addition to bugzilla.mozilla.org (as it 
would make my life, as maintainer of most-frequent-bugs at the moment) a lot 
easier :-)

Gerv
Keywords: patch
Attached file ZIP of patch — 

    
    

    
  
Chris--can you get this on landfill, and at some point could you give me access
to landfill?

:)
Assignee: tara → cyeh

    
    

    
  

*** This bug has been marked as a duplicate of 38850 ***
Status: NEW → RESOLVED
Closed: 24 years ago
Resolution: --- → DUPLICATE

    
    

    
  
And verifying bud as true duplicate
Status: RESOLVED → VERIFIED

    
    

    
  
Since when do you get to verify your own change?  I thought that was the point of 
verifying was to get a second opinion.  And I disagree with this being marked a 
duplicate of bug 38850.  And according to your comments on 38850, so do you, so 
I'm confused as to why you did this...


    
    

    
  
*** Bug 38850 has been marked as a duplicate of this bug. ***

    
    

    
  
Based on this statement from bug 38850:

>I believe the duplicate should be marked the other way round. Bug 25693 has
>more information than this bug. I'll try to mark 25693 as the initial bug and
>this one (38850) as the duplicate

And the fact that I agree with the reasoning behind this statement, I am changing 
it back to match, and making this the original, and 38850 the duplicate.  
Reopening.

Status: VERIFIED → REOPENED
Resolution: DUPLICATE → ---

    
    

    
  
If anyone disagrees, please discuss before switching it back.  As someone who 
stands a good chance of helping with development in making this change, I want 
this bug open because this is the one that has the most information in it.  If 
you have a valid reason to close this in favor of a different duplicate, speak 
up, and I might be convinced, but otherwise please leave this one open.

Thanks, and apologies to everyone else reading this for the spam.

Status: REOPENED → ASSIGNED

    
    

    
  
This is all very well, but is there any chance anyone is going to get around to 
evaluating and checking in my PATCH for this problem? Then we can close both 
bugs ;-)

Gerv

    
    

    
  
Yeah, I left that out, this bug has a patch on it, too, which is a definite plus 
for keeping this one open. :)

I believe the hold up at the moment is waiting for the configuration changes on 
landfill so some others of us besides Chris can apply patches to landfill.  
Otherwise we're waiting on Chris for anything that needs trial time on landfill.

(Chris?)


    
    

    
  
landfill has been both moved and reconfigured to allow for outside people 
access. working with Gerv on getting the patch in. It's a large enough change 
that we'll want to have people hammer it and give it some bake time.



    
    

    
  
trying to install your patch onto landfill, and i ran into two things:
1) $regenerateshadow = 1 should be $::regenerateshadow = 1;
2) collectstats.pl makes calls to dbmopen and dbmclose, which barfs the script 
with the following error:
>Can't locate NDBM_File.pm in @INC (@INC contains: 
/usr/local/lib/perl5/5.6.0/i686-linux /usr/local/lib/perl5/5.6.0 
/usr/local/lib/perl5/site_perl/5.6.0/i686-linux 
/usr/local/lib/perl5/site_perl/5.6.0 /usr/local/lib/perl5/site_perl .) at (eval 
7) line 3.

Is this a perl module that I need to get from CPAN, or is my perl installation 
screwed up?



    
    

    
  
yes.

    
    

    
  
fixed it. a use DB_File; line was missing from the patch

    
    

    
  
okay, i got this working. I also had to add a use DB_File; to duplicates.cgi, 
and to fix up a typo in collectstats.pl where you wrote the file without a .db 
extension, yet were expecting to read in a file that had it. i added .db to the 
output file and it works.

you did test this before you gave me the patch, right? :)

it's up and running on landfill. please pound it and see if it behaves the way 
you expect it to. if you generate a lot of duplicates i'll have to re-run 
collectstats.pl by hand to get those to show up in a quick fashion.

    
    

    
  
> you did test this before you gave me the patch, right? :)

Er, yeah - but quite a while ago now...

I'll pound on it on landfill. Did the import work correctly?

Gerv

    
    

    
  
import? what am i importing where?

    
    

    
  
When you update (run checksetup.pl) it should have automatically picked up any 
dupes in the database. But there weren't any, I now discover ;-) So that bit 
will need testing in some way (I did test it, but only on a small scale.)

Gerv

    
    

    
  
How's it importing the existing duplicates?  If it's going by the "this bug has 
been marked a duplicate of xxxxx" in the text, you might want to ask for manual 
intervention if loops are detected (or are you already doing that? I haven't 
looked it over real good yet).  This bug right here is a real good example of 
one that'll trip it up, since it got closed as a duplicate of another bug that 
was already a dupe of this one, and then reversed again.

    
    

    
  
It detects the text on bugs currently resolved DUPLICATE, and adds the two bugs 
in a relationship in a hash - this means that only the last dupe marking counts, 
as earlier ones get overwritten. It doesn't need to do loop detection - as 
there's no "right answer" in that case, it just does "something". I remember 
checking the code against loops; I can't remember how it decides what to do, but 
it turns out it doesn't need to do anything special.

Gerv

    
    

    
  
*** Bug 38857 has been marked as a duplicate of this bug. ***

    
    

    
  
adding myself and endico to this bug.


    
  
Blocks: 6999

    
    

    
  
Checked in Gervase' patch.
Status: ASSIGNED → RESOLVED
Closed: 24 years ago → 24 years ago
Resolution: --- → FIXED

    
  
Blocks: DupeLoop

    
    

    
  
In search of accurate queries....  (sorry for the spam)

Target Milestone: --- → Bugzilla 2.12

    
    

    
  
reopening... the duplicate scan failed miserably on bugzilla.mozilla.org's 
database when Dawn tried to do a test update on a copy of the database.  The 
regexp that tests for the duplicate string in the bug text is apparently not 
strict enough, and is catching invalid dupes.  Bug 26913 is an example of a bug 
that tripped it up.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

    
    

    
  
DBD::mysql::db do failed: You have an error in your SQL syntax near 't feel like 
generating more spam just to be picky, making VERIFIED.
', '26913')' at line 1 at ./checksetup.pl line 2048.


    
    

    
  
The attached patch worked on my database, and also worked correctly on b.m.o's 
data.


    
    

    
  
at least, as correctly as it could. The highest numbered bug in my test db
(a backup from this morning) is 70984. I searched for dups and dups of, and
found quite a lot.  


mysql> select * from duplicates where dupe_of > 70984;
+---------+-------+
| dupe_of | dupe  |
+---------+-------+
|   71024 | 71040 |
|   71042 | 71043 |
|   71024 | 71061 |
|   71024 | 71072 |
|   71082 | 71083 |  all of these except the last entry have both the 
|   71024 | 71084 |  dup and dup_of field bogus. i can understand one field
|   71096 | 71104 |  being bad, but both?
|   71109 | 71111 |
|   71024 | 71120 |
|   91026 | 21027 |  dave fixed this bug
+---------+-------+
10 rows in set (0.01 sec)

mysql> select * from duplicates where dupe > 70984;
+---------+-------+
| dupe_of | dupe  |
+---------+-------+
|   64100 | 70985 |
|   68336 | 70986 |
|   70773 | 70997 |
|   50758 | 71001 |
|   70756 | 71007 |
|   67574 | 71017 |
|   70773 | 71039 |
|   71024 | 71040 |
|   71042 | 71043 |
|   71024 | 71061 |
|   71024 | 71072 |
|   49141 | 71073 |
|   71082 | 71083 |
|   71024 | 71084 |
|   70361 | 71092 |
|   70057 | 71093 |
|   60151 | 71095 |
|   70924 | 71103 |
|   71096 | 71104 |
|   59655 | 71109 |
|   71109 | 71111 |
|   71024 | 71120 |
|   43847 | 71149 |
+---------+-------+
23 rows in set (0.03 sec)


    
    

    
  
those are all accurate.

select max(bug_id) from bugs

instead of 

select count(*) from bugs    :)


    
    

    
  
oops, never mind. I computed the max bug number incorrectly. There was only
one incorrect value and Dave just fixed the bug. Someone had marked a bug
a dup of an invalid bug number.

    
    

    
  
Since this worked on b.m.o and on mine, I'm thinking to just go ahead and check 
it in, but since Gerv knows this code better, I'll wait for his say-so.


    
    

    
  
Oh, come on, how was I supposed to know people would do pathologically evil 
things like bug 26913? ;-)

Looks good to me. Sorry I haven't got to this earlier; I've been, er, asleep :-)

Gerv

    
    

    
  
OK, it's checked in.

Status: REOPENED → RESOLVED
Closed: 24 years ago → 24 years ago
Resolution: --- → FIXED

    
    

    
  
When printing bug summaries, the summary text should first quote html
meta-characters.

The summary of bug 39992 begins with <SELECT> and a little dropdown
box is displayed instead of the summary.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

    
    

    
  
Jake's patch from bug 22041 fixed this bug too. marking fixed.
Status: REOPENED → RESOLVED
Closed: 24 years ago → 24 years ago
Resolution: --- → FIXED

    
    

    
  
Verified fixed. We have a table for duplicates now :-)
Status: RESOLVED → VERIFIED

    
    

    
  
Moving closed bugs to Bugzilla product
Component: Bugzilla → Bugzilla-General
Product: Webtools → Bugzilla
QA Contact: matty
Version: other → unspecified
QA Contact: matty_is_a_geek → default-qa

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: