543459 - The new duplicates.cgi is somewhat slow on large databases

Status: RESOLVED FIXED

(Bugzilla :: Bugzilla-General, defect)

Description

[Max Kanat-Alexander]

Just for the heck of it, I tested out the new duplicates.cgi on my local copy of the bmo database. It was so slow I couldn't get it to return, even after several minutes. It looks like the problem is actually in the Perl code, not in what we're doing in the database, so I'm going to do some profiling and figure things out.

I want bug 514970 to land first, though, so that my performance changes aren't going to conflict with that very large cleanup patch.

Comment 1

[Max Kanat-Alexander]

Okay, I did some testing, and actually what was happening on the bmo database was that there was a type of duplicate loop that the loop-detection code wasn't catching. I'm going to just add the fix to the patch in bug 514970.

It does seem that duplicates.cgi is still a bit slow, though, so I'm going to investigate what I can do to speed it up.

Comment 2

[Max Kanat-Alexander]

Okay, so the problem is the line:

my %since_dups = @{$dbh->selectcol_arrayref(

That is, populating since_dups takes up most of the time.

Comment 3

[Max Kanat-Alexander]

Attachment: v1

Okay, this pretty much fixes it! :-) Apparently using INTERVAL is much faster than using TO_DAYS.

I also noticed that massive calling of Bugzilla->params was adding a small bit of time to the profile in some places, and it was an easy fix, so I fixed it.

Comment 4

[Frédéric Buclin]

Comment on attachment 424609 [details] [diff] [review]
v1

I have a too small DB to test perf improvements. I will trust your testing. r=LpSolit

Comment 5

[Frédéric Buclin]

(In reply to comment #1)
> Okay, I did some testing, and actually what was happening on the bmo database
> was that there was a type of duplicate loop that the loop-detection code wasn't
> catching.

Shouldn't we have a separate patch for 3.4, with only this fix?

Comment 6

[Max Kanat-Alexander]

(In reply to comment #5)
> Shouldn't we have a separate patch for 3.4, with only this fix?

  Oh, no, add_indirect_dups is new for 3.6.

Comment 7

[Max Kanat-Alexander]

Committing to: bzr+ssh://[e-mail]/bugzilla/trunk/                                                                                                   
modified duplicates.cgi
modified Bugzilla/User.pm
Committed revision 6953.