851267 - Bugzilla times out when a user has several thousands of votes

Status: RESOLVED FIXED

(Bugzilla :: Extensions, defect)

Description

[Carlos Alén Silva]

Ever since Bugzilla was upgraded to version 4.2 I cannot longer vote/unvote for bugs. I always get a "Service unavailable" error when trying to vote. This happens when I log in with Firefox. When I tried Internet Explorer, Google Chrome and Opera, I just could not even log-in and got "Service unavailable" instead of being logged-in. I didn't use Persona to log in.

STR:
1. Log in to bugzilla.mozilla.org.
2. Choose a random bug and try to vote for it.
3. Get a "Service unavailable" error.

Reproducible: YES.

Actual results:
I cannot vote/unvote for bugs in Bugzilla. Always get a "Service Unavailable" error.

Expected results:
I should be able to vote normally.

Comment 1

[David Lawrence [:dkl]]

I was able to vote for this bug several times which makes me think this is an intermittent issue. We have had some other issues with specific webheads and access but I thought we had those resolved. And those were related to Persona (which you say you are not using) which is different as the voting code does not need access outside of Bugzilla itself. 

Will need to poke around a bit.

dkl

Comment 2

[Frédéric Buclin]

This is just another symptom of the same root cause. Voting on a bug is not more special than commenting into a bug. So it's a dupe.

Comment 3

[Carlos Alén Silva]

(In reply to Frédéric Buclin from comment #2)

Fair enough, but I find it somewhat odd that I can create bugs and post comments but I haven't been able to vote/unvote for any bug since 10th March no matter how many times I've tried. This is not intermittent for me, it's constant and doesn't affect my comment capability.

Comment 4

[Tony Mechelynck [:tonymec]]

Mozilla/5.0 (X11; Linux x86_64; rv:22.0) Gecko/20100101 Firefox/22.0 SeaMonkey/2.19a1 ID:20130327003001 c-c:1f5700c1bd18 m-c:178a4a770bb1

Works for me on Linux64 with the current BMO bz 4.2.5+

Comment 5

[Carlos Alén Silva]

FWIW:

I cannot vote/unvote bugs. However, a friend of mine logged in from my computer just a minute after I tried for the nth time and she was able to vote/unvote as many times as she wanted. Could this be a problem with my account? I have thousands of votes, if that is relevant.

Comment 6

[Tony Mechelynck [:tonymec]]

(In reply to Carlos Alén Silva from comment #5)
> FWIW:
> 
> I cannot vote/unvote bugs. However, a friend of mine logged in from my
> computer just a minute after I tried for the nth time and she was able to
> vote/unvote as many times as she wanted. Could this be a problem with my
> account? I have thousands of votes, if that is relevant.

Hm, bug 849607 is RESOLVED FIXED yet you still see this problem? Then it must be something different. Can you access https://bugzilla.mozilla.org/page.cgi?id=voting%2Fuser.html (while being logged in to Bugzilla)? Please describe exactly what happens when you do, and while you're at it, check how many votes you have with the maximum-per-Product. (The maximum can be exceeded if a bug for which you have voted gets moved to that Product; but then Bugzilla won't let you click [ Change my Votes ] unless it brings the count lower than the maximum for each concerned Product.)

Also, please note the exact date and time (with timezone) the next time you see the bug.

Comment 7

[Carlos Alén Silva]

Hello, there. Unfortunately, I cannot access https://bugzilla.mozilla.org/page.cgi?id=voting/user.html while I'm logged into Bugzilla. I get the same message as when I try to vote/unvote for any bug:

"Service Unavailable

The service is temporarily unavailable. Please try again later."

Actually, I had tried just that before submitting my previous comment.

I don't remember exactly, but I know that I had reached my vote limit for some of the lower-limit products/components (25 votes); then, I had 1000 votes for the Firefox product and 10000 for 'Core'.

The last time I saw the bug was between 23:16:56 to 23:17:00 UTC+01:00 (Brussels, Copenhagen, Madrid, Paris) on March 30 2013 which should be 15:16:56 to 15:17:00 PST on March 30 2013.

If you need more information, I'll be glad to help you as best as I can.

Comment 8

[Tony Mechelynck [:tonymec]]

You mean you had voted for ten thousand different Core bugs? This surely is a lot. Or do you mean *the maximums* are 1000 for Firefox and 10000 for Core (they are) but you don't think you had reached them?

Maybe you will need help from some Bugzilla admin to help you sort your problems out. (I'm just a user.)

Comment 9

[Carlos Alén Silva]

I voted for 1000 Firefox bugs and 10000 Core bugs. So far, I hadn't had any problem because of that. If I had reached a maximum of votes and wanted to vote again, Bugzilla just wouldn't let me cast a new vote until I unvoted another bug first.

Comment 10

[Frédéric Buclin]

Bugzilla has so many votes to fetch and display that Apache probably times out before it got any data to return.

Comment 11

[Frédéric Buclin]

I was right in comment 10: it doesn't take too long for MySQL to collect the data, but then it takes a huge time for Template Toolkit to generate the HTML page, and Apache gives up before it got the data (TT is unable to flush data).

It's very uncommon to see a user with so many votes, and we should limit the number of bugs to list. I did some testing, and 5000 bugs seems a reasonable limit. The first step is to limit the list to votes in the given product only (the same product as the bug the user is editing his votes), and if it still exceeds the 5000 bugs limit, then split the list again.

Comment 12

[Frédéric Buclin]

Attachment: patch, v1

I did some profiling and there are major bottlenecks in the Voting code. I think I fixed them all in my patch:

- One major problem was that we called Bugzilla::Bug->new() and ->check on each bug, one by one. This is terribly slow. Now I first call visible_bugs() to first cache security data, and then call ->new_from_list() which is way faster than looping over each bug.

- Another major bottleneck is | bug.id FILTER bug_link(bug.id) | which calls Bugzilla::Template::get_bug_link() which again calls bug/link.html.tmpl. The time penalty is *huge*. Now we call this template directly.

- Instead of deleting *all* user data from the votes table and then reinserting them again, I now only update rows which need to be updated. It's very rare that a user edits 500 votes at once. Most of the time, he edits only a few at once. The win is also big as it severely decreases the number of calls to the DB.

- I also optimized the code by (re)using bug objects as much as possible instead of querying the DB again and again or to recreate bug objects in each subroutine, especially in _confirm_if_vote_confirmed().

- I replaced the useless FILTER html by FILTER none as bug IDs come from the DB and so are safe.

- I realized that CGI->self_url is terribly slow when the URL has many parameters. The culprit is CGI->query_string which eats a lot of time (0.3 second in my testing with 900 votes). So I store it in Bugzilla::CGI->redirect_search_url as a minor improvement. Also, each call to $cgi->header is slow because it calls url_is_attachment_base() for security reasons which itself calls self_url(), and so we get the same time penalty. url_is_attachment_base() doesn't need nor use parameters passed by CGI and so it now calls url() instead of self_url(). Due to https://rt.cpan.org/Public/Bug/Display.html?id=87790, there is currently no time win because CGI->url calls CGI->query_string even when unused. But I already applied the patch reported upstream and the win is very visible. I hope it will be included in CGI 3.64.


Now some data. I have votes on 887 bugs and doing nothing but simply clicking "Change my votes" gives the following results:

without my patch:

Profile of page.cgi for 6.11s (of 7.98s), executing 1690724 statements and 605727 subroutine calls


with my patch:

Profile of page.cgi for 2.44s (of 3.29s), executing 763938 statements and 308101 subroutine calls

As you can see, numbers reported by Devel::NYTProf are divided by 2 with my patch. With Devel::NYTProf disabled, page.cgi now takes 0.9 second to load instead of 3.0 seconds. That's a huge win.

Comment 13

[Simon Green]

Comment on attachment 788954 [details] [diff] [review]
patch, v1

After speaking to LpSolit, I've asked dkl to do the review since he is more experienced with the Voting system.

Comment 14

[David Lawrence [:dkl]]

Comment on attachment 788954 [details] [diff] [review]
patch, v1

Review of attachment 788954 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good and works as expected. 25-30% decrease in load time also when when applied to BMO code base and looking at a user with a large number of votes. r=dkl

Comment 15

[Frédéric Buclin]

Committing to: bzr+ssh://[e-mail]/bugzilla/trunk/
modified Bugzilla/CGI.pm
modified extensions/Voting/Extension.pm
modified extensions/Voting/template/en/default/pages/voting/user.html.tmpl
Committed revision 8763.

Committing to: bzr+ssh://[e-mail]/bugzilla/4.4/
modified Bugzilla/CGI.pm
modified extensions/Voting/Extension.pm
modified extensions/Voting/template/en/default/pages/voting/user.html.tmpl
Committed revision 8615.

Comment 17

[Frédéric Buclin]

(In reply to Frédéric Buclin from comment #12)
> Due to https://rt.cpan.org/Public/Bug/Display.html?id=87790, there is currently no
> time win because CGI->url calls CGI->query_string even when unused.

This problem has been fixed in CGI 4.05.