All users were logged out of Bugzilla on October 13th, 2018

Bugzilla is slower than cold molasses

RESOLVED FIXED

Status

()

RESOLVED FIXED
16 years ago
7 years ago

People

(Reporter: junruh, Assigned: endico)

Tracking

({meta, perf})

Details

(Reporter)

Description

16 years ago
It takes 15 seconds just for http://bugzilla.mozilla.org/ to appear.

Comment 1

16 years ago
Right. AOL-TW should donate $1M for investments in faster servers and phatter pipes.

Updated

16 years ago
Keywords: perf
Yeah, apparently it appears that bugzilla is getting queries meant for google,
possibly via the sidebar, which is placing a massive load on bugzilla at the moment

-> myk
Assignee: justdave → endico
Component: Bugzilla-General → Bugzilla: Other moz.org Issues
Product: Bugzilla → mozilla.org
QA Contact: matty → myk
Version: unspecified → other
(Assignee)

Comment 3

16 years ago
I've been looking at b.m.o's http log files and it looks like we've had
this problem since last spring.

I searched for buglist.cgi queries containing the 'allwordssubstr'
parameter in log files for oct2, jul31, may1, mar6 and jan9 and
only the jan9 log didn't contain inappropriate looking queries.
(The jan 9 file had only 9 allwordssubstr hits). The oct 9
file seemed like nearly all the allwordsubstr queries were
meant for google. The march file seemed like it had a higher
ratio of valid stuff. A lot of the query strings were urls.
There was a wide variety of user agents and platforms.

Here's a small selection of search strings from the log covering
sep25-oct2. Since so many of these search strings seem to have
nothing to do with mozilla, we think that somehow bugzilla is
getting set as the default search engine for some people and
their normal queries are getting sent to bugzilla instead of
netscape.com, google or someplace sensible.


Harvard+school+of+Public+Heath+ "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.1) Gecko/20020826"
Healthy+Bites+Grill "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1)
Gecko/20020826"
Heart+of+Winter+Patches "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.0.0) Gecko/20020530"
Helga+M.+Novak+ "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1a)
Gecko/20020611"
Helga+M.+Novak++%22eis%22 "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.1a) Gecko/20020611"
Herv+Peterschmitt "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.0)
Gecko/20020530"
Hilton+Minneapolis "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2b) Gecko/20020926"
Hollidat+Inn+Inc "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1)
Gecko/20020826"
Holliday+Inn+Inc "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1)
Gecko/20020826"
How+To+Change+the+Windows+XP+Boot+Logo "Mozilla/5.0 (Windows; U; Windows NT 5.0;
en-US; rv:1.1b) Gecko/20020721"
Hulton "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.1) Gecko/20020826"
I-DEAS+troubleshooting "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2a)
Gecko/20020910"
IFF+Fighter+Follow+on+Training "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US;
rv:1.0.1) Gecko/20020826"
IFF+Fighter+Follow+on+Training%2C+F-16 "Mozilla/5.0 (Windows; U; Win 9x 4.90;
en-US; rv:1.0.1) Gecko/20020826"
divx+for+linux "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0.0)
Gecko/20020530"
divx+lunacy "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.1) Gecko/20020826"
divx+player "Mozilla/5.0 (Windows; U; Windows NT 5.0; it-IT; rv:1.1) Gecko/20020826"
divx+player+linux "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20020918"
dod+hq "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2a) Gecko/20020910"
dog "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.1) Gecko/20020826"
dog+collar "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1) Gecko/20020826"
dog+star "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US; rv:1.2a)
dom+refresh+ "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020913
Debian/1.1-1"don+quijote "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.1) Gecko/20020826"
direct+cable+connection+program "Mozilla/5.0 (Windows; U; Windows NT 5.0; it-IT;
rv:1.2a)
Gecko/20020910"
disassemble+bootsector+ "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.2a) Gecko/20020910"
discount+airline+boston+chicage "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US;
rv:1.0.0)discount+airline+boston+chicago "Mozilla/5.0 (Macintosh; U; PPC Mac OS
X; en-US; rv:1.0.0)disk+juggler+4+serial "Mozilla/5.0 (Windows; U; Win 9x 4.90;
en-US; rv:1.1) Gecko/20020826"
Case+studies+on+alberta+fisheries+and+waste+treatment+facility+effects+in+Swan+hills
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.0) Gecko/20020710"
Discovery+Wings+Channel "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.0.1)
Gecko/20020826"
Dk+Pub+Merchandise "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2a)
Gecko/20020910"  
Do+Androids+Dream+of+Electric+Sheep%3F "Mozilla/5.0 (Windows; U; Windows NT 5.0;
en-US; rv:1.2a) Gecko/20020910"
(Assignee)

Comment 4

16 years ago
note that if http://bugzilla.mozilla.org/ is no longer a static page
and if b.m.o is busy processing queries then it will take a long time
to load any page.

Comment 5

16 years ago
I believe the proper description is "Slower than maple syrup on a cold Vermont
morning".  Or something like that :-)
(Assignee)

Updated

16 years ago
Blocks: 179176
(Assignee)

Comment 6

16 years ago
The current theory is that the recent slowness is due to the fact that we're
now forcing newbies to use the bug helper, which also uses simple search and
therefore there are a lot more simple searches going on than before.

On the other hand, i looked at allwordssubstr queries for today and there
still seems to be an abundance of inappropriate queries. If the bugzilla
helper theory were true then it seems like we'd be flooded with lots more
appropriate stuff. But my measurement of appropriate to inappropriate 
queries was hardly scientific, so so maybe we really are getting more
appropriate stuff.

One thing to note is that only about half of the allwordssubstr queries
have more than one word (as measured by grepping for +).

I'm wondering if this is happening because of aggregation people appear to be
doing - if you have bugzilla and gooole selected, you'd get both, but may not
notice if buzilla didn't return results.

Even if bugzilla did return results, however, you still wouldnt' notice, because
the bugzilla search plugin's parsing hooks weren't updated to deal with the
templaitsed output for buglist.cgi, which changes last time bmo upgraded,
several months ago. So the user may jsut select everything on teh grounds that
its more likly to find a match, and then would jsut see the XUL page of results,
and so not notice a problem, since there weren't any bugzilla entries returning
false positives. Does that sound like a plausable scenario?

The fact that noone has reported bugzilla's serach plugin being broken in the
first place a tany point in the past 5-6 months may be a good reason to drop
that plugin - its obviously not being used as a data source, and we now have the
bugzilla sidebar for people who were using it just as a plain query textfield

Also, the bugzilla.src file has an update url, but the file that url points to
has a really wierd query. Does mozilla downlaod these updates, ever?
The reason we're noticing this now is possibly because mysql sucks at optimising
OR queries, and product OR component thing now adds an extra two joins to the
query which weren't present before the upgrade. Oh, for a subselect ;)

I also wonder if this is the cause of the 'select everything' issue which used
to be present on bmo, which has been fixed by bugzilla forbidding queries with
no search terms.
(Assignee)

Comment 9

16 years ago
Given the fact that we have been getting google queries since last spring,
it seems like that's not what is our immediate problem today. (although that
should still be investigated). I'm blaming today's major slowdown on 
something that changed with the upgrade. Moving this back to the bugzilla
product. This might be the query problem that bbaetz mentioned or a problem
with doing so many text searches now that we're doing more bugzilla helpers.
Component: Bugzilla: Other moz.org Issues → Query/Bug List
Product: mozilla.org → Bugzilla
Version: other → 2.17.1
Yeah, I'm blaming the fact that we're getting these queries on mozilla and the
slowdown on the upgrade due to the table joins + mysql's lack of optimisation
for OR.

Can someone take one of these silly queries, and attach the SQL + the mysql
EXPLAIN output, to confirm that? Both with and without the product/component,
please.

If theres some way of determining that a request came from the search plugin,
then we cna just ignore those requests. sgehani?
Actually, I don't think that this is bugzilla helper. We only get 100-200 bugs a
day, and some of those used to search anyway. I'm guessing that the largest
issue is from quicksearch.

If we change the product/component search to 'equals' rather than 'contains' in
localconfig.js, do we get any better? That own't affect sidebar searches, but
those don't appear to have changed recently.
Well, QuickSearch has been taken off the front page. Where are people running it
from? Or has the performance improved since we did that?


Gerv
Its still runnable from bugzilla helper - see bug 179608. Altough the user just
gets an error in that case.

Its also still runnable from the sidebar, although thats what the error (which
probably won't be seen by the user) is meant to catch. Also, the sidebar doesn't
use quickserach, it runs buglist.cgi automatically. See
http://lxr.mozilla.org/seamonkey/source/xpfe/components/search/datasets/bugzilla.src#13

As this search plugin is broken (see comment 7), and has been broken since May,
would anyone object ot me filing a Bugzilla bug to remove it? Which we can then
get on the 1.2 branch before 1.2 is released, to hopefully stop as many people
as possible?

If we can work out what happens with updates, we could also just push an update
to the m.o site which effectively disabled the plugin by pointing the url to a
dummy .html page. That would catch people who didn't update their mozilla build,
 too.
Another possible solution is that according to
http://www.mozilla.org/projects/search/technical.html multi-site searches have
the http header 'MultiSite: true' (I tested that, and the documentation is
correct! ;). So we could block the sidebar searches via that, either by testing
in buglist.cgi or via a RewriteCond RewriteRule in bmo's .htaccess file. The
latter is probably faster, since you wouldn't have to spend time loading all the
Template modules.

Roughly how often are these requests coming, anyway?

Thats all separate to changing the product/component to be an exact match,
though, assuming that that suggestion does produce a speedup.
... and you may want to block lxr and the 'mozilla.org' search engine from
multisite searches, too, in the same way.

endico: Are you seeing wierd queries for those two engines, too? If so, I'll
file an m.o bug on rejecting those in that .htaccess too.

Before we do that, though, it would be nice to have some confirmation taht this
is whats causing all the searches. Is there some way that you could easily
change apache's logging stuff to print that 'MultiSite' header to the log, or
just have buglist output to a file if thats the case, or something along those
lines?
and for the lsta update before I got to bed:

The reason taht we started seeing these in Februray was because  bug 83052 was
checked in on 2002-02-14 16:58.

That bug moved the Bugzilla.src file (but not the one stored on m.o) from doing
a 'search by assignee' search to doing the current OR'd query, which was based
on what the quicksearch code did. Search by assignee is cheap, though, so its
unlikly that we sould ahve noticed unless someon ewas watching the logs on purpose.

I'm guessing that if dawn did her access_log search on assignees, w'ed see the
same destined-for-google responses appearing before February.

Anywa, to summarise, I think there are three things to do:

- Drop |MultiSearch: true| queries, preferably at the apache level.
  This is a bmo-specific bug

- Change/drop the various search plugins, and preferably get that into the 1.2
branch
  - Update the copy on m.o so that existing clients get updated.
  This is a browser (and m.o, for the second part) bug

- Drop the product/component stuff from QS
  - have the bugzilla helper version of QS restrict to the specific product,
probably by prepending |product:foo| before using quicksearch.js
  This is a bugzilla product bug.

Re the third point, I think the reason that this was done was so that people
could search for 'ftp login failure' and get ftp bugs. But this will in fact
give back _all_ ftp bugs, because its an OR. Searching for |network login
failure| will return all ftp/http/smtp/pop/etc bugs, because those components
contain the string 'network'. Searching for 'browser locks up' will return all
browser bugs.

quicksearch.js has a refernce to a blacklist of search strings (defined in
localconfig.js) which shouldn't be considered as a substring to patch for
products; I don't think that that is sufficient. afranke, you originally wrote
this, what was the reason for including this?

Oh, theres one more bug in the sidebar. For multiword searches, (say, |foo
bar|), quicksearch does ('product' 'subtring' 'foo' OR 'component' 'substring'
'foo' OR ...) AND ('product' 'substring' 'bar' OR 'component' 'substring' 'bar').

the sidebar does ('product' 'allwordsubstr' 'foo bar') OR ('component'
'alwordsubstr' 'foo bar'). This is silly, since its asking for all bugs with a
product containing 'foo' and 'bar' AND a component containting 'foo' and 'bar'.

This only does return any results because bugzilla has a bug - it brackets this
incorrectly, and so its expanded to the equivalent of ('product 'substring'
'foo' AND 'product' 'substr' 'bar' OR 'component' 'substring' 'foo' AND
'component' 'substring' 'bar' AND ...). I don't blame mysql for not managing to
pull out the joins ;) Since it can't, this becomes a query over every record,
adn wince the bmo upgrade, there are the two additional joins. I've filed this
as bug 179697, but although fixing that may mitigate the efffect of all these
searches, which seem mostly multiword from dawn's list, its not really related
to this bug
>the sidebar does ('product' 'allwordsubstr' 'foo bar') OR ('component'
>'alwordsubstr' 'foo bar'). This is silly, since its asking for all bugs with a
>product containing 'foo' and 'bar' AND a component containting 'foo' and 'bar'.

s/AND/or/
s/and/AND/

is what I meant to write...
rjc, comments on the search stuff above? (Specifically, comment #7)

Updated

16 years ago
Depends on: 179827
No longer depends on: 179827
This is m.o specific, since its only an issue because bugzilla.mozilla.org is in
the search sidebar.

I'll go file bugs now to make this bug depend on this - see comment 16
Component: Query/Bug List → Bugzilla: Other moz.org Issues
Depends on: 179827
Keywords: meta
Product: Bugzilla → mozilla.org
Version: 2.17.1 → other

Updated

16 years ago
Depends on: 180866
> Re the third point, I think the reason that this was done was so that people
> could search for 'ftp login failure' and get ftp bugs. But this will in fact
> give back _all_ ftp bugs, because its an OR. Searching for |network login
> failure| will return all ftp/http/smtp/pop/etc bugs, because those components
> contain the string 'network'. Searching for 'browser locks up' will return all
> browser bugs.

I think this is a misunderstanding. Searching for "ftp login failure" will
return just a single bug, since all of "ftp", "login" and "failure" must occur
in some attribute of the bug. Same for the other two examples.

> quicksearch.js has a refernce to a blacklist of search strings (defined in
> localconfig.js) which shouldn't be considered as a substring to patch for
> products; I don't think that that is sufficient. afranke, you originally wrote
> this, what was the reason for including this?

If you search for "table row", then you don't want all "Browser" bugs with
"table" in the summary, so triggering the "Browser" product with its substring
"row" is an extreme case where the pure quicksearch method does not work. That's
why there is the blacklist to disable this substring trigger for products and
components for some substrings that are likely to be used in other contexts.
Another example is the word "new" which should not trigger the Product
"MailNews". But these are only very few exceptions, and they can be added to the
blacklist case by case whenever some substring relation is found to be
inappropriate and annoying.
afranke: thats correct, but its the 'some attribute' part which is why we search
product and component as well as just the summary.
Bug 179960, the main issue from the upgrade, has been fixed, and b.m.o is now
back to its previous level of performance.  Removing dependency on the
regressions meta-bug.
No longer blocks: 179176
AFAIK Bugzilla have been since the last post .. This bug needs reevaluating.
Bugzilla sure don't seem "slower than cold molasses" to me at least.

Though that's not saying it couldn't be made faster :)

Implementing Bug 247853 would speed up the download of the pages 300% - 400% and
also save a lot of bandwidth.

Bug 179827 and 180866 also seem to have been fixed although they have not been
marked as such.
The first line should have read "AFAIK Bugzilla have been UPGRADED since the
last post"

With the word upgraded in this makes so much more sense .. sorry for the small
tyop :)
Bugzilla is certainly no longer slower than cold molasses, since it's moved to
"mecha".

Is this bug still useful?

Gerv
(In reply to comment #25)
> Bugzilla is certainly no longer slower than cold molasses, since it's moved to
> "mecha".
> 
> Is this bug still useful?

No.  The specific issues mentioned in this bug have been resolved a long time ago.

Bugzilla still has a few performance issues, but they're covered on other bugs
(like bug 75488)
Status: NEW → RESOLVED
Last Resolved: 14 years ago
Resolution: --- → FIXED
Component: Bugzilla: Other b.m.o Issues → General
Product: mozilla.org → bugzilla.mozilla.org
You need to log in before you can comment on or make changes to this bug.