179500 - Bugzilla is slower than cold molasses

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, defect)

Description

[John Unruh]

It takes 15 seconds just for http://bugzilla.mozilla.org/ to appear.

Comment 1

[Niklas Dougherty]

Right. AOL-TW should donate $1M for investments in faster servers and phatter pipes.

Comment 2

[Bradley Baetz (:bbaetz)]

Yeah, apparently it appears that bugzilla is getting queries meant for google,
possibly via the sidebar, which is placing a massive load on bugzilla at the moment

-> myk

Comment 3

[Dawn Endico]

I've been looking at b.m.o's http log files and it looks like we've had
this problem since last spring.

I searched for buglist.cgi queries containing the 'allwordssubstr'
parameter in log files for oct2, jul31, may1, mar6 and jan9 and
only the jan9 log didn't contain inappropriate looking queries.
(The jan 9 file had only 9 allwordssubstr hits). The oct 9
file seemed like nearly all the allwordsubstr queries were
meant for google. The march file seemed like it had a higher
ratio of valid stuff. A lot of the query strings were urls.
There was a wide variety of user agents and platforms.

Here's a small selection of search strings from the log covering
sep25-oct2. Since so many of these search strings seem to have
nothing to do with mozilla, we think that somehow bugzilla is
getting set as the default search engine for some people and
their normal queries are getting sent to bugzilla instead of
netscape.com, google or someplace sensible.


Harvard+school+of+Public+Heath+ "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.1) Gecko/20020826"
Healthy+Bites+Grill "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1)
Gecko/20020826"
Heart+of+Winter+Patches "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.0.0) Gecko/20020530"
Helga+M.+Novak+ "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1a)
Gecko/20020611"
Helga+M.+Novak++%22eis%22 "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.1a) Gecko/20020611"
Herv+Peterschmitt "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.0)
Gecko/20020530"
Hilton+Minneapolis "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2b) Gecko/20020926"
Hollidat+Inn+Inc "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1)
Gecko/20020826"
Holliday+Inn+Inc "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1)
Gecko/20020826"
How+To+Change+the+Windows+XP+Boot+Logo "Mozilla/5.0 (Windows; U; Windows NT 5.0;
en-US; rv:1.1b) Gecko/20020721"
Hulton "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.1) Gecko/20020826"
I-DEAS+troubleshooting "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2a)
Gecko/20020910"
IFF+Fighter+Follow+on+Training "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US;
rv:1.0.1) Gecko/20020826"
IFF+Fighter+Follow+on+Training%2C+F-16 "Mozilla/5.0 (Windows; U; Win 9x 4.90;
en-US; rv:1.0.1) Gecko/20020826"
divx+for+linux "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0.0)
Gecko/20020530"
divx+lunacy "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.1) Gecko/20020826"
divx+player "Mozilla/5.0 (Windows; U; Windows NT 5.0; it-IT; rv:1.1) Gecko/20020826"
divx+player+linux "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20020918"
dod+hq "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2a) Gecko/20020910"
dog "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.1) Gecko/20020826"
dog+collar "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1) Gecko/20020826"
dog+star "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US; rv:1.2a)
dom+refresh+ "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020913
Debian/1.1-1"don+quijote "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.1) Gecko/20020826"
direct+cable+connection+program "Mozilla/5.0 (Windows; U; Windows NT 5.0; it-IT;
rv:1.2a)
Gecko/20020910"
disassemble+bootsector+ "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
rv:1.2a) Gecko/20020910"
discount+airline+boston+chicage "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US;
rv:1.0.0)discount+airline+boston+chicago "Mozilla/5.0 (Macintosh; U; PPC Mac OS
X; en-US; rv:1.0.0)disk+juggler+4+serial "Mozilla/5.0 (Windows; U; Win 9x 4.90;
en-US; rv:1.1) Gecko/20020826"
Case+studies+on+alberta+fisheries+and+waste+treatment+facility+effects+in+Swan+hills
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.0) Gecko/20020710"
Discovery+Wings+Channel "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.0.1)
Gecko/20020826"
Dk+Pub+Merchandise "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2a)
Gecko/20020910"  
Do+Androids+Dream+of+Electric+Sheep%3F "Mozilla/5.0 (Windows; U; Windows NT 5.0;
en-US; rv:1.2a) Gecko/20020910"

Comment 4

[Dawn Endico]

note that if http://bugzilla.mozilla.org/ is no longer a static page
and if b.m.o is busy processing queries then it will take a long time
to load any page.

Comment 5

[Steve Dagley]

I believe the proper description is "Slower than maple syrup on a cold Vermont
morning".  Or something like that :-)

Comment 6

[Dawn Endico]

The current theory is that the recent slowness is due to the fact that we're
now forcing newbies to use the bug helper, which also uses simple search and
therefore there are a lot more simple searches going on than before.

On the other hand, i looked at allwordssubstr queries for today and there
still seems to be an abundance of inappropriate queries. If the bugzilla
helper theory were true then it seems like we'd be flooded with lots more
appropriate stuff. But my measurement of appropriate to inappropriate 
queries was hardly scientific, so so maybe we really are getting more
appropriate stuff.

One thing to note is that only about half of the allwordssubstr queries
have more than one word (as measured by grepping for +).

Comment 7

[Bradley Baetz (:bbaetz)]

I'm wondering if this is happening because of aggregation people appear to be
doing - if you have bugzilla and gooole selected, you'd get both, but may not
notice if buzilla didn't return results.

Even if bugzilla did return results, however, you still wouldnt' notice, because
the bugzilla search plugin's parsing hooks weren't updated to deal with the
templaitsed output for buglist.cgi, which changes last time bmo upgraded,
several months ago. So the user may jsut select everything on teh grounds that
its more likly to find a match, and then would jsut see the XUL page of results,
and so not notice a problem, since there weren't any bugzilla entries returning
false positives. Does that sound like a plausable scenario?

The fact that noone has reported bugzilla's serach plugin being broken in the
first place a tany point in the past 5-6 months may be a good reason to drop
that plugin - its obviously not being used as a data source, and we now have the
bugzilla sidebar for people who were using it just as a plain query textfield

Also, the bugzilla.src file has an update url, but the file that url points to
has a really wierd query. Does mozilla downlaod these updates, ever?

Comment 8

[Bradley Baetz (:bbaetz)]

The reason we're noticing this now is possibly because mysql sucks at optimising
OR queries, and product OR component thing now adds an extra two joins to the
query which weren't present before the upgrade. Oh, for a subselect ;)

I also wonder if this is the cause of the 'select everything' issue which used
to be present on bmo, which has been fixed by bugzilla forbidding queries with
no search terms.

Comment 9

[Dawn Endico]

Given the fact that we have been getting google queries since last spring,
it seems like that's not what is our immediate problem today. (although that
should still be investigated). I'm blaming today's major slowdown on 
something that changed with the upgrade. Moving this back to the bugzilla
product. This might be the query problem that bbaetz mentioned or a problem
with doing so many text searches now that we're doing more bugzilla helpers.

Comment 10

[Bradley Baetz (:bbaetz)]

Yeah, I'm blaming the fact that we're getting these queries on mozilla and the
slowdown on the upgrade due to the table joins + mysql's lack of optimisation
for OR.

Can someone take one of these silly queries, and attach the SQL + the mysql
EXPLAIN output, to confirm that? Both with and without the product/component,
please.

If theres some way of determining that a request came from the search plugin,
then we cna just ignore those requests. sgehani?

Comment 11

[Bradley Baetz (:bbaetz)]

Actually, I don't think that this is bugzilla helper. We only get 100-200 bugs a
day, and some of those used to search anyway. I'm guessing that the largest
issue is from quicksearch.

If we change the product/component search to 'equals' rather than 'contains' in
localconfig.js, do we get any better? That own't affect sidebar searches, but
those don't appear to have changed recently.

Comment 12

[Gervase Markham [:gerv]]

Well, QuickSearch has been taken off the front page. Where are people running it
from? Or has the performance improved since we did that?


Gerv

Comment 13

[Bradley Baetz (:bbaetz)]

Its still runnable from bugzilla helper - see bug 179608. Altough the user just
gets an error in that case.

Its also still runnable from the sidebar, although thats what the error (which
probably won't be seen by the user) is meant to catch. Also, the sidebar doesn't
use quickserach, it runs buglist.cgi automatically. See
http://lxr.mozilla.org/seamonkey/source/xpfe/components/search/datasets/bugzilla.src#13

As this search plugin is broken (see comment 7), and has been broken since May,
would anyone object ot me filing a Bugzilla bug to remove it? Which we can then
get on the 1.2 branch before 1.2 is released, to hopefully stop as many people
as possible?

If we can work out what happens with updates, we could also just push an update
to the m.o site which effectively disabled the plugin by pointing the url to a
dummy .html page. That would catch people who didn't update their mozilla build,
 too.

Comment 14

[Bradley Baetz (:bbaetz)]

Another possible solution is that according to
http://www.mozilla.org/projects/search/technical.html multi-site searches have
the http header 'MultiSite: true' (I tested that, and the documentation is
correct! ;). So we could block the sidebar searches via that, either by testing
in buglist.cgi or via a RewriteCond RewriteRule in bmo's .htaccess file. The
latter is probably faster, since you wouldn't have to spend time loading all the
Template modules.

Roughly how often are these requests coming, anyway?

Thats all separate to changing the product/component to be an exact match,
though, assuming that that suggestion does produce a speedup.

Comment 15

[Bradley Baetz (:bbaetz)]

... and you may want to block lxr and the 'mozilla.org' search engine from
multisite searches, too, in the same way.

endico: Are you seeing wierd queries for those two engines, too? If so, I'll
file an m.o bug on rejecting those in that .htaccess too.

Before we do that, though, it would be nice to have some confirmation taht this
is whats causing all the searches. Is there some way that you could easily
change apache's logging stuff to print that 'MultiSite' header to the log, or
just have buglist output to a file if thats the case, or something along those
lines?

Comment 16

[Bradley Baetz (:bbaetz)]

and for the lsta update before I got to bed:

The reason taht we started seeing these in Februray was because  bug 83052 was
checked in on 2002-02-14 16:58.

That bug moved the Bugzilla.src file (but not the one stored on m.o) from doing
a 'search by assignee' search to doing the current OR'd query, which was based
on what the quicksearch code did. Search by assignee is cheap, though, so its
unlikly that we sould ahve noticed unless someon ewas watching the logs on purpose.

I'm guessing that if dawn did her access_log search on assignees, w'ed see the
same destined-for-google responses appearing before February.

Anywa, to summarise, I think there are three things to do:

- Drop |MultiSearch: true| queries, preferably at the apache level.
  This is a bmo-specific bug

- Change/drop the various search plugins, and preferably get that into the 1.2
branch
  - Update the copy on m.o so that existing clients get updated.
  This is a browser (and m.o, for the second part) bug

- Drop the product/component stuff from QS
  - have the bugzilla helper version of QS restrict to the specific product,
probably by prepending |product:foo| before using quicksearch.js
  This is a bugzilla product bug.

Re the third point, I think the reason that this was done was so that people
could search for 'ftp login failure' and get ftp bugs. But this will in fact
give back _all_ ftp bugs, because its an OR. Searching for |network login
failure| will return all ftp/http/smtp/pop/etc bugs, because those components
contain the string 'network'. Searching for 'browser locks up' will return all
browser bugs.

quicksearch.js has a refernce to a blacklist of search strings (defined in
localconfig.js) which shouldn't be considered as a substring to patch for
products; I don't think that that is sufficient. afranke, you originally wrote
this, what was the reason for including this?

Oh, theres one more bug in the sidebar. For multiword searches, (say, |foo
bar|), quicksearch does ('product' 'subtring' 'foo' OR 'component' 'substring'
'foo' OR ...) AND ('product' 'substring' 'bar' OR 'component' 'substring' 'bar').

the sidebar does ('product' 'allwordsubstr' 'foo bar') OR ('component'
'alwordsubstr' 'foo bar'). This is silly, since its asking for all bugs with a
product containing 'foo' and 'bar' AND a component containting 'foo' and 'bar'.

This only does return any results because bugzilla has a bug - it brackets this
incorrectly, and so its expanded to the equivalent of ('product 'substring'
'foo' AND 'product' 'substr' 'bar' OR 'component' 'substring' 'foo' AND
'component' 'substring' 'bar' AND ...). I don't blame mysql for not managing to
pull out the joins ;) Since it can't, this becomes a query over every record,
adn wince the bmo upgrade, there are the two additional joins. I've filed this
as bug 179697, but although fixing that may mitigate the efffect of all these
searches, which seem mostly multiword from dawn's list, its not really related
to this bug

Comment 17

[Bradley Baetz (:bbaetz)]

>the sidebar does ('product' 'allwordsubstr' 'foo bar') OR ('component'
>'alwordsubstr' 'foo bar'). This is silly, since its asking for all bugs with a
>product containing 'foo' and 'bar' AND a component containting 'foo' and 'bar'.

s/AND/or/
s/and/AND/

is what I meant to write...

Comment 18

[Bradley Baetz (:bbaetz)]

rjc, comments on the search stuff above? (Specifically, comment #7)

Comment 19

[Bradley Baetz (:bbaetz)]

This is m.o specific, since its only an issue because bugzilla.mozilla.org is in
the search sidebar.

I'll go file bugs now to make this bug depend on this - see comment 16

Comment 20

[Andreas Franke (gone)]

> Re the third point, I think the reason that this was done was so that people
> could search for 'ftp login failure' and get ftp bugs. But this will in fact
> give back _all_ ftp bugs, because its an OR. Searching for |network login
> failure| will return all ftp/http/smtp/pop/etc bugs, because those components
> contain the string 'network'. Searching for 'browser locks up' will return all
> browser bugs.

I think this is a misunderstanding. Searching for "ftp login failure" will
return just a single bug, since all of "ftp", "login" and "failure" must occur
in some attribute of the bug. Same for the other two examples.

> quicksearch.js has a refernce to a blacklist of search strings (defined in
> localconfig.js) which shouldn't be considered as a substring to patch for
> products; I don't think that that is sufficient. afranke, you originally wrote
> this, what was the reason for including this?

If you search for "table row", then you don't want all "Browser" bugs with
"table" in the summary, so triggering the "Browser" product with its substring
"row" is an extreme case where the pure quicksearch method does not work. That's
why there is the blacklist to disable this substring trigger for products and
components for some substrings that are likely to be used in other contexts.
Another example is the word "new" which should not trigger the Product
"MailNews". But these are only very few exceptions, and they can be added to the
blacklist case by case whenever some substring relation is found to be
inappropriate and annoying.

Comment 21

[Bradley Baetz (:bbaetz)]

afranke: thats correct, but its the 'some attribute' part which is why we search
product and component as well as just the summary.

Comment 22

[Myk Melez [:myk] [@mykmelez]]

Bug 179960, the main issue from the upgrade, has been fixed, and b.m.o is now
back to its previous level of performance.  Removing dependency on the
regressions meta-bug.

Comment 23

[Christian 'CeeJay' Jensen]

AFAIK Bugzilla have been since the last post .. This bug needs reevaluating.
Bugzilla sure don't seem "slower than cold molasses" to me at least.

Though that's not saying it couldn't be made faster :)

Implementing Bug 247853 would speed up the download of the pages 300% - 400% and
also save a lot of bandwidth.

Bug 179827 and 180866 also seem to have been fixed although they have not been
marked as such.

Comment 24

[Christian 'CeeJay' Jensen]

The first line should have read "AFAIK Bugzilla have been UPGRADED since the
last post"

With the word upgraded in this makes so much more sense .. sorry for the small
tyop :)

Comment 25

[Gervase Markham [:gerv]]

Bugzilla is certainly no longer slower than cold molasses, since it's moved to
"mecha".

Is this bug still useful?

Gerv

Comment 26

[Dave Miller [:justdave]]

(In reply to comment #25)
> Bugzilla is certainly no longer slower than cold molasses, since it's moved to
> "mecha".
> 
> Is this bug still useful?

No.  The specific issues mentioned in this bug have been resolved a long time ago.

Bugzilla still has a few performance issues, but they're covered on other bugs
(like bug 75488)