All users were logged out of Bugzilla on October 13th, 2018
It takes 15 seconds just for http://bugzilla.mozilla.org/ to appear.
Right. AOL-TW should donate $1M for investments in faster servers and phatter pipes.
Yeah, apparently it appears that bugzilla is getting queries meant for google, possibly via the sidebar, which is placing a massive load on bugzilla at the moment -> myk
Assignee: justdave → endico
Component: Bugzilla-General → Bugzilla: Other moz.org Issues
Product: Bugzilla → mozilla.org
QA Contact: matty → myk
Version: unspecified → other
I've been looking at b.m.o's http log files and it looks like we've had this problem since last spring. I searched for buglist.cgi queries containing the 'allwordssubstr' parameter in log files for oct2, jul31, may1, mar6 and jan9 and only the jan9 log didn't contain inappropriate looking queries. (The jan 9 file had only 9 allwordssubstr hits). The oct 9 file seemed like nearly all the allwordsubstr queries were meant for google. The march file seemed like it had a higher ratio of valid stuff. A lot of the query strings were urls. There was a wide variety of user agents and platforms. Here's a small selection of search strings from the log covering sep25-oct2. Since so many of these search strings seem to have nothing to do with mozilla, we think that somehow bugzilla is getting set as the default search engine for some people and their normal queries are getting sent to bugzilla instead of netscape.com, google or someplace sensible. Harvard+school+of+Public+Heath+ "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1) Gecko/20020826" Healthy+Bites+Grill "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1) Gecko/20020826" Heart+of+Winter+Patches "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0.0) Gecko/20020530" Helga+M.+Novak+ "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1a) Gecko/20020611" Helga+M.+Novak++%22eis%22 "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1a) Gecko/20020611" Herv+Peterschmitt "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.0) Gecko/20020530" Hilton+Minneapolis "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2b) Gecko/20020926" Hollidat+Inn+Inc "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1) Gecko/20020826" Holliday+Inn+Inc "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1) Gecko/20020826" How+To+Change+the+Windows+XP+Boot+Logo "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1b) Gecko/20020721" Hulton "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.1) Gecko/20020826" I-DEAS+troubleshooting "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2a) Gecko/20020910" IFF+Fighter+Follow+on+Training "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.0.1) Gecko/20020826" IFF+Fighter+Follow+on+Training%2C+F-16 "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.0.1) Gecko/20020826" divx+for+linux "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0.0) Gecko/20020530" divx+lunacy "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.1) Gecko/20020826" divx+player "Mozilla/5.0 (Windows; U; Windows NT 5.0; it-IT; rv:1.1) Gecko/20020826" divx+player+linux "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20020918" dod+hq "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2a) Gecko/20020910" dog "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.1) Gecko/20020826" dog+collar "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1) Gecko/20020826" dog+star "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US; rv:1.2a) dom+refresh+ "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020913 Debian/1.1-1"don+quijote "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1) Gecko/20020826" direct+cable+connection+program "Mozilla/5.0 (Windows; U; Windows NT 5.0; it-IT; rv:1.2a) Gecko/20020910" disassemble+bootsector+ "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2a) Gecko/20020910" discount+airline+boston+chicage "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US; rv:1.0.0)discount+airline+boston+chicago "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US; rv:1.0.0)disk+juggler+4+serial "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.1) Gecko/20020826" Case+studies+on+alberta+fisheries+and+waste+treatment+facility+effects+in+Swan+hills "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.0) Gecko/20020710" Discovery+Wings+Channel "Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.0.1) Gecko/20020826" Dk+Pub+Merchandise "Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2a) Gecko/20020910" Do+Androids+Dream+of+Electric+Sheep%3F "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2a) Gecko/20020910"
note that if http://bugzilla.mozilla.org/ is no longer a static page and if b.m.o is busy processing queries then it will take a long time to load any page.
I believe the proper description is "Slower than maple syrup on a cold Vermont morning". Or something like that :-)
The current theory is that the recent slowness is due to the fact that we're now forcing newbies to use the bug helper, which also uses simple search and therefore there are a lot more simple searches going on than before. On the other hand, i looked at allwordssubstr queries for today and there still seems to be an abundance of inappropriate queries. If the bugzilla helper theory were true then it seems like we'd be flooded with lots more appropriate stuff. But my measurement of appropriate to inappropriate queries was hardly scientific, so so maybe we really are getting more appropriate stuff. One thing to note is that only about half of the allwordssubstr queries have more than one word (as measured by grepping for +).
I'm wondering if this is happening because of aggregation people appear to be doing - if you have bugzilla and gooole selected, you'd get both, but may not notice if buzilla didn't return results. Even if bugzilla did return results, however, you still wouldnt' notice, because the bugzilla search plugin's parsing hooks weren't updated to deal with the templaitsed output for buglist.cgi, which changes last time bmo upgraded, several months ago. So the user may jsut select everything on teh grounds that its more likly to find a match, and then would jsut see the XUL page of results, and so not notice a problem, since there weren't any bugzilla entries returning false positives. Does that sound like a plausable scenario? The fact that noone has reported bugzilla's serach plugin being broken in the first place a tany point in the past 5-6 months may be a good reason to drop that plugin - its obviously not being used as a data source, and we now have the bugzilla sidebar for people who were using it just as a plain query textfield Also, the bugzilla.src file has an update url, but the file that url points to has a really wierd query. Does mozilla downlaod these updates, ever?
The reason we're noticing this now is possibly because mysql sucks at optimising OR queries, and product OR component thing now adds an extra two joins to the query which weren't present before the upgrade. Oh, for a subselect ;) I also wonder if this is the cause of the 'select everything' issue which used to be present on bmo, which has been fixed by bugzilla forbidding queries with no search terms.
Given the fact that we have been getting google queries since last spring, it seems like that's not what is our immediate problem today. (although that should still be investigated). I'm blaming today's major slowdown on something that changed with the upgrade. Moving this back to the bugzilla product. This might be the query problem that bbaetz mentioned or a problem with doing so many text searches now that we're doing more bugzilla helpers.
Component: Bugzilla: Other moz.org Issues → Query/Bug List
Product: mozilla.org → Bugzilla
Version: other → 2.17.1
Yeah, I'm blaming the fact that we're getting these queries on mozilla and the slowdown on the upgrade due to the table joins + mysql's lack of optimisation for OR. Can someone take one of these silly queries, and attach the SQL + the mysql EXPLAIN output, to confirm that? Both with and without the product/component, please. If theres some way of determining that a request came from the search plugin, then we cna just ignore those requests. sgehani?
Actually, I don't think that this is bugzilla helper. We only get 100-200 bugs a day, and some of those used to search anyway. I'm guessing that the largest issue is from quicksearch. If we change the product/component search to 'equals' rather than 'contains' in localconfig.js, do we get any better? That own't affect sidebar searches, but those don't appear to have changed recently.
Well, QuickSearch has been taken off the front page. Where are people running it from? Or has the performance improved since we did that? Gerv
Its still runnable from bugzilla helper - see bug 179608. Altough the user just gets an error in that case. Its also still runnable from the sidebar, although thats what the error (which probably won't be seen by the user) is meant to catch. Also, the sidebar doesn't use quickserach, it runs buglist.cgi automatically. See http://lxr.mozilla.org/seamonkey/source/xpfe/components/search/datasets/bugzilla.src#13 As this search plugin is broken (see comment 7), and has been broken since May, would anyone object ot me filing a Bugzilla bug to remove it? Which we can then get on the 1.2 branch before 1.2 is released, to hopefully stop as many people as possible? If we can work out what happens with updates, we could also just push an update to the m.o site which effectively disabled the plugin by pointing the url to a dummy .html page. That would catch people who didn't update their mozilla build, too.
Another possible solution is that according to http://www.mozilla.org/projects/search/technical.html multi-site searches have the http header 'MultiSite: true' (I tested that, and the documentation is correct! ;). So we could block the sidebar searches via that, either by testing in buglist.cgi or via a RewriteCond RewriteRule in bmo's .htaccess file. The latter is probably faster, since you wouldn't have to spend time loading all the Template modules. Roughly how often are these requests coming, anyway? Thats all separate to changing the product/component to be an exact match, though, assuming that that suggestion does produce a speedup.
... and you may want to block lxr and the 'mozilla.org' search engine from multisite searches, too, in the same way. endico: Are you seeing wierd queries for those two engines, too? If so, I'll file an m.o bug on rejecting those in that .htaccess too. Before we do that, though, it would be nice to have some confirmation taht this is whats causing all the searches. Is there some way that you could easily change apache's logging stuff to print that 'MultiSite' header to the log, or just have buglist output to a file if thats the case, or something along those lines?
and for the lsta update before I got to bed: The reason taht we started seeing these in Februray was because bug 83052 was checked in on 2002-02-14 16:58. That bug moved the Bugzilla.src file (but not the one stored on m.o) from doing a 'search by assignee' search to doing the current OR'd query, which was based on what the quicksearch code did. Search by assignee is cheap, though, so its unlikly that we sould ahve noticed unless someon ewas watching the logs on purpose. I'm guessing that if dawn did her access_log search on assignees, w'ed see the same destined-for-google responses appearing before February. Anywa, to summarise, I think there are three things to do: - Drop |MultiSearch: true| queries, preferably at the apache level. This is a bmo-specific bug - Change/drop the various search plugins, and preferably get that into the 1.2 branch - Update the copy on m.o so that existing clients get updated. This is a browser (and m.o, for the second part) bug - Drop the product/component stuff from QS - have the bugzilla helper version of QS restrict to the specific product, probably by prepending |product:foo| before using quicksearch.js This is a bugzilla product bug. Re the third point, I think the reason that this was done was so that people could search for 'ftp login failure' and get ftp bugs. But this will in fact give back _all_ ftp bugs, because its an OR. Searching for |network login failure| will return all ftp/http/smtp/pop/etc bugs, because those components contain the string 'network'. Searching for 'browser locks up' will return all browser bugs. quicksearch.js has a refernce to a blacklist of search strings (defined in localconfig.js) which shouldn't be considered as a substring to patch for products; I don't think that that is sufficient. afranke, you originally wrote this, what was the reason for including this? Oh, theres one more bug in the sidebar. For multiword searches, (say, |foo bar|), quicksearch does ('product' 'subtring' 'foo' OR 'component' 'substring' 'foo' OR ...) AND ('product' 'substring' 'bar' OR 'component' 'substring' 'bar'). the sidebar does ('product' 'allwordsubstr' 'foo bar') OR ('component' 'alwordsubstr' 'foo bar'). This is silly, since its asking for all bugs with a product containing 'foo' and 'bar' AND a component containting 'foo' and 'bar'. This only does return any results because bugzilla has a bug - it brackets this incorrectly, and so its expanded to the equivalent of ('product 'substring' 'foo' AND 'product' 'substr' 'bar' OR 'component' 'substring' 'foo' AND 'component' 'substring' 'bar' AND ...). I don't blame mysql for not managing to pull out the joins ;) Since it can't, this becomes a query over every record, adn wince the bmo upgrade, there are the two additional joins. I've filed this as bug 179697, but although fixing that may mitigate the efffect of all these searches, which seem mostly multiword from dawn's list, its not really related to this bug
>the sidebar does ('product' 'allwordsubstr' 'foo bar') OR ('component' >'alwordsubstr' 'foo bar'). This is silly, since its asking for all bugs with a >product containing 'foo' and 'bar' AND a component containting 'foo' and 'bar'. s/AND/or/ s/and/AND/ is what I meant to write...
rjc, comments on the search stuff above? (Specifically, comment #7)
This is m.o specific, since its only an issue because bugzilla.mozilla.org is in the search sidebar. I'll go file bugs now to make this bug depend on this - see comment 16
Component: Query/Bug List → Bugzilla: Other moz.org Issues
Product: Bugzilla → mozilla.org
Version: 2.17.1 → other
> Re the third point, I think the reason that this was done was so that people > could search for 'ftp login failure' and get ftp bugs. But this will in fact > give back _all_ ftp bugs, because its an OR. Searching for |network login > failure| will return all ftp/http/smtp/pop/etc bugs, because those components > contain the string 'network'. Searching for 'browser locks up' will return all > browser bugs. I think this is a misunderstanding. Searching for "ftp login failure" will return just a single bug, since all of "ftp", "login" and "failure" must occur in some attribute of the bug. Same for the other two examples. > quicksearch.js has a refernce to a blacklist of search strings (defined in > localconfig.js) which shouldn't be considered as a substring to patch for > products; I don't think that that is sufficient. afranke, you originally wrote > this, what was the reason for including this? If you search for "table row", then you don't want all "Browser" bugs with "table" in the summary, so triggering the "Browser" product with its substring "row" is an extreme case where the pure quicksearch method does not work. That's why there is the blacklist to disable this substring trigger for products and components for some substrings that are likely to be used in other contexts. Another example is the word "new" which should not trigger the Product "MailNews". But these are only very few exceptions, and they can be added to the blacklist case by case whenever some substring relation is found to be inappropriate and annoying.
afranke: thats correct, but its the 'some attribute' part which is why we search product and component as well as just the summary.
Bug 179960, the main issue from the upgrade, has been fixed, and b.m.o is now back to its previous level of performance. Removing dependency on the regressions meta-bug.
No longer blocks: 179176
AFAIK Bugzilla have been since the last post .. This bug needs reevaluating. Bugzilla sure don't seem "slower than cold molasses" to me at least. Though that's not saying it couldn't be made faster :) Implementing Bug 247853 would speed up the download of the pages 300% - 400% and also save a lot of bandwidth. Bug 179827 and 180866 also seem to have been fixed although they have not been marked as such.
The first line should have read "AFAIK Bugzilla have been UPGRADED since the last post" With the word upgraded in this makes so much more sense .. sorry for the small tyop :)
Bugzilla is certainly no longer slower than cold molasses, since it's moved to "mecha". Is this bug still useful? Gerv
(In reply to comment #25) > Bugzilla is certainly no longer slower than cold molasses, since it's moved to > "mecha". > > Is this bug still useful? No. The specific issues mentioned in this bug have been resolved a long time ago. Bugzilla still has a few performance issues, but they're covered on other bugs (like bug 75488)
Status: NEW → RESOLVED
Last Resolved: 14 years ago
Resolution: --- → FIXED
Component: Bugzilla: Other b.m.o Issues → General
Product: mozilla.org → bugzilla.mozilla.org
You need to log in before you can comment on or make changes to this bug.