447758 - bonsai-l10n can't handle the load

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Axel Hecht [:Pike]]

bonsai-l10n seems to be underpowered. Loads of cvsquery.cgi time out on a rather regular basis, and the queries are generally a good deal slower than the ones on bonsai.mozilla.org.

Can we get more power to that?

Comment 1

[Dave Miller [:justdave]]

We've been having issues with bonsai-l10n cron jobs piling up on each other, too, that we've been trying to figure out...  and I think I just did.  See the upcoming attachments.

Comment 2

[Dave Miller [:justdave]]

Attachment: main bonsai checkins schema

Comment 3

[Dave Miller [:justdave]]

Attachment: bonsai-l10n checkins schema

Open these two attachments in separate tabs, and flip back and forth between them, the difference should be obvious.

These are both running identical code, their schemas should be identical as well.

Comment 4

[Reed Loden [:reed]]

Executed the following on bonsai_l10n and bonsai_www databases:

ALTER TABLE `checkins` MODIFY COLUMN `type` ENUM('Change','Add','Remove')  CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT 'Change',
 ADD PRIMARY KEY (`repositoryid`, `dirid`, `fileid`, `revision`)
, DROP INDEX `repositoryid`,
 ADD INDEX `rep_branch_when`(`repositoryid`, `branchid`, `ci_when`),
 ADD INDEX `when_rep_branch`(`ci_when`, `repositoryid`, `branchid`);

Comment 5

[Axel Hecht [:Pike]]

It seemed to be snappier today, but then I landed a new locale (l10n/th), and now bonsai-l10n ain't picking up new changes.

http://bonsai-l10n.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=HEAD&branchtype=match&dir=l10n&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=day&mindate=1131523200&maxdate=&cvsroot=%2Fl10n shows the initial check-in for thai, but there was a fix up in reporter which doesn't show up on bonsai for some 2.5 hours, 

cvs log th/extensions/reporter/chrome/reporterOverlay.properties

RCS file: /l10n/l10n/th/extensions/reporter/chrome/reporterOverlay.properties,v
Working file: th/extensions/reporter/chrome/reporterOverlay.properties
head: 1.1
branch:
locks: strict
access list:
symbolic names:
keyword substitution: kv
total revisions: 1;     selected revisions: 1
description:
----------------------------
revision 1.1
date: 2008/08/04 11:06:19;  author: axel%pike.org;  state: Exp;
bug 299423/443373, initial landing of the Thai localization, patch by [e-mail], r/patches=[e-mail]
=============================================================================

I see bugmail that indicates further l10n checkins, too.

Reopening as this is probably either not fixed or regressed something else.

Comment 6

[Axel Hecht [:Pike]]

bonsai-l10n seemed to come back to life about an hour ago.

I have more locales to land, if you want to coordinate so that you can watch in real time, we should come up with a landing time that's good for both you and I.

Comment 7

[Reed Loden [:reed]]

Cron mail is spewing this:

Lock unavailable: Resource temporarily unavailable at /var/www/webtools/bonsai-l10n/processMail.pl line 51.

Looks like it's just taking a while to run processMail.pl...

Comment 8

[Reed Loden [:reed]]

bear/cls/timeless/justdave: ideas on comment #7? This is new since the switchover to processMail.pl.

Comment 9

[Reed Loden [:reed]]

Ok, I did the simple thing of just closing and reopening the tree. This seems to have sped bonsai-l10n up tremendously. It looks like bonsai-l10n was loading a 25MB file of checkins every time it tried to do something, so no wonder it was taking forever. Everything seems to be pretty quick now. Reopen if you're still seeing problems.