623821 - build-4hr not updating on schedule

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Zandr Milewski [:zandr]]

One of the frequent alerts has been on the age of the build-4hr file.

IT set up alerts in Bug 602377 at 7 minutes for all of these files. As I understand it, these files are generated every 5min, and an rsync runs every 2min. 7 minutes is therefore a hair-trigger alert for worst-case timing, but it doesn't seem to alert often.

Except for the 4hr file, which seems to alert a few times a day. I opened up the timing to 20 minutes

Comment 1

[Zandr Milewski [:zandr]]

bah, hit submit too early.

...timing to 20 minutes, but it still alerts occaisonally. This suggests that something is keeping the file from being generated on a timely basis.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Having a look at cruncher, it's a very busy box.  It may just be the time that I'm running it, but the job currently pegging the CPU is

python reporter.py -z -o /var/www/html/builds/builds-4hr.js.gz --starttime 1294941881

So we might want to look at moving that elsewhere, but perhaps we need to optimize that process somehow?  Is there a table that's gotten too big?  Catlee, can you take a brief look just to make sure there's not some low-hanging fruit we can pick up?

Comment 3

[Chris AtLee [:catlee]]

(In reply to comment #2)
> Having a look at cruncher, it's a very busy box.  It may just be the time that
> I'm running it, but the job currently pegging the CPU is
> 
> python reporter.py -z -o /var/www/html/builds/builds-4hr.js.gz --starttime
> 1294941881
> 
> So we might want to look at moving that elsewhere, but perhaps we need to
> optimize that process somehow?  Is there a table that's gotten too big? 
> Catlee, can you take a brief look just to make sure there's not some
> low-hanging fruit we can pick up?

already on it

Comment 4

[Chris AtLee [:catlee]]

Ok, I added some caching with memcached so that the 4hr report takes ~30s on average to generate (down from 1:30), and the daily report takes ~80s (down from around 4 minutes).

This should remove lock contention and allow the builds-4hr file to be generated on schedule.