585691 - split incoming build processing from every other part of tinderbox

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

Currently dm-webtools02 is used for all aspects of Tinderbox. Lately we've been hitting a load of load issues on that machine, which I'm told are mostly from:
showlog.cgi
showbuilds.cgi
addnote.cgi
and processbuild.pl

showlog.cgi is already behind a varnish proxy, for some incremental gains. showbuilds.cgi invalidates so frequently that it's probably not worth it to put it behind one, but we might want to look at that data.
addnote.cgi is for POST'ing only, so caching doesn't do anything there.

In an e-mail thread, justdave brought up the idea of splitting off processbuild.pl from the rest of Tinderbox to alleviate a bunch of load. I'm told that we need additional hardware to do this (and obviously, there's IT time required to set it up), but that it should otherwise be possible.

The tree has been closed multiple times in the past week due to this issue and I believe that it will only get worse as we head towards Firefox 4.0 end game. Therefore, I'd really like to see this happen soon.

Comment 1

[Robert Helmer [:rhelmer]]

I have a patch for showlog.cgi which basically splits processing and request:

https://bug390341.bugzilla.mozilla.org/attachment.cgi?id=446579

This patch creates a brief.gz and full.gz every time notes or start/end emails come in, and if the client accepts gzip encoding then it essentially outputs the files as-is.

This was part of bug 390341 but we decided to go the simpler route and have Apache gzip the output using mod_deflate instead.

Comment 2

[Robert Helmer [:rhelmer]]

showbuilds.cgi could do something similar; the only problem I see is that it sets maxdate to the current date; if it instead set this to the time of the last build, it would allow better caching.

The problem with showbuilds.cgi is that people want to go "back in time", which causes it to load the data files from disk and grep them, per request (which can be very slow.)

Tinderbox server *could* create a new cached output file for every build that comes in, which would decrease request time and memory usage but would increase disk usage.

All that said, I am working right now on profiling Tinderbox server so we can figure out where it's really spending time, I think we should do this before making changes (so we can measure if it actually makes a difference or not.)

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to comment #0)
> In an e-mail thread, justdave brought up the idea of splitting off
> processbuild.pl from the rest of Tinderbox to alleviate a bunch of load. I'm
> told that we need additional hardware to do this (and obviously, there's IT
> time required to set it up), but that it should otherwise be possible.

Sorry, I missed this bit (and the point of this bug); yes splitting off processbuild.pl should probably help, although keep in mind that users can edit certain things (notes, administrative settings like scrape builds). These requests could be redirected to the "write" server perhaps.

I don't want to hijack this bug for comment 1 or comment 2, those should go somewhere else.

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to comment #3)
> I don't want to hijack this bug for comment 1 or comment 2, those should go
> somewhere else.

bug 582246, please excuse the noise.

Comment 5

[Dave Miller [:justdave]]

incoming mail processing is now handled on dm-webtools02 and web traffic goes to dm-webtools05.  Seems to have helped some at first glance.