Closed Bug 585691 Opened 10 years ago Closed 10 years ago

split incoming build processing from every other part of tinderbox

Categories

(mozilla.org Graveyard :: Server Operations, task, critical)

x86
macOS
task
Not set
critical

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: justdave)

References

Details

Currently dm-webtools02 is used for all aspects of Tinderbox. Lately we've been hitting a load of load issues on that machine, which I'm told are mostly from:
showlog.cgi
showbuilds.cgi
addnote.cgi
and processbuild.pl

showlog.cgi is already behind a varnish proxy, for some incremental gains. showbuilds.cgi invalidates so frequently that it's probably not worth it to put it behind one, but we might want to look at that data.
addnote.cgi is for POST'ing only, so caching doesn't do anything there.

In an e-mail thread, justdave brought up the idea of splitting off processbuild.pl from the rest of Tinderbox to alleviate a bunch of load. I'm told that we need additional hardware to do this (and obviously, there's IT time required to set it up), but that it should otherwise be possible.

The tree has been closed multiple times in the past week due to this issue and I believe that it will only get worse as we head towards Firefox 4.0 end game. Therefore, I'd really like to see this happen soon.
Assignee: server-ops → justdave
I have a patch for showlog.cgi which basically splits processing and request:

https://bug390341.bugzilla.mozilla.org/attachment.cgi?id=446579

This patch creates a brief.gz and full.gz every time notes or start/end emails come in, and if the client accepts gzip encoding then it essentially outputs the files as-is.

This was part of bug 390341 but we decided to go the simpler route and have Apache gzip the output using mod_deflate instead.
showbuilds.cgi could do something similar; the only problem I see is that it sets maxdate to the current date; if it instead set this to the time of the last build, it would allow better caching.

The problem with showbuilds.cgi is that people want to go "back in time", which causes it to load the data files from disk and grep them, per request (which can be very slow.)

Tinderbox server *could* create a new cached output file for every build that comes in, which would decrease request time and memory usage but would increase disk usage.

All that said, I am working right now on profiling Tinderbox server so we can figure out where it's really spending time, I think we should do this before making changes (so we can measure if it actually makes a difference or not.)
(In reply to comment #0)
> In an e-mail thread, justdave brought up the idea of splitting off
> processbuild.pl from the rest of Tinderbox to alleviate a bunch of load. I'm
> told that we need additional hardware to do this (and obviously, there's IT
> time required to set it up), but that it should otherwise be possible.

Sorry, I missed this bit (and the point of this bug); yes splitting off processbuild.pl should probably help, although keep in mind that users can edit certain things (notes, administrative settings like scrape builds). These requests could be redirected to the "write" server perhaps.

I don't want to hijack this bug for comment 1 or comment 2, those should go somewhere else.
(In reply to comment #3)
> I don't want to hijack this bug for comment 1 or comment 2, those should go
> somewhere else.

bug 582246, please excuse the noise.
Blocks: 582246
Depends on: 586505
No longer depends on: 586505
incoming mail processing is now handled on dm-webtools02 and web traffic goes to dm-webtools05.  Seems to have helped some at first glance.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.