Closed Bug 585691 Opened 12 years ago Closed 12 years ago
split incoming build processing from every other part of tinderbox
Currently dm-webtools02 is used for all aspects of Tinderbox. Lately we've been hitting a load of load issues on that machine, which I'm told are mostly from: showlog.cgi showbuilds.cgi addnote.cgi and processbuild.pl showlog.cgi is already behind a varnish proxy, for some incremental gains. showbuilds.cgi invalidates so frequently that it's probably not worth it to put it behind one, but we might want to look at that data. addnote.cgi is for POST'ing only, so caching doesn't do anything there. In an e-mail thread, justdave brought up the idea of splitting off processbuild.pl from the rest of Tinderbox to alleviate a bunch of load. I'm told that we need additional hardware to do this (and obviously, there's IT time required to set it up), but that it should otherwise be possible. The tree has been closed multiple times in the past week due to this issue and I believe that it will only get worse as we head towards Firefox 4.0 end game. Therefore, I'd really like to see this happen soon.
I have a patch for showlog.cgi which basically splits processing and request: https://bug390341.bugzilla.mozilla.org/attachment.cgi?id=446579 This patch creates a brief.gz and full.gz every time notes or start/end emails come in, and if the client accepts gzip encoding then it essentially outputs the files as-is. This was part of bug 390341 but we decided to go the simpler route and have Apache gzip the output using mod_deflate instead.
showbuilds.cgi could do something similar; the only problem I see is that it sets maxdate to the current date; if it instead set this to the time of the last build, it would allow better caching. The problem with showbuilds.cgi is that people want to go "back in time", which causes it to load the data files from disk and grep them, per request (which can be very slow.) Tinderbox server *could* create a new cached output file for every build that comes in, which would decrease request time and memory usage but would increase disk usage. All that said, I am working right now on profiling Tinderbox server so we can figure out where it's really spending time, I think we should do this before making changes (so we can measure if it actually makes a difference or not.)
(In reply to comment #0) > In an e-mail thread, justdave brought up the idea of splitting off > processbuild.pl from the rest of Tinderbox to alleviate a bunch of load. I'm > told that we need additional hardware to do this (and obviously, there's IT > time required to set it up), but that it should otherwise be possible. Sorry, I missed this bit (and the point of this bug); yes splitting off processbuild.pl should probably help, although keep in mind that users can edit certain things (notes, administrative settings like scrape builds). These requests could be redirected to the "write" server perhaps. I don't want to hijack this bug for comment 1 or comment 2, those should go somewhere else.
(In reply to comment #3) > I don't want to hijack this bug for comment 1 or comment 2, those should go > somewhere else. bug 582246, please excuse the noise.
incoming mail processing is now handled on dm-webtools02 and web traffic goes to dm-webtools05. Seems to have helped some at first glance.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.