Closed Bug 397298 Opened 12 years ago Closed 12 years ago

Tinderbox server configuration needs to be audited

Categories

(mozilla.org Graveyard :: Server Operations: Projects, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: justdave, Assigned: reed)

References

Details

I currently get about 400 to 800 cron-generated emails per day from the tinderbox servers complaining of various warnings while processing incoming mail (the mail in question currently goes directly to me instead of sysalerts to avoid flooding the list).  We've finally gotten enough debugging information into the code to figure out why the errors are happening, and it turns out to be a whole mess of misconfigured trees.  Each tree in Tinderbox is tied to a tree in Bonsai, which is used to determine whether the Tinderbox tree in question is closed for checkins or not (the controls to open and close the trees are actually in Bonsai).  The problem is that many of the trees in Tinderbox are tied to Bonsai trees that don't exist.  In some cases, the tree may have formerly existed, and then later gone away.  In most of the cases, it appears that the person who set up the tree in Tinderbox didn't understand what the setting was for, and just named it at random.  Many of them have the name of the Tinderbox tree in the Bonsai tree field, when there's no tree by that name in Bonsai.

We need to generate a list of which Tinderbox trees are currently pointed at which Bonsai trees, and for the ones that are pointed at Bonsai trees that don't exist, coordinate with build (or the developers) to determine which Bonsai tree they should be tied to.

While we're in there....  Tinderbox very recently gained the ability to talk to more than one Bonsai installation (we have three Bonsais, for three different CVS repos).  This means that each Tinderbox tree also needs to be tied to a specific CVS repo and a specific Bonsai.  The L10n stuff in particular is mostly busted in this regard, and so all of the L10n trees need to be set to actually point at the L10n Bonsai.

It would probably be relatively easy to write up a script that would read in all of the treedata.pl files in tinderbox and generate a nice report for us that summarizes the data we need to audit for each viewing and sharing.  I would pull the Tinderbox tree name, its CVSROOT, which Bonsai it's tied to, and which Bonsai Tree it's tied to, in a CVS file and/or HTML table that we can post somewhere.

This is probably something pretty much anyone with access to the server could handle.  Not sure how soon I could get to it if it waits for me, since Justin has me on some other projects right now.
Assignee: server-ops → nobody
Component: Server Operations → Server Operations: Projects
This should be done sooner rather than later, we're getting lots of complaints about the missing checkin data on the l10n pages.  Anyone in server ops could do this (it's a grunt work thing), but if you really want it to wait for me it'll happen after the Bugzilla upgrade.
Assignee: nobody → server-ops
Component: Server Operations: Projects → Server Operations
since the upgrade's over, and nobody else took this yet, guess it's mine now :(
Assignee: server-ops → justdave
If you tell me how to do one or a few of these or even what the process is / how to start, I can help.
mrz and I agreed that given the (relative) low priority on this it's probably better back in projects for now.  Maybe we'll let an intern hack on it in the summer, and hopefully the tools to make this easier (bug 412710, bug 398049, bug 398050) will be done by then, too.
Assignee: justdave → nobody
Component: Server Operations → Server Operations: Projects
Depends on: 398049, 398050, 412710
Assignee: nobody → reed
I've successfully audited all 195 tinderbox tree configs. Multiple problems were fixed. I'm still working on the dependent bugs just to help make this process easier for the future.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.