Closed Bug 399693 Opened 13 years ago Closed 12 years ago

Upgrade bonsai(\-(www|l10n))?.mozilla.org

Categories

(mozilla.org Graveyard :: Server Operations, task, P1)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: reed, Assigned: reed)

References

Details

(Whiteboard: Thursday @ 8pm PDT 2008/05/15)

Both tinderbox.mozilla.org and bonsai.mozilla.org are running old checkouts from CVS (bonsai is currently older than tinderbox). It would be nice to get them updated to HEAD.
Assignee: server-ops → nobody
Component: Server Operations → Server Operations: Projects
This would pickup my changes from bug 394250 and let us resolve bug 379327.
showcheckins.cgi is useless with current Bonsai, and upgrading Bonsai would fix that (bug 325903). :)

Also, ben_h's work from the summer will be included (bug 390556).

Bonsai will now queue mail so that mail doesn't get lost when there is a large check-in (bug 52573).
Summary: Upgrade tinderbox.mozilla.org and bonsai.mozilla.org → Upgrade tinderbox.mozilla.org and bonsai(\-(www|l10n))?.mozilla.org
I'd like to see this get done sooner than later so we can make the unit test logs useful.  Currently people have to look at the full log when unit tests fail, and search for various strings, which sucks badly.
Blocks: 379327
Upgrading tinderbox will be pretty easy, there's no earth-shattering changes there.

Bonsai, however, will require some significant prep work for a couple reasons:

1) Bonsai now runs setuid bonsai.  This means a bonsai user needs to be created on the production box, and sufficient steps taken during the upgrade to make sure the bonsai user can access all of the appropriate data.

2) Incoming mail processing has changed to queue mail instead of immediately processing it.  This means a cron job needs to be set up to handle the actual processing (this is just like what we did to Tinderbox in the last big upgrade there).
Flags: needs-outage-window+
bug 395003 is another bonsai patch that's now in CVS that would be good to have on bonsai.m.o.
This is now on my short-term todo list.
Assignee: nobody → justdave
Component: Server Operations: Projects → Server Operations
Whiteboard: Tentatively scheduled for Thurs 11/15
OK, in order to meet the current schedule on this, bonsai-stage needs to be upgraded ASAP so we can make sure it all works before the downtime announcement goes out.  Will try to get this done later tonight.
Priority: -- → P1
Depends on: 52573
Tinderbox got updated tonight, Bonsai didn't because bug 52573 is incomplete, and not deployable without a working Makefile.  I attempted to cook one up this afternoon, but didn't finish before the window started.
Whiteboard: Tentatively scheduled for Thurs 11/15 → Tentatively scheduled for Tues 11/20
OK, I have a patch on bug 52573, and bonsai is deployed with that patch on
http://bonsai-stage.mozilla.org/ ,
http://bonsai-l10n-stage.mozilla.org/ , and
http://bonsai-www-stage.mozilla.org/

There seem to be some minor issues at the moment, trying to hash those out on #mozwebtools.
Whiteboard: Tentatively scheduled for Tues 11/20 → Tentatively scheduled for Thurs 11/22
tinderbox already got upgraded, updating summary.
Summary: Upgrade tinderbox.mozilla.org and bonsai(\-(www|l10n))?.mozilla.org → Upgrade bonsai(\-(www|l10n))?.mozilla.org
ok, the issues have been hashed.  Turned out to be missing prereqs on the staging box.  Bug 404832 filed for that.

So unless anyone finds any other issues, I think this is ready to push to production.
Couldn't get the mail handling code working properly, reverted back to the old version.  (see bug 53573 for details).  We have no way to test the mail handling on the staging system (would require an alternate cvs repo and a database separate from production), but I was under the impression that it was already in use on the bonsai system maintained by the guy who wrote the new code and naively assumed it would work... (it's almost the same code in use on tinderbox, which we already know works)
Whiteboard: Tentatively scheduled for Thurs 11/22 → postponed for bug 52573
OK, so here's what we need to do to get this moving again...

1) clone the bonsai database (say, bonsai_test or something)
2) have bonsai-stage talk to that database instead of the production one
3) split the mail at the production bonsai (dm-webtools02) so it delivers both locally and to bonsai-stage (dm-webtools03).  See tinderbox's aliases for examples, tinderbox-stage is already set up this way.

Once the above is done, we can upgrade bonsai-stage and freely debug it there.

Ideally we should put a fake cvs repo on it, too, but we can always find some obscure part of the tree that nobody cares if we screw with and check in a bunch of whitespace changes to test.
This isn't going anywhere for a while, obviously, no reason to have it cluttering the main queue.
Component: Server Operations → Server Operations: Projects
Blocks: 431975
Assignee: justdave → reed
(In reply to comment #13)
> OK, so here's what we need to do to get this moving again...
> 
> 1) clone the bonsai database (say, bonsai_test or something)
> 2) have bonsai-stage talk to that database instead of the production one
> 3) split the mail at the production bonsai (dm-webtools02) so it delivers both
> locally and to bonsai-stage (dm-webtools03).  See tinderbox's aliases for
> examples, tinderbox-stage is already set up this way.
> 
> Once the above is done, we can upgrade bonsai-stage and freely debug it there.

All of the above things have been completed. I've also enabled $debug in processMail.pl.
I believe I have figured out the problem. I've attached a patch in bug 52573.
Status: NEW → ASSIGNED
Bug 52573 is fixed. This is ready to go. Just need a downtime window.
Component: Server Operations: Projects → Server Operations
Whiteboard: postponed for bug 52573 → need downtime window
Whiteboard: need downtime window → Tuesday @ ?pm PDT 2008/05/13
Whiteboard: Tuesday @ ?pm PDT 2008/05/13 → Thursday @ ?pm PDT 2008/05/15
Whiteboard: Thursday @ ?pm PDT 2008/05/15 → Thursday @ 8pm PDT 2008/05/15
This was completed earlier tonight. Please file bugs for any issues that may come up as a result of this much overdue upgrade.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.