399693 - Upgrade bonsai(\-(www|l10n))?.mozilla.org

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task, P1)

Description

[Reed Loden [:reed]]

Both tinderbox.mozilla.org and bonsai.mozilla.org are running old checkouts from CVS (bonsai is currently older than tinderbox). It would be nice to get them updated to HEAD.

Comment 1

[(not currently active) Ted Mielczarek]

This would pickup my changes from bug 394250 and let us resolve bug 379327.

Comment 2

[Reed Loden [:reed]]

showcheckins.cgi is useless with current Bonsai, and upgrading Bonsai would fix that (bug 325903). :)

Also, ben_h's work from the summer will be included (bug 390556).

Bonsai will now queue mail so that mail doesn't get lost when there is a large check-in (bug 52573).

Comment 3

[(not currently active) Ted Mielczarek]

I'd like to see this get done sooner than later so we can make the unit test logs useful.  Currently people have to look at the full log when unit tests fail, and search for various strings, which sucks badly.

Comment 4

[Dave Miller [:justdave]]

Upgrading tinderbox will be pretty easy, there's no earth-shattering changes there.

Bonsai, however, will require some significant prep work for a couple reasons:

1) Bonsai now runs setuid bonsai.  This means a bonsai user needs to be created on the production box, and sufficient steps taken during the upgrade to make sure the bonsai user can access all of the appropriate data.

2) Incoming mail processing has changed to queue mail instead of immediately processing it.  This means a cron job needs to be set up to handle the actual processing (this is just like what we did to Tinderbox in the last big upgrade there).

Comment 5

[Christian :Biesinger (don't email me, ping me on IRC)]

bug 395003 is another bonsai patch that's now in CVS that would be good to have on bonsai.m.o.

Comment 6

[Dave Miller [:justdave]]

This is now on my short-term todo list.

Comment 7

[Dave Miller [:justdave]]

OK, in order to meet the current schedule on this, bonsai-stage needs to be upgraded ASAP so we can make sure it all works before the downtime announcement goes out.  Will try to get this done later tonight.

Comment 8

[Dave Miller [:justdave]]

Tinderbox got updated tonight, Bonsai didn't because bug 52573 is incomplete, and not deployable without a working Makefile.  I attempted to cook one up this afternoon, but didn't finish before the window started.

Comment 9

[Dave Miller [:justdave]]

OK, I have a patch on bug 52573, and bonsai is deployed with that patch on
http://bonsai-stage.mozilla.org/ ,
http://bonsai-l10n-stage.mozilla.org/ , and
http://bonsai-www-stage.mozilla.org/

There seem to be some minor issues at the moment, trying to hash those out on #mozwebtools.

Comment 10

[(not currently active) Ted Mielczarek]

tinderbox already got upgraded, updating summary.

Comment 11

[Dave Miller [:justdave]]

ok, the issues have been hashed.  Turned out to be missing prereqs on the staging box.  Bug 404832 filed for that.

So unless anyone finds any other issues, I think this is ready to push to production.

Comment 12

[Dave Miller [:justdave]]

Couldn't get the mail handling code working properly, reverted back to the old version.  (see bug 53573 for details).  We have no way to test the mail handling on the staging system (would require an alternate cvs repo and a database separate from production), but I was under the impression that it was already in use on the bonsai system maintained by the guy who wrote the new code and naively assumed it would work... (it's almost the same code in use on tinderbox, which we already know works)

Comment 13

[Dave Miller [:justdave]]

OK, so here's what we need to do to get this moving again...

1) clone the bonsai database (say, bonsai_test or something)
2) have bonsai-stage talk to that database instead of the production one
3) split the mail at the production bonsai (dm-webtools02) so it delivers both locally and to bonsai-stage (dm-webtools03).  See tinderbox's aliases for examples, tinderbox-stage is already set up this way.

Once the above is done, we can upgrade bonsai-stage and freely debug it there.

Ideally we should put a fake cvs repo on it, too, but we can always find some obscure part of the tree that nobody cares if we screw with and check in a bunch of whitespace changes to test.

Comment 14

[Dave Miller [:justdave]]

This isn't going anywhere for a while, obviously, no reason to have it cluttering the main queue.

Comment 15

[Reed Loden [:reed]]

(In reply to comment #13)
> OK, so here's what we need to do to get this moving again...
> 
> 1) clone the bonsai database (say, bonsai_test or something)
> 2) have bonsai-stage talk to that database instead of the production one
> 3) split the mail at the production bonsai (dm-webtools02) so it delivers both
> locally and to bonsai-stage (dm-webtools03).  See tinderbox's aliases for
> examples, tinderbox-stage is already set up this way.
> 
> Once the above is done, we can upgrade bonsai-stage and freely debug it there.

All of the above things have been completed. I've also enabled $debug in processMail.pl.

Comment 16

[Reed Loden [:reed]]

I believe I have figured out the problem. I've attached a patch in bug 52573.

Comment 17

[Reed Loden [:reed]]

Bug 52573 is fixed. This is ready to go. Just need a downtime window.

Comment 18

[Reed Loden [:reed]]

This was completed earlier tonight. Please file bugs for any issues that may come up as a result of this much overdue upgrade.