726710 - build new production bugzilla instance in scl3

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

The plan to handle bugzilla moving out of sjc1 is to build a new instance in scl3 and then decommission the sjc1 instance while the phx1 instance is active.

As for the other hardware:

api-dev may need to move, if :dkl doesn't finish the replacement for BzAPI in time.  David, can you link that bug here?

pm-adm-bugs01 (a VM) will need to move.

Once the instance in scl3 is created, the following hosts are spare

    dm-bugstage01
    dm-bugstage02
    dm-bugstage03
    pm-app-bugs03
    pm-app-bugs04
    pm-app-bugs05
    tm-bugs01-master01
    tm-bugs01-sanitarium
    tm-bugs01-slave01
    tm-bugs01-slave02
    tm-bugs01-slave03
    tm-bugstage01-master01
    tm-bugstage01-slave01

and should be marked as such per https://mana.mozilla.org/wiki/display/websites/Killing+sjc1+hosts

Comment 1

[David Lawrence [:dkl]]

Will we (BMO devels) be able to utilize any of these for use as a development staging server once they are available? This may be related to bug 716641.

dkl

Comment 2

[Dave Miller [:justdave]]

This is actually two separate clusters listed here, which can (and should) be moved independently.

The following is the staging cluster:
>    dm-bugstage01
>    dm-bugstage02
>    dm-bugstage03
>    tm-bugstage01-master01
>    tm-bugstage01-slave01

And appears to be already covered by bug 658743 (the original intention was to move these to phx1, which was already in progress and stalled prior to starting the scl3 move proceedings)

The following is the production cluster:
>    pm-app-bugs03
>    pm-app-bugs04
>    pm-app-bugs05
>    tm-bugs01-master01
>    tm-bugs01-sanitarium
>    tm-bugs01-slave01
>    tm-bugs01-slave02
>    tm-bugs01-slave03

And should be directly dealt with on this bug.

tm-bugs01-sanitarium has nothing time-critical on it at the moment.  It has cron jobs that do data crunching, and the automated ones have been busted for a few months and nobody complained.  We can live without it for a few days if the box is physically moved.  On the other hand, it's on crap hardware and I'd just as soon replace it anyway if the option is available.

Comment 3

[Ashish Vijayaram [:ashish]]

Please don't leave behind pm-adm-bugs01, which is a VM :)

Comment 4

[Dave Miller [:justdave]]

Ah yes, that goes with the production cluster, too.  That might be easier to rebuild than move.  It currently functions solely as an outbound mail relay for the cluster.  It really should take over all of the functions mradm02 is currently performing (cron jobs and file distribution to the webheads).  Then again, that's probably another bug.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

pm-adm-bugs01 needs to be out of sjc1 by 4/9 - is that a rebuild, then, or a move?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

api-dev, aka dm-bugzilla-api02, aka bugzilla-api01 (see https://bugzilla.mozilla.org/show_bug.cgi?id=722795#c10) needs to move as well, unless :dkl's work on BzAPI has made great progress.

Comment 7

[David Lawrence [:dkl]]

(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> api-dev, aka dm-bugzilla-api02, aka bugzilla-api01 (see
> https://bugzilla.mozilla.org/show_bug.cgi?id=722795#c10) needs to move as
> well, unless :dkl's work on BzAPI has made great progress.

It will not be complete before the planned move date so please keep those services up.

dkl

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

api-dev/dm-bugzilla-api02/bugzilla-api01 will move in bug 741379.

Comment 9

[Shyam Mani [:fox2mike]]

Does not block scl3, bugzilla stuff will be rebuilt from scratch in scl3 on RHEL6.

Comment 10

[Shyam Mani [:fox2mike]]

Dev services Q3 goal

Comment 11

[Shyam Mani [:fox2mike]]

Moving this to Q4 as bug 780820 is not going to be fixed in time for us to have enough testing + move time this quarter.

Comment 12

[Mark Côté [:mcote]]

This was scheduled to be completed this week. Where are we on it?

Comment 13

[Shyam Mani [:fox2mike]]

Mark,

The work on the Bugzilla Push stuff + my travel took away some time away from this. I'll try to work on this and have it ready by next week.

Comment 14

[Shyam Mani [:fox2mike]]

This might not happen this quarter because of the b2g convergence window in the week of the proposed upgrade (9th to 14th December). This will probably be pushed to Q1, 2013.

Comment 15

[cmtalbert]

So, we are planning to release 4.2 the first week in February. This needs to be completed and ready to go before that week. What is its ETA and when can we expect these machines to be live and ready for use?

Comment 16

[Mark Côté [:mcote]]

Need an ETA on this to plan for the release.  It will need to be up at least a few days before we launch.  Can we get a firm date for this, as we have for the staging cluster?

Comment 17

[Shyam Mani [:fox2mike]]

(In reply to Mark Côté ( :mcote ) from comment #16)
> Need an ETA on this to plan for the release.  It will need to be up at least
> a few days before we launch.  Can we get a firm date for this, as we have
> for the staging cluster?

I was hoping to hit the same date, but that won't happen. I'm at LCA from Jan 25th through Feb 1st and I'll try to get it that week. Going to be on PTO from Feb 2nd through 10th and I really want to get it before that.

Comment 18

[Shyam Mani [:fox2mike]]

(In reply to Mark Côté ( :mcote ) from comment #16)
> Need an ETA on this to plan for the release.  It will need to be up at least
> a few days before we launch.  Can we get a firm date for this, as we have
> for the staging cluster?

I've sent email communication regarding firm dates and we'll proceed once you've all responded.

Comment 19

[Shyam Mani [:fox2mike]]

Bug 842269 filed with the CAB has more information on the proposed date/time for the move.

Comment 20

[Shyam Mani [:fox2mike]]

bugzilla.mozilla.org is now being served out of the new production instance in scl3.