Closed Bug 753011 Opened 14 years ago Closed 14 years ago

cg-vmware01 unplanned move to scl3

Categories

(Infrastructure & Operations :: Virtualization, task, P1)

Product:
Infrastructure & Operations
Component:
Virtualization
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: wicked, Assigned: dparsons)

Details

Looks like landfill.bugzilla.org (a VM on cg-vmware01) is not responding at the moment. We'd need to get it back up ASAP since that's hosting our update XML that tells all Bugzilla Installations all over the world what our current versions are.
This was on cg-vmware01, which was incorrectly put on a move truck. I *think* this used local storage on the blade. We're working out the details.
Host was inadvertently put on today's scheduled data center move before the VMs had been migrated off. We're prioritized getting this hardware back online in the new location and will work quickly to move the VMs off this hardware and onto the new ESX cluster.
I'm broadening this out a little. This host had two necessary VMs on it: * landfill.buzilla.org (per comment 0) * cg-centos01 (see bug 720544) Landfill was still running production services in sjc1. We may be able to transition one of those (updates) to an existing host in scl3, in bug 753019, which would take a bit of the firedrill edge off. cg-centos01 wasn't quite evacuated yet, and reed will need access again, but it was not running production services so he can have that restored post-firedrill. Even then, he only needs it long enough for an rsync, so if leaving the VM on cg-vmware01 is easier than migrating to ESX, let's do that.
Summary: landfill.bugzilla.org is offline and with it important Bugzilla update snippet information → cg-vmware01 unplanned move to scl3
(and I said this in IRC, but so it's in writing: cb-vmware01 only had cb-jump01 left, so at this point it's completely unnecessary)
FWIW, we (Bugzilla Project) consider having our updates XML serving offline a blocking issue for us, see bug 705421 for details what happens when it's down. I was planning to get it moved to it's new host without any downtime before we migrate landfill, so this is quite unfortunate surprise.
Looks like bug 753019 is going to get done. Please, dustin, could you point updates.bugzilla.org to 63.245.223.29 in the DNS, thanks!
DNS change is done.
Landfill is recovering now. ETA was 9h while running in parallel, but will hopefully be shorter when it's the only thing transferring. Dan, when that's done, we'll need to get cg-centos01 up enough to rsync off of it -- on the prod cluster or straight off cg-vmware01, whichever is easier for you. The other hosts on cg-vmware01 and cb-vmeware01 are completely unnecessary.
New estimate is 3h for landfill.
OK, landfill and cg-centos01 are up. I'll get them reconfigured soon.
They're reconfigured. Let's call this closed, and follow up on bug 719715 and bug 720544.
Component: Server Operations → Server Operations: Virtualization
QA Contact: phong → dparsons
Assignee: server-ops → dparsons
Closing as per last comment.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.