Last Comment Bug 491767 - Migrate some VM off equallogic before the firmware upgrade, then move them back
: Migrate some VM off equallogic before the firmware upgrade, then move them back
Status: RESOLVED FIXED
needs-scheduling for moving them back
:
Product: mozilla.org Graveyard
Classification: Graveyard
Component: Server Operations (show other bugs)
: other
: All All
: -- normal (vote)
: ---
Assigned To: Phong Tran [:phong]
: matthew zeier [:mrz]
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-05-06 16:02 PDT by John O'Duinn [:joduinn] (please use "needinfo?" flag)
Modified: 2015-03-12 08:17 PDT (History)
6 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description John O'Duinn [:joduinn] (please use "needinfo?" flag) 2009-05-06 16:02:09 PDT
From meetings and email threads this week, it looks like the following list of VMs need to be moved before next week's firmware upgrade on equal-logic arrays.

production-master 
try-master
try-win32-slave01
try-win32-slave02
try-win32-slave03
 

If it turns out there is space to migrate a couple of additional try slaves, let us know and we'll add them to the list. These additional slaves would make easier for sheriffs on the day, but are "nice to have", not "must have".
Comment 1 Nick Thomas [:nthomas] 2009-05-06 16:20:21 PDT
I thought that people were OK with no try server for the four hours of the eql outage.
Comment 2 matthew zeier [:mrz] 2009-05-07 09:29:26 PDT
VMs with storage requirements:

production-master      60GB
try-master             30GB
try-win32-slave01      60GB
try-win32-slave02      60GB
try-win32-slave03      60GB

geodns01               10GB
pm-ns03                10GB

About 300GB needed and my quick spot check looks like there's enough NetApp space to handle this.
Comment 3 John O'Duinn [:joduinn] (please use "needinfo?" flag) 2009-05-07 10:22:41 PDT
(In reply to comment #1)
> I thought that people were OK with no try server for the four hours of the eql
> outage.

Yep, but when I saw that moving just a few VMs would give us some (tiny) TryServer, and make a big difference to the sheriffs that day, it seemed worth trying for (bad pun, sorry). 

If we can get any TryServer going thats great. Worst case, we'll go ahead with the downtime taking down TryServer. Make sense?
Comment 4 John O'Duinn [:joduinn] (please use "needinfo?" flag) 2009-05-07 10:26:20 PDT
(In reply to comment #4)
> From meetings and email threads this week, it looks like the following list of
> VMs need to be moved before next week's firmware upgrade on equal-logic arrays.
> 
> production-master 
> try-master
> try-win32-slave01
> try-win32-slave02
> try-win32-slave03
> 
mrz: is there room to migrate these also?

graphs.m.o
build.m.o
anonymous hg.m.o
Comment 5 matthew zeier [:mrz] 2009-05-07 12:04:53 PDT
graphs.m.o -> dm-graphs01   30GB


build.m.o -> dm-wwwbuild01  80GB

anonymous hg.m.o 
* dm-vcview01   10GB
* dm-vcview02   10GB


So another 130GB.  That might be tight (the build NetApp has more space than the IT ones).  I'll see what we can do.
Comment 6 matthew zeier [:mrz] 2009-05-07 14:00:10 PDT
 
> anonymous hg.m.o 
> * dm-vcview01   10GB
> * dm-vcview02   10GB


Not needed to move - dm-vcview03 is physical hardware and can handle load during the window.
Comment 7 John O'Duinn [:joduinn] (please use "needinfo?" flag) 2009-05-11 10:54:57 PDT
(In reply to comment #2)
> VMs with storage requirements:
> 
> production-master      60GB
> try-master             30GB
> try-win32-slave01      60GB
> try-win32-slave02      60GB
> try-win32-slave03      60GB

Most of these look like they were migrated off in advance of downtime. However, "try-master" is still on eq01-bm01... can you migrate that also?
Comment 8 Phong Tran [:phong] 2009-05-11 13:18:48 PDT
it looks like only the 10GB drive needs to be migrated off.  this should be a quick migration.
Comment 9 Nick Thomas [:nthomas] 2009-05-11 19:29:59 PDT
(In reply to comment #8)
> it looks like only the 10GB drive needs to be migrated off.  this should be a
> quick migration.

This turned out to hit an error (got any more deets Phong ?) and the disks ended up back on eql-bm01. The buildbot process then went a bit nuts CPU-load-wise, so I took the opportunity to shutdown the machine and migrate the storage to d-sata-build-003. Two try runs that were interrupted, and two that occurred during the migration, were resubmitted.

All done here ?
Comment 10 matthew zeier [:mrz] 2009-05-11 20:01:42 PDT
Probably but I want to keep this around for tracking to eventually move the VMs back.  We grabbed a lot of temporary SAN space.
Comment 11 matthew zeier [:mrz] 2009-07-02 14:27:17 PDT
Need to plan on moving this back...
Comment 12 John O'Duinn [:joduinn] (please use "needinfo?" flag) 2009-07-09 01:50:05 PDT
(In reply to comment #11)
> Need to plan on moving this back...

Probably too late to include in our downtime tmrw (Thurs) morning. Could we do this in another downtime maybe next week?
Comment 13 matthew zeier [:mrz] 2009-08-06 21:02:52 PDT
RelEng, please let me know when we can schedule this and toss back when it can get in the windows.
Comment 14 Chris Cooper [:coop] 2009-08-10 08:44:06 PDT
Re-assigning to joduinn for scheduling.
Comment 15 Ben Hearsum (:bhearsum) 2009-08-10 12:48:36 PDT
We've got a downtime tomorrow and I was to do this during it. It's hard to decode exactly what needs to be done, though. Which VMs need to be moved? Which datastores do I move them to?
Comment 16 Phong Tran [:phong] 2009-08-10 14:17:29 PDT
production-master      60GB
try-master             30GB
try-win32-slave01      60GB
try-win32-slave02      60GB
try-win32-slave03      60GB

499G  224G  275G  44% /vmfs/volumes/eql02-bm03
499G  306G  193G  61% /vmfs/volumes/eql01-bm12
Those 2 datastores have some free space.

on the IT Cluster:
graphs.m.o -> dm-graphs01   30GB
build.m.o -> dm-wwwbuild01  80GB
Comment 17 Ben Hearsum (:bhearsum) 2009-08-11 07:00:11 PDT
(In reply to comment #16)
> production-master      60GB
> try-master             30GB
> try-win32-slave01      60GB
> try-win32-slave02      60GB
> try-win32-slave03      60GB
> 

I migrated all of these in the downtime today.
Comment 18 Ben Hearsum (:bhearsum) 2009-08-12 06:48:47 PDT
Phong, is there anything else to do here?
Comment 19 Phong Tran [:phong] 2009-08-12 10:42:59 PDT
When can we migrate the other 2?

on the IT Cluster:
graphs.m.o -> dm-graphs01   30GB
build.m.o -> dm-wwwbuild01  80GB
Comment 20 Ben Hearsum (:bhearsum) 2009-08-13 09:47:58 PDT
(In reply to comment #19)
> When can we migrate the other 2?
> 
> on the IT Cluster:
> graphs.m.o -> dm-graphs01   30GB
> build.m.o -> dm-wwwbuild01  80GB

Let's look to do this in a downtime next week. I'll let you know on Monday exactly when.
Comment 21 Ben Hearsum (:bhearsum) 2009-08-17 13:36:35 PDT
We're going to have downtime to do this tomorrow morning. Starting anytime between 5am and 7am is fine. Phong, whatever works best for you in the window sounds good to me.
Comment 22 Phong Tran [:phong] 2009-08-18 06:06:44 PDT
dm-graphs01 migrated.
Comment 23 Phong Tran [:phong] 2009-08-18 07:50:41 PDT
dm-wwwbuild01 also migrated.  I think we are all done here.

Note You need to log in before you can comment on or make changes to this bug.