Closed Bug 798509 Opened 7 years ago Closed 7 years ago

Align VMDKs in community servers

Categories

(SeaMonkey :: Release Engineering, defect, P4)

x86
All
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gcox, Assigned: Callek)

References

Details

Following up on our email, one at a time and in any order, I would like to take the following VMs down for about an hour each, to align the virtual disks for better performance of our storage systems.

nss-vm-win32-1.community.scl3.mozilla.com
nss-vm-win32-2.community.scl3.mozilla.com
sea-vm-linux32-1.community.scl3.mozilla.com
sea-vm-linux32-2.community.scl3.mozilla.com
sea-vm-linux32-3.community.scl3.mozilla.com
sea-vm-linux32-4.community.scl3.mozilla.com
sea-vm-linux32-5.community.scl3.mozilla.com
sea-vm-linux32-6.community.scl3.mozilla.com
sea-vm-win32-1.community.scl3.mozilla.com
sea-vm-win32-2.community.scl3.mozilla.com
sea-vm-win32-3.community.scl3.mozilla.com
sea-vm-win32-4.community.scl3.mozilla.com

Please let me know when I can have any of them to work on.  Thanks!
Product: mozilla.org → SeaMonkey
Version: other → unspecified
(In reply to Greg Cox [:gcox] from comment #0)
> Following up on our email, one at a time and in any order, I would like to
> take the following VMs down for about an hour each, 

"an hour each" can that be a handful done concurrent to still be an hour, or is it longer if we do concurrent?

> nss-vm-win32-1.community.scl3.mozilla.com
> nss-vm-win32-2.community.scl3.mozilla.com

These actually should be coordinated with :kai, sorry that I missed the distinction in my e-mail. CC'ing him so we can coord that here, or if he wants a seperate bug for that.

> Please let me know when I can have any of them to work on.  Thanks!

For the sea-* set, how's thur/friday next week in 2 batches, set a time in the day[s] for me to deal with it.
I can't go concurrent (they're disk intensive) but I can go serially.  I.e., if if you want to give me X boxes and X the amount of time, I can go with that (and I can boot servers back as they finish).  Or we can do one at a time and tag in and out.  Whatever is easiest for you.

I'm on eastern US time, so I can take them as soon as you're willing to down them on those days.  Mornings are awesome.
I didn't even know we have a nss-vm-win32-2.community.scl3.mozilla.com ! :)
                                         =^=

Please feel free to take down 
    nss-vm-win32-1.community.scl3.mozilla.com
whenever it fits your schedule.

A downtime of 1-3 hours isn't a problem.
Kal:
I did nss-vm-win32-1 on Friday night, thank you.
Where I said nss-vm-win32-2, it's apparently just what it was called on the disk; it's actually cal-vm-win32-1 that is the hostname that needs work.
(In reply to Greg Cox [:gcox] from comment #4)
> Kal:
> I did nss-vm-win32-1 on Friday night, thank you.
> Where I said nss-vm-win32-2, it's apparently just what it was called on the
> disk; it's actually cal-vm-win32-1 that is the hostname that needs work.

In which case that's :Fallen that needs to be pinged
Any time is fine, go ahead.
cal-vm-win32-1 aligned and put back in service, thanks.  That just leaves the sea-vm-* batch for this bug.
Hey Greg, due to the need to spin a chemspill, if at all possible can we defer this downtime until sometime next week. I don't mind the days, and am willing to let you do it all in one day with an extended Downtime of our hosts.

I apologize for the delay in telling you, if its not easily possible or if this comes with a higher urgency we can still do the downtime, but will be a bit more coordination needed as I'll take out a handful of machines, but smaller "batches" (basically 4 batches across these two days) to account for the chemspill.
Nono, I /want/ to singlehandedly introduce changes during a chemspill!

Righto.  Deferred it is.
Per an IRC discussion, let's shoot for Thursday and Friday.  EDT morning is best for me, as the light load makes for a little bit better speeds.

Whatever you want to give me, in whatever order.  Just halt them and let me know.
(In reply to Greg Cox [:gcox] from comment #0)
> sea-vm-linux32-1.community.scl3.mozilla.com
> sea-vm-linux32-2.community.scl3.mozilla.com
> sea-vm-linux32-3.community.scl3.mozilla.com
> sea-vm-linux32-4.community.scl3.mozilla.com

Did a handoff of these 4 just now. |/sbin/shutdown -h now|

> sea-vm-win32-1.community.scl3.mozilla.com
> sea-vm-win32-2.community.scl3.mozilla.com
> sea-vm-win32-3.community.scl3.mozilla.com
> sea-vm-win32-4.community.scl3.mozilla.com

These can also be done today, but are harder to takedown from my end, given that its an RDP needed to really get them gracefully shut off (since they reboot after every job)  sea-vm-win32-1 was having issues yesterday it seems, where every step (even the simple ones) was failing with an esoteric return code, so might be bad.

I'd say go ahead and tackle the win ones when you're done with the linux/ready, Windows are currently up, so will need a shutdown.

We'll graceful/do the last 2 linux ones tomorrow, I'll get them shutdown tonight before I head to bed, so you can do them when you get up.
Linux 1-4 back up.  Didn't get to the windows ones.
Anything you can shut down for me for tomorrow, or bless me to take down on my own, we'll get to on Friday.
Windows 1-4 and linux 5-6 done today with IRC coordination.
That's everything listed on this bug, so, closing out.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.