1580952 - Snapshot/Image of Ccure server

Status: RESOLVED FIXED

(Infrastructure & Operations :: Virtualization, task)

Description

[Jennifer Hayashi [:jen]]

Hi - We're planning to upgrade the Ccure server on 9/27. I was wondering what my options would be to backup or take a copy of the changes before I start in case I need to roll back to before we attempted to upgrade?

Comment 1

[Chris Knowles [:cknowles]]

What's the name of the server in question? (looking for "ccure" in the VC, I see a "ccure-dev" box, but I don't think that's your prod box)

We can take a snapshot of a VM - when it's powered down. And then that snapshot gives us a point to return to in case of issues. This snapshot should be taken as close to the start of the changes as you're comfortable returning to. (So, if it's a low churn box, a day might be sufficient, but if there's lots of state change and things, then an hour might be all you'd be willing to lose if we have to revert.)

So, let me know the VM we're working with, and any specific times you'd like to bring the box down and snapshot it, and I would be happy to work with you.

Comment 2

[Jennifer Hayashi [:jen]]

Hi Chris -

The server is: officebadging1.doorsec.mdc1.mozilla.com. I'm going to ask that the security stop making changes about 4 hours before we start. If we could take down the box and take a snapshot around 4:45amPT/7:45am ET that would cool. I wanted to start the upgrade at 5:00am PT/8:00 ET. But can do it earlier if that's easier. Also, not sure how long you need to take the snapshot -- or if I need to let you know everything was ok so you can apply any cached changes. or you'll just delete the image. Please let me know and I'll build it into my plan.

Comment 3

[Greg Cox [:gcox]]

Snapshots are on the order of "seconds" for our work, plus the time it takes the VM to stop and boot back up (minutes). So, call it "5 minutes" and you'll probably be fine.
If you have to roll back, you'll want to let us know when you want us to revert the VM to that snapshot.
If you land the change successfully, we'll want to know when we can delete the snapshot. Seeing as you're planning for a Friday window, keeping it over the weekend or even longer is fine, but by Monday it's probably starting to get to the point where you wouldn't want to roll back, but that's me guessing and you're the service owner, so you'd know best on how much data you'd lose in a rollback scenario.

Comment 4

[Chris Knowles [:cknowles]]

"What Greg said, and..."

0745 ET on 9/27 works for me. Question - can we just shut the box gracefully, or do you need to do anything to shut the box down? (safe-ing databases, other things) if I can do it, I'll just do it at 0745 Eastern. Else I'll contact you and wait for your OK to do things.

And yes, the snapshot timing - it takes longer for me to type a meaningful title for the snap than for a powered off VM to take the snap.

I'll power it back on after the snap is successfully taken, and then you're clear for your window.

As to other things:
You mention "apply[ing] any cached changes"? I'm not sure what your expecting here? Snapshots are just a frozen image in time of the VM, if we need a new set of things frozen, that's a new snapshot, reverting to a snapshot returns the VM to its state at that moment in time. (Hence why powered off is so desirable - off VMs can't be in the middle of things to get confused) So you can't change a snapshot once it has been taken. (Though suddenly I want to look at doing things like this, and punning off of "photoshop" for the name - get it - manipulating snapshots?)

And as for deleting - Greg's point about "An old snapshot is a useless snapshot" important - there comes a time where reverting to an old state would be more damaging than fixing whatever is wrong. For many of our services that timeframe is measured in minutes or hours of production use - but some would happily revert after weeks. It's mainly a case of "If I revert to N days ago, I'm going to lose X data that happened in the intervening time" - if that thought is too painful - it's likely that the snapshot is of no more use to you.

Anyway - let me know about if I need to wait for you for the power down on that Friday. And any other questions you might have.

Comment 5

[Jennifer Hayashi [:jen]]

I'll need to shutdown the services gracefully so I'll see you at 7:45am ET. It is a little early for me, but that seems to be the sweet spot of being less impactful but also maximizing my support options. :)

As for the cached changes. Ignore me. I thought there was another option where you could sort of 'bookmark' when you started making changes so you could roll back to that time - or accept them and it applied all the changes. Sorry for the confusion. Let's just take the machine down, take the snapshot and I will definitely let you know whether we need it or can delete it.

Comment 6

[Chris Knowles [:cknowles]]

Ohh - a calendar entry - how fancy. :) I'll be there with bells on. (And I'm totally flexible - if moving it an hour later would make it so that you're not as exhausted - a major upgrade on too little rest can be no fun - or a couple hours earlier - if you just don't want to go to bed before the upgrade)

Comment 7

[Chris Knowles [:cknowles]]

Snapshot taken at 0745 - Per conversation with Jen, will pester about removal on Monday - unless they give me clearance before then.

Leaving this open until clean up.

Comment 8

[Chris Knowles [:cknowles]]

Got the all clear from Jen, have removed the snapshot.