Closed Bug 743815 Opened 12 years ago Closed 8 years ago

NetVault: Backup releng windows server infrastructure

Categories

(Infrastructure & Operations :: Infrastructure: Backups, task)

x86_64
Windows 7
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mlarrain, Unassigned)

References

()

Details

We need to look into what is needed for backing up Windows Domain Controllers as they are very picky about being backed up.
Assignee: server-ops-releng → mlarrain
So I was doing some research last night and came across this; http://www.trainsignal.com/blog/backup-and-restore-active-directory-on-windows-server-2008
Component: Server Operations: RelEng → Server Operations: Infrastructure
QA Contact: arich → jdow
We use NetVault for our Linux servers, but it can also backup Windows machines.
They currently have plugins for SQL server (Supported Editions: Developer, 2000 MSDE, 2005/2008 Express, Workgroup, Standard, Enterprise and Web Edition.) and Exchange (Supported Editions: Standard and Enterprise).

What details do you need before trying those out? I can shoot them an email and ask more about those plugins.
Assignee: mlarrain → dgherman
Ask them if it is possible to backup and active directory database/server and how easy it is to deploy from backup.
bumping on this as this is a priority for the releng windows rollover that is happening on Friday. If nothing else we need a system backup of these systems.
Severity: normal → major
Per our IRC conversation, I need to know the following:
- AD version that we use
- Windows flavor

Thanks!
dumitru we are using Windows Server 2008 R2 and AD Version: 6.1.7601.17514
Case ID : 1056331 with Quest created.
Quest's reply was long, so I forwarded it to digipengi.
Waiting for his input after reading the options we have.
Agreed with Dumitru the email was really long but we are going to go with using System State backups. I will perform the initial run through and post a wiki page on how to do this in the future.
dumitru do you have a server path for backups I can use?
per our discussion, this is handled differently.
Assignee: dgherman → mlarrain
The goal now is to switch from windows dhcp to ISCDHCP and thus we will not have to backup these systems as they are setup for redundancy. Closing this bug as wont fix.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WONTFIX
Assignee: mlarrain → q
Component: Server Operations: Infrastructure → Infrastructure: Other
Product: mozilla.org → Infrastructure & Operations
Reopening this since it has some prior discussion about how we might go about this. Q and I discussed revisiting this issue to back up at least one domain controller, wds server, and probably the kms server so we have a way to recover the information in the case of corruption or site-wide loss.

I'll let Q provide the technical details of what we should back up and how.
Status: RESOLVED → REOPENED
Flags: needinfo?(q)
Resolution: WONTFIX → ---
Summary: Figure out how to backup Windows Domain Controllers → Backup releng windows server infrastructure
Most data in the relops windows infrastructure has DFS redundancy so if we back up the whole file system we get all the shared data. 

WDS1.releng.ad.mozilla.com
 * E:\  nightly or weekly if space is a concern
 * Whole system monthly for IDR

DC1.ad.mozilla.com
 * C:\ nightly
 * Active directory ( global catalog or possibly ldap client ) nightly 

DC6.releng.ad.mozilla.com
 * C:\ nightly
 * Active directory ( global catalog or possibly ldap client ) nightly 

KMS1.ad.mozilla.com
 * Whole system weekly
all machines are 2008 R2 unless otherwise specified.
Flags: needinfo?(q)
How's the backup process going to look like? I'm asking because I need to filter out the traffic it's going to generate from my NSM system, so it does not overload it.
We'll use Netvault, so same port range (20031-21031 (TCP and UDP)).
Its been over a month now, so I'm checking back in to see when we'll have this implemented.  This is fairly important since the windows infrastructure has no backups at all at this point, and if we were to suffer a catastrophic failure, we would need to rebuild things from scratch by hand.
Assignee: q → dgherman
Flags: needinfo?(dgherman)
Usul can take care of this.
As you see, the bug was not assigned to me nor a needinfo was requested, so I had no idea I need to take action on it.
Assignee: dgherman → ludovic
Flags: needinfo?(dgherman)
Are the machines accessible via RDP ?
Status: REOPENED → ASSIGNED
Flags: needinfo?(q)
(In reply to Ludovic Hirlimann [:Usul] from comment #20)
> Are the machines accessible via RDP ?

?
Flags: needinfo?(arich)
Yes, they're all accessible via RDP. Credentials are in the relops GPG password file.
Flags: needinfo?(q)
Flags: needinfo?(arich)
Client is installed on :
WDS1.releng.ad.mozilla.com
DC6.releng.ad.mozilla.com
KMS1.ad.mozilla.com

Couldn't connect on DC1.ad.mozilla.com
usul: were you able to get into dc1.ad.mozilla.com once you tried using the correct credentials?
(In reply to Amy Rich [:arich] [:arr] from comment #24)
> usul: were you able to get into dc1.ad.mozilla.com once you tried using the
> correct credentials?

Nope but I'll give it one more try today - was busy with other backup stuff.
> WDS1.releng.ad.mozilla.com
>  * E:\  nightly or weekly if space is a concern

First full for that is currently running. - running nightlies on it.
Rah and today I can't manage to connect to any using the password in the file. Did it change and the file didn't get updated ?
(In reply to qfortier from comment #14)

> KMS1.ad.mozilla.com
>  * Whole system weekly

Weekly that we keep for a month - done evry saturday at 5PM
(In reply to Amy Rich [:arich] [:arr] from comment #24)
> usul: were you able to get into dc1.ad.mozilla.com once you tried using the
> correct credentials?

done with my account - software is installed.
Are all of these machines being backed up now, or is there more work to do?
Flags: needinfo?(ludovic)
Still need to do the AD I'll close the bug when things work properly.
Flags: needinfo?(ludovic)
Any update here, it's been almost another 3 weeks and we're still without backups on some of these systems.
nope. Work week this week so don't expcet anything until monday. sorry arr.
Update :

Status update (sorry it took so long)
Reading the docs for AD and making sure I understand them to have proper backup of AD.

Looking at backups I have WDS1, DC1 and KMS1 seen as client.

KMS1 has weekly jobs running.
WDS1 as E:\ as Full and INcs running.

Added C:\ Nighthlies on DC1.

I currently can't see DC6 from backup1.
> I currently can't see DC6 from backup1.

so I get : 
[root@backup1.private.scl3 ~]# nc -z 10.22.69.18 20031
[root@backup1.private.scl3 ~]#

and I can ping backup1 from DC6.

[root@backup1.private.scl3 ~]# nc -z 10.22.69.16 20031
Connection to 10.22.69.16 20031 port [tcp/*] succeeded! is what I get for DC1.

So either something is wrong on the network or with the install
Q did you set any specific local firewall rules on DC6 ? eg I need ports 20031 - 21031/tcp to be open to talk to backup1.private.scl3.mozilla.com ?
Flags: needinfo?(q)
The outbound ports should be ooen. I am testing some gpo based exception rules to white list the needed ports.  I shoukd have it rolling out tomorrow.
Flags: needinfo?(q)
(In reply to Q from comment #37)
> The outbound ports should be ooen. I am testing some gpo based exception
> rules to white list the needed ports.  I shoukd have it rolling out tomorrow.

so how did the rolling out go ?
TCP ports 20031 - 21031 are now listed as open inbound and outbound explicitly to any internal address.
It's been another month...  Is this working now?
Flags: needinfo?(ludovic)
DC6 is now visible.
Flags: needinfo?(ludovic)
(In reply to Ludovic Hirlimann [:Usul] from comment #41)
> DC6 is now visible.

Added the nighly job.
Hey Usul,

As discussed, please note the timeframe and requirements in the bug.

Thank you.
Flags: needinfo?(ludovic)
Need to kick my ass over this. Will try to finish this by the end of this month. Will probably need to Windowns sysadmin help to test if restores are fine as my AD knowledge is close to null.
Flags: needinfo?(ludovic)
Hey, it's after the end of the month.  Checking in to see if there's been any progress.
(In reply to Amy Rich [:arich] [:arr] from comment #45)
> Hey, it's after the end of the month.  Checking in to see if there's been
> any progress.

I'll take the stick and punch me with it - sorry - What I really need to do is read a kb article and make sure I understand it fully. Have not been able to do so - It's still very high on my list (promise).
Ok need to look at the options I've ticket for the backup and VSS/OFM means in the netvault vocabulary.
Needinfo me so I really don't forget.
Flags: needinfo?(ludovic)
So we do full backups of the host. Using a client's version greater than 6.17 as per the docs.
I don't see any of the options from the kb article in the UI I get - ain't sure where to go with this (as I can't currently open support tickets).
Flags: needinfo?(ludovic)
Any luck getting our support contract renewed?
Flags: needinfo?(ludovic)
:Usul, any update on this, in terms of support contract renewal, and getting this stuff backed up?
(In reply to Dan Parsons [:lerxst] from comment #51)
> :Usul, any update on this, in terms of support contract renewal, and getting
> this stuff backed up?

Status is :
 Machine drives are backed up.
 Ain't sure about the Active directory bits.
For the licence renewals I'll defer to dmoore.
Flags: needinfo?(ludovic) → needinfo?(dmoore)
Summary: Backup releng windows server infrastructure → NetVault: Backup releng windows server infrastructure
Assignee: ludovic → infra
removin the needinfo as we won't renew licences
Flags: needinfo?(dmoore)
Severity: major → normal
Status: ASSIGNED → NEW
Component: Infrastructure: Other → Infrastructure: Backups
These systems are being backed up via bacula.
Status: NEW → RESOLVED
Closed: 12 years ago8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.