789352 - initiate backups for git->hg conversion machines

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[u429623]

per discussions, initiate backups of local disk on:
 github-sync1.dmz.scl3.mozilla.com
 github-sync1-dev.dmz.scl3.mozilla.com

Note: currently, not much is on the local drives due to small size. For this project, 2 key repositories will be on the SSD (i.e. the size will jump soonish)

Comment 1

[Shyam Mani [:fox2mike]]

Ops,

Backup up everything on these machines. Thanks.

Comment 2

[Dumitru Gherman [:dumitru]]

Hal said to backup "local disk", but Shyam said "everything".
So I'm confused, since I see a NFS share there.
[[e-mail]3 ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              66G   11G   53G  17% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/sda1             985M   64M  871M   7% /boot
10.22.74.11:/vol/github_sync
                      450G   76G  375G  17% /mnt/netapp/github_sync

Please tell me exactly what needs to be backed up, thanks!

Comment 3

[Shyam Mani [:fox2mike]]

Dumitru,

I thought Netapp has it's own backup mechanism? Let's discuss on IRC? Thanks!

Comment 4

[Dumitru Gherman [:dumitru]]

Done.
https://mana.mozilla.org/wiki/display/SYSADMIN/git-%3Ehg+conversion+machines

Comment 5

[u429623]

Reopening for clarification - the mana docs do not show that /home/hwine is being backed up, and that is the crucial data on the conversion machines.

Please add that to the backup, or clarify in mana that it is being backed up.

(Note: the new account requested in bug 791169 will also need it's /home directory backed up)

Comment 6

[Dumitru Gherman [:dumitru]]

Why is crucial stuff in your home directory?
We've seen situations when people used to run production stuff out of their home directories on servers, and when they left the company and the account got closed, disaster happened.
It is not a good practice at all to have crucial data in your home directory. Is this in puppet at least? Can it be moved somewhere else?
We usually do not backup users home directories for those reasons...we have some exceptions, of course, but for a system that important like this one...I don't think that's the best approach.

Comment 7

[Justin Dow [:jabba]]

Please do not run anything important from your home directory. This is unsupportable from many aspects.

Comment 8

[u429623]

I agree 1,000%, and we're moving there.

However, we got pushed "live" ahead of time to backfill a failed system and satisfy an urgent business need.

This is purely a temporary stopgap until a more proper production deployment can be done. But, we do need that stopgap - recreating the data takes over 6 days of CPU time, and would be developer outage for that duration.

Comment 9

[Justin Dow [:jabba]]

Instead of our temporary stopgap, why not just mv /home/hwine /data/gitconversion or something? What's the timeline for the proper production deployment? What will the migration look like? There is no such thing as "temporary" in the world of IT. It is unsupportable to have stuff in a user's home directory from many aspects. The first aspect is: we don't backup /home.

Why was this allowed into production with stuff in your home directory? I don't think it is fair for us to have to have to change our policies and procedures for this because you allowed this to go into production in your home directory.

Comment 10

[Justin Dow [:jabba]]

If it is absolutely impossible to change this right now. I can offer a 2-week deadline where we will back up /home/hwine on these machines. But this is a hard deadline and we will be removing the backups 2 weeks from today. Let me know if this is something that would help get this unblocked today.

Comment 11

[matthew zeier [:mrz]]

To provide some context - 

Backing up /home creates a one-off that's hard to maintain.  We've also had a number of problems with content living in /home/$user when $user is no longer at Mozilla.  Becomes a scramble so we're highly motivated to avoid that.

But we're all in agreement.  Hoping two weeks gets you enough time to cp/mv  your data and fix up whatever scripts/processes use /home/hwine.

Comment 12

[Dumitru Gherman [:dumitru]]

I added /home/hwine for being backed up from the servers mentioned in comment 0.
Keeping the bug open to remove this in two weeks.

Comment 13

[u429623]

Thanks to all for working through this policy exception. 

Bug 791291 has been opened to track the remaining work of removing the backup, as agreed to in #infra with :dumitru., and the whiteboard marker moved there.