624283 - Subversion repository for big files deployed by releng puppet

Status: RESOLVED WONTFIX

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

We currently have a big NFS directory at
 10.2.71.136:/export/buildlogs/puppet-files/production
containing a lot of files that shouldn't be in hg because
 1. they're huge; and/or
 2. they're secret; and/or
 3. they're licensed content we can't make publicly available

Most of those files are small text-ish config files that should move into our puppet manifests in the form of modules.  However, the rest need to go somewhere - somewhere private.

fox2mike mentioned that RPMs should be distributed via local repositories.  This is true, but I don't think we have a parallel infrastructure set up for Darwin, W2K3, Win7, and Debian.  Given this heterogeneity, I'd prefer to keep all of our packages distributed the same way.  At any rate, some of these files aren't packages at all, so we need somewhere to store them.

The summary mentions subversion, but I'm open to other options.  Advantages:
 * history is nice, as it ties people, bugs, and files together
 * svn doesn't download the entire history, saving space on clients
 * IT-hosted version control (presumably?) gets us backups for free

Comment 1

[Corey Shields [:cshields]]

Going to punt this over to zandr for review and comment.  (zandr: the issue at hand is overloading an nfs server and creating that dependency versus some other means of file distribution)

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

The issue as I understand it is that we're using NFS between datacenters and thus burning a lot of bandwidth.  I don't think the load on the NFS server itself from this use is terribly high.

Comment 3

[Zandr Milewski [:zandr]]

So, in my other life, I'm actively trying to keep people from using svn for big blobs.

I agree that the history aspects are nice, but if inter-DC bandwidth is the issue (and I'm guessing that means 650... we have fat pipes between scl1 and sjc1) then I think a mirror arrangement of some sort is going to be correct.

GlusterFS has been mentioned; all of Mozilla IT's experience is old, but wasn't very good. Doesn't sound like rsync+cron is wrong, though.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I missed a bullet point in comment #0:
 * dead-simple deployment ('svn up')

I'm working on a fleet of puppet-related pushes, and generally these bugs are a patch (for puppet-manifests) with a text file containing instructions for re-jiggering the puppet files, and occasionally a binary or two to toss in there.

Aside from being confusing, this can be difficult to roll out efficiently, and even more difficult to roll back.  In the deployment of the nrpe.cfg changes last night, I got bitten on some 20 slaves from having deployed the puppet-files changes at a different speed than the puppet-manifests.

Being able to commit a reviewed patch to hg and svn *before* updating repositories on each puppet server would make it a lot easier to roll such things out smoothly, and to roll back in the event of an error.  The 'svn log' is good, too, for figuring out which bug was involved in adding a particular package, for example.

Honestly, the inner developer in me recoils at the thought of the canonical form of part of our system configuration being a directory on some server.  It just seems so tenuous and easy to kill with a stray 'rm myfile<tab> *'..

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I feel stronger about this as the days go by.  I've come across a case in bug 624213 where some files in puppet-files are different from what they should be, at least on some masters.  Having a revision-control system would let me (a) see quickly if these files were changed from what they should be (svn status) and (b) find out if/why they were changed, if the change was checked in (svn log).  As it is, I'm reverting a change that someone may have made for a very good reason.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

And again, bogus files on a puppet master bring down a bunch of slaves: bug 635521.  Admittedly, these configuration files should be generated in a module, but the lack of versioning here is making it difficult to review such changes (as this one was obviously bogus) or find out when they were made after the fact.

Comment 7

[Amy Rich [:arr] [:arich]]

Based on several conversations over the months, the inherent issues of putting large binary files into any vcs, and the plans to set up repos, I'm going to close this bug out.

Comment 8

[John Hopkins (:jhopkins)]

FWIW, I have used the Perforce version control system for large files (several gigabytes), including a single commit of 50GB worth of data.  No issues.