769071 - abandon csync2^Wrdist for synchronizing /data

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

csync2 appears to fall over in some key circumstances -- initializing new masters and deleting directories being two of the most painful.

It seems to work OK for the cert syncing, since that's just adding files, but I'm particularly keen to remove it from syncing /data, where it's causing a lot of pain.

Some options suggested are:
 - rdist (which requires a single master, something we don't have now)
 - aerofs/s3, which syncs to S3 and back
 - dropbox
but I'm open to other ideas.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

http://lists.linbit.com/pipermail/csync2/2012-June/000887.html

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

File synchronization between the masters is broken right now, although they all match.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I disabled csync2, so at least we won't get spammed anymore.  I'm working on setting up rdist.

P.S. mrz wins a cookie - he suggested rdist

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Well, that was annoying -- there's a lot of setup required to get ssh working, get rdist working, make sure permissions work out, and so on.

It's also incredibly slow, as it's calling lstat(..) on every file in /data.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

root@relabs-puppet /home/dmitchell # time puppetagain-sync
releng-puppet1.build.scl1.mozilla.com: /data/repos/DMGs/screenresolution-1.5.dmg: updating

real    36m46.902s
user    0m3.543s
sys     0m4.884s

So, I'll run it once an hour, plus on demand.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I'd like to return to this eventually and see if I can make it go faster (best fix I can envision is to switch to rsync..), but this will do for now, and I have bigger fish to fry.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I have a patch ready to land, but I'm heading out the door.  Tomorrow.

OMG rsync is so much faster!

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Landed.  I'll write up docs on the wiki for it.