754174 - rsync exclude files for product delivery need to be in the rsync docroot

Status: RESOLVED FIXED

(Infrastructure & Operations :: Infrastructure: Other, task)

Description

[Dave Miller [:justdave]]

The exclude files used by rsync in the product delivery rsync servers used to be symlinked to copies that lived in the rsync docroot.  The reason for this is that those same configs are used by the downstream rsync mirrors (because one two of the modules are subsets of another one, and this way only one of them has to be synced and the other two can be generated by using the same exclude files).

Since these configs got moved into puppet control, the symlinks went away, and the copies that go to the downstream mirrors will now be out-of-date the next time releng makes a change.

A possible way to deal with this is a cron job on one of the boxes which copies the puppet-deployed versions to their respective locations in the rsync docroot periodically (probably at least as often as puppet runs).  Or, you could just have puppet deploy a copy in both places (which would be interesting if both hosts collided trying to do so since it's a shared NFS mount...)

Comment 1

[Brian Hourigan [:digi]]

I added a file resource for $rsync_doc_root/$excludes on upload-cron.private.scl3.mozilla.com so there will be no race condition. The rsync systems only have read only access to the product delivery mounts.

otice: /Stage[main]/Productdelivery::Upload_cron/File[/mnt/netapp/stage/archive.mozilla.org/pub/rsyncd-mozilla-current.exclude]/ensure: defined content as '{md5}96ff83e60111d1413b51de8bd8634320'
notice: /Stage[main]/Productdelivery::Upload_cron/File[/mnt/netapp/stage/archive.mozilla.org/pub/rsyncd-mozilla-prereleases.exclude]/ensure: defined content as '{md5}311bd6a618ec9fe226695cbd53e59030'
notice: /Stage[main]/Productdelivery::Upload_cron/File[/mnt/netapp/stage/archive.mozilla.org/pub/rsyncd-mozilla-releases.exclude]/ensure: defined content as '{md5}525cb8baa9eac954b7487d8bf00dd63a'
notice: /Stage[main]/Productdelivery::Upload_cron/File[/mnt/netapp/stage/releases.mozilla.com/rsyncd-mozilla-com-releases.exclude]/ensure: defined content as '{md5}28dd85fa050b1ddbf0c8cc5f700547bb'

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Please put the rsync exclude files in /mnt/netapp/stage/archive.mozilla.org/pub/zz/ - the zz/ suffix is missing right now - and clean up the misplaced files.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

For these ones anyway:
 rsyncd-mozilla-current.exclude
 rsyncd-mozilla-prereleases.exclude
 rsyncd-mozilla-releases.exclude

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Looks good now.