829612 - makefiles: use rsync to mkdir/copy/install files

Status: NEW

(Firefox Build System :: General, defect)

Description

[Joey Armstrong [:joey]]

rsync [--checksum] --delete should be used to replace commands for copying/installing files like the rule below.  The command can address multiple problems:
  o directory creation
  o only modified files will be updated.
  o --delete will purge content when sources are removed.

--checksum can be used if/when timestamps are not reliable or content may have been modified for l10n.

rsync is may not be available on all platforms, in the interim config/makefiles/makeutils.mk::copy_dir can be written as:
  rsync || tar

to use rsync when available else fall back to existing command use.




build/mobile/sutagent/Makefile.in
=================================

export::
        $(NSINSTALL) -D res
        @(cd $(srcdir)/res && tar $(TAR_CREATE_FLAGS) - *) | (cd $(DEPTH)/build/mobile/sutagent/android/res && tar -xf -)

Comment 1

[Joey Armstrong [:joey]]

	Callek	joey: so, rsync is *definitely* not in mozillabuild
	Callek	joey: so in the windows case, we cant rely on it, for certain
	joey	Callek: adding an '||' condition will handle that. For platforms that do not yet have rsync available just fall back to the current tar command
	joey	platforms like windows could begin using it later on when the command is deployed
	Callek	joey: no strong objection, but we have to recognize the condition where rsync fails BUT not for the fact of missing it
	Callek	but I don't need to handhold you, you're quite capable as well
	Callek	(frankly not sure why I still watch Build Config, I haven't done work other than simply reviewing in c-c build config lately)
	joey	well create rsync.py and just wrapper any relevant details in there
	Ms2ger	approves of python
	joey	Callek: oh also rsync related failures are not a big deal here until the pipeline is removed. Tar would provide a builtin retry.
	Callek	joey: more my concern is along the line of rsync fails -- leaves behind partial/corrupt file, tar then doesn' overwrite file/dir fully -- boom
	Callek	joey: but yea, I trust you to do the proper magic
	joey	Callek: if rsync fails it will likely be due to the filesystem being full, perm problems, etc, etc.
	Callek	yea
	Callek	E_TOO_LITTLE_FAITH
	joey	when that happens tar will either over-write the file later or fail for exactly the same reason
	joey	if tar bails out, make exits with non-zero status and we have a visible indicator

Comment 2

[Mike Hommey [:glandium]]

The new packager has code that does things similar to rsync. It would be trivial to write a rsync.py around the new packager code.

Comment 3

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #2)
> The new packager has code that does things similar to rsync. It would be
> trivial to write a rsync.py around the new packager code.

The code would look like this:

from mozpack.copier import FileCopier
from mozpack.files import FileFinder

def sync(dir1, dir2):
    finder = FileFinder(dir1)
    copier = FileCopier()
    for p, f in finder:
        copier.add(p, f)
    copier.copy(dir2)