1105140 - Don't use shutil.copy2 in mozpack.files.*.copy on Windows

Status: NEW

(Firefox Build System :: General, defect, P3)

Description

[Gregory Szorc [:gps]]

Install manifest performance on Windows is teh crap. There are 2 main reasons:

1) stat() is slow
2) shutil.copyfile()

shutil.copyfile() is the worst offender. Below is a profile snapshot installing ~15000 files into the _tests directory during install manifest processing.

If you run outside of profiling mode and start from a clean slate, it takes ~160s to run to completion. Contrast with <20s on POSIX platforms. This is unacceptable.

I suspect we can reimplement shutil.copyfile() using Win32 APIs on top of ctypes to achieve better performance and shave seconds off of build times.

From _tests: Kept 18853 existing; Added/updated 15043; Removed 0 files and 0 directories.
         6968133 function calls (6883451 primitive calls) in 69.551 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.006    0.006   69.551   69.551 process_install_manifest.py:5(<module>)
        1    0.012    0.012   69.459   69.459 process_install_manifest.py:33(main)
        1    0.005    0.005   69.445   69.445 process_install_manifest.py:17(process_manifest)
        1    0.191    0.191   65.209   65.209 copier.py:179(copy)
    33872    0.123    0.000   63.416    0.002 files.py:245(copy)
    33872    0.138    0.000   63.235    0.002 files.py:113(copy)
    15040    0.107    0.000   59.903    0.004 shutil.py:122(copy2)
    15040   32.397    0.002   48.652    0.003 shutil.py:66(copyfile)
    15043    0.113    0.000   10.913    0.001 shutil.py:93(copystat)
    30087    8.228    0.000    8.228    0.000 {open}
    15040    0.058    0.000    5.893    0.000 shutil.py:46(copyfileobj)
    32085    5.533    0.000    5.533    0.000 {method 'read' of 'file' objects}
    15043    5.499    0.000    5.499    0.000 {nt.utime}
   132202    5.007    0.000    5.007    0.000 {nt.stat}
    15043    4.786    0.000    4.786    0.000 {nt.chmod}
        1    0.083    0.083    4.109    4.109 manifests.py:299(populate_registry)
    15520    0.014    0.000    2.276    0.000 files.py:672(find)
31210/15520    0.033    0.000    2.258    0.000 files.py:799(_find_glob)
    18832    0.087    0.000    2.187    0.000 files.py:83(is_older)
    37664    0.047    0.000    2.100    0.000 genericpath.py:52(getmtime)
63597/15443    0.065    0.000    1.668    0.000 files.py:761(_find_dir)
    33896    0.176    0.000    1.488    0.000 copier.py:50(add)
    49337    0.109    0.000    1.247    0.000 genericpath.py:15(exists)
    32280    1.051    0.000    1.051    0.000 {nt._isdir}
    33872    0.046    0.000    0.961    0.000 files.py:68(exists)
    33896    0.161    0.000    0.943    0.000 copier.py:38(_partial_paths)
    17303    0.047    0.000    0.913    0.000 files.py:747(_find)
    67275    0.544    0.000    0.865    0.000 ntpath.py:398(normpath)
   250802    0.141    0.000    0.754    0.000 path.py:43(dirname)
    15040    0.048    0.000    0.745    0.000 shutil.py:54(_samefile)
   196997    0.405    0.000    0.672    0.000 ntpath.py:63(join)
    30081    0.045    0.000    0.612    0.000 ntpath.py:466(abspath)
    15470    0.064    0.000    0.504    0.000 path.py:85(match)
    30912    0.054    0.000    0.466    0.000 files.py:781(_find_file)
23028/2986    0.034    0.000    0.436    0.000 walk.py:8(walk)
   250802    0.281    0.000    0.421    0.000 posixpath.py:127(dirname)
    24810    0.329    0.000    0.357    0.000 scandir.py:273(scandir)
    33897    0.152    0.000    0.310    0.000 collections.py:495(update)
    30940    0.024    0.000    0.304    0.000 re.py:144(sub)
    17033    0.302    0.000    0.302    0.000 {method 'write' of 'file' objects}
   215506    0.174    0.000    0.288    0.000 ntpath.py:55(isabs)
    63586    0.057    0.000    0.287    0.000 path.py:35(join)
   314452    0.164    0.000    0.270    0.000 path.py:16(normsep)
    30940    0.146    0.000    0.242    0.000 {method 'sub' of '_sre.SRE_Pattern' objects}
     2986    0.227    0.000    0.227    0.000 {nt.mkdir}
   204113    0.067    0.000    0.167    0.000 {isinstance}
   427274    0.156    0.000    0.156    0.000 {method 'replace' of 'unicode' objects}
   282786    0.156    0.000    0.156    0.000 ntpath.py:122(splitdrive)
    94107    0.119    0.000    0.119    0.000 {hasattr}
     1791    0.114    0.000    0.114    0.000 {nt.listdir}
    67812    0.074    0.000    0.102    0.000 collections.py:121(iteritems)
    33900    0.062    0.000    0.100    0.000 abc.py:128(__instancecheck__)
    33872    0.042    0.000    0.093    0.000 files.py:239(__init__)
        2    0.000    0.000    0.087    0.044 manifests.py:96(__init__)
        1    0.047    0.047    0.087    0.087 manifests.py:115(_load_from_fileobj)

Comment 1

[Gregory Szorc [:gps]]

I should have said "don't use shutil.copy2." We can write our own optimized version built on top of CopyFileEx.

Comment 2

[Mike Hommey [:glandium]]

The thing is, on POSIX platforms, we don't copy... we symlink.

Comment 3

[Sylvestre Ledru [:Sylvestre]]

Moving to p3 because no activity for at least 24 weeks.