703986 - Update version of findutils in MozillaBuild to 4.5.9, to reduce search times by ~60x

Status: RESOLVED INCOMPLETE

(Firefox Build System :: MozillaBuild, task)

Description

[Ed Morley [:emorley]]

tl;dr the version of findutils in MozillaBuild (and also the version in the latest official release of msys) takes ~60x longer than the newer version cygwin uses / using the native windows command line and |dir|.

This makes doing (the obvious) things like:
find . ! -wholename "*/.hg*" \
         -type f \
      \( -iname "*.cpp" \
         -o -iname "*.h" \
         -o -iname "*.cc" \
         -o -iname "*.mm" \) | \
    xargs -n 1 sed -i -e 's/\bPR_TRUE  /true     /g' \
...etc extremely slow.

Up until now I thought the slowness was just due to my ageing (and failing) non-SSD HD - and have just been accepting it/using MXR (no idea how much time this has wasted over the last 6-8 months :/).

As it turns out: 
(both run using msys in the 1.6rc version of MozillaBuild)

time find . ! -wholename "*/.hg*" -type f -name "Makefile.in"
real    5m18.506s
user    0m9.048s
sys     1m4.209s

time /c/cygwin/bin/find.exe . ! -wholename "*/.hg*" -type f -name "Makefile.in"
real    0m5.374s
user    0m0.000s
sys     0m0.031s

Version / enabled feature settings:

MozillaBuild: 
GNU find version 4.3.0   
Features enabled: LEAF_OPTIMISATION FTS

Cygwin 1.7: 
find (GNU findutils) 4.5.9     
Features enabled: D_TYPE O_NOFOLLOW(enabled) LEAF_OPTIMISATION FTS(FTS_CWDFD) CBO(level=2)

Not sure if it's the version bump or the (compile time?) enabled features making the difference here. Findutils docs are next to useless & a quick attempt at building from source (to see what features cygwin had to enable manually) failed as I didn't have all the prerequisites. 

I also tried the latest development version of msys (mingw-get-inst-20111118.exe) and it was just as slow as the one currently in MozillaBuild. It uses: find (GNU findutils) 4.4.2, Features enabled: O_NOFOLLOW(disabled) LEAF_OPTIMISATION FTS() CBO(level=0), if that helps for elimination.

Comment 1

[Ed Morley [:emorley]]

Just to clarify (based on IRC comments):

This isn't about trying to mix and match cygwin/msys programs, instead:

1) Finding out why the cygwin version is faster. (Newer version or the possibly compile time enabled features?).

2) Make these same changes to the msys version of findutils bundled with MozillaBuild.

3) \o/ when searches then take <10s rather than 5 minutes.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

MSYS1 appears to be pretty much dead-ended at version 4.4.2. Unless we want to try rolling our own, this might end up having to wait for MSYS2, whenever that is.

Comment 3

[Ed Morley [:emorley]]

Mass-closing old bugs I filed that have not had recent activity/no longer affect me.