1897079 - all_mozbuild_paths is too slow

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[alphare33]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0

Steps to reproduce:

Run any command that scans for **/moz.build files like mach taskgraph morphed with a profiler (I used py-spy).

Actual results:

all_mozbuild_paths takes around 6s on a quite recent and fast machine (AMD Ryzen 9 7950X, Samsung SSD 980 PRO, plenty of fast RAM).

Expected results:

I expect this step to take a few hundreds of milliseconds on such a machine. Using fd 'moz.build' -E python/mozbuild/mozbuild/test -E testing/mozbase/moztest/tests/data -E 'obj*' -E '*/config.status', with fd being the fd-find Rust binary, I get around 300ms.

Comment 1

[alphare33]

Unfortunately, from previous discussions with glandium and sergesanspaille, using Rust (for easy parallelization) in this instance is not possible since all_mozbuild_paths is used during an early enough stage of the build system that compiled extensions are not really an option.

However, a comment left in this function could be the solution :

# In the future, we may traverse moz.build files by looking
# for DIRS references in the AST, even if a directory is added behind
# a conditional. For now, just walk the filesystem.

Comment 2

[alphare33]

Attachment: Bug 1897079 - Improve speed of fs traversal in `all_mozbuild_paths`. r=glandium!?sergesanspaille

This duplicates some of the logic w.r.t. ignore logic to prioritize speed.
Hopefully the tradeoff is acceptable.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dc95deda78cc
Improve speed of fs traversal in `all_mozbuild_paths`. r=glandium?sergesanspaille

Comment 4

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/dc95deda78cc