1326329 - Each cl invocation spawns at least 5 processes

Status: RESOLVED WORKSFORME

(Firefox Build System :: General, defect)

Description

[Eric Rahm [:erahm]]

Per bug 1095293, comment 6 it looks we spawn 3 sh.exe processes, 1 python.exe process, and one cl.exe process each time we invoke |cl|. Creating new processes on Windows is known to be slow, it seem like we should just have mozmake call cl directly.

Comment 1

[Mike Hommey [:glandium]]

Copy/pasting bug 1095293 comment 9, since it's relevant here:
  sh can probably be skipped with some effort rewriting the commands used to compile such that make calls CreateProcess on its own instead of going through the shell. Python can't be because that's what gets us the dependency files for make to figure out when to rebuild something (although sccache can do this too, and the rust rewrite should have less overhead than python).

Comment 2

[(not currently active) Ted Mielczarek]

I couldn't find any evidence that we'd ever tried this, so I submitted a request to the Visual Studio suggestion box asking for support for generating dependency files like GCC/Clang can:
https://visualstudio.uservoice.com/forums/121579-visual-studio-ide/suggestions/18649837-add-support-for-generating-a-dependency-file-like

Can't hurt, right?

Comment 3

[(no longer active)]

FWIW someone measured the overhead of CreateProcess on Windows here <https://bugs.llvm.org/show_bug.cgi?id=20253#c4> to be around 60ms!  If that or anything around it is to be trusted, doing that 5 times is 300 milli-seconds per invocations...

Comment 4

[(not currently active) Ted Mielczarek]

I don't see any obvious reason we ought to be spawning extraneous shells during compilation. On my machine, a compile invocation looks like:
 c:/build/mozilla-central/debug-mozilla-central/_virtualenv/Scripts/python.exe -m mozbuild.action.cl  cl.exe -Fomozjemalloc_compat.obj -c  -DDEBUG=1 -DTRACING=1 -DMOZ_JEMALLOC4 -DMOZ_REPLACE_JEMALLOC -Ic:/build/mozilla-central/memory/replace/jemalloc -Ic:/build/mozilla-central/debug-mozilla-central/memory/replace/jemalloc -Ic:/build/mozilla-central/debug-mozilla-central/memory/jemalloc/src/include -Ic:/build/mozilla-central/memory/jemalloc/src/include/msvc_compat -Ic:/build/mozilla-central/debug-mozilla-central/dist/include  -Ic:/build/mozilla-central/debug-mozilla-central/dist/include/nspr -Ic:/build/mozilla-central/debug-mozilla-central/dist/include/nss        -MD -FI c:/build/mozilla-central/debug-mozilla-central/mozilla-config.h -DMOZILLA_CLIENT   -TC -nologo -wd4091 -D_HAS_EXCEPTIONS=0 -W3 -Gy -Zc:inline -utf-8 -arch:SSE2 -FS -Gw -wd4244 -wd4267 -we4553  -Zi -Oy-   -Fdgenerated.pdb  c:/build/mozilla-central/memory/build/mozjemalloc_compat.c

I can't see that there are any shell special characters there, so mozmake ought to be able to call CreateProcess with that.

Comment 5

[Nathan Froyd [:froydnj]]

I think this is resolved; at the very least, we no longer have cl.py, since clang-cl knows how to write make-style dependencies on its own, and comment 4 indicates that we ought to be calling CreateProcess in any event.