1326353 - Reduce amount of console output from Windows |mach build|

Status: NEW

(Firefox Build System :: General, defect)

Description

[Eric Rahm [:erahm]]

For a clobber build I see a 40 minute build time on my Windows 10 laptop, if I pipe the output to /dev/null I can reduce the build time by *5 minutes*.

I propose that we somehow reduce the amount that is actually output (perhaps writing a full log to a file instead). It would be nice to still see some sort of progress and of course build errors.

It's also possible we could just tweak how the output is logged, perhaps stdout on Windows isn't buffered or is being flushed?

Comment 1

[Gregory Szorc [:gps]]

The Windows terminal is historically teh suck. I recall as part of developing mach that I made the default to run `make -s` because the default/verbose output was measured to have a substantial impact on clobber build times - to the tune of dozens of seconds. And that was with iTerm2.app on OS X! However, I spent ~5 minutes searching my inbox and couldn't find a bug or mailing list post referencing that. I know it is recorded somewhere, just not where.

I haven't measured the impact of terminal writing latency in a few years. I expect it to matter. But honestly, the 5 minute claim (12.5% of wall time assuming a 40 minute build) is a bit hard to believe. I'd *really* like to reproduce that.

Comment 2

[Eric Rahm [:erahm]]

FWIW this isn't anecdotal, I timed it on my machine.

Comment 3

[Gregory Szorc [:gps]]

On my i7-6700K Windows 10 home desktop using cmd.exe from a vanilla MozillaBuild install using start-msvc.bat, a `mach build` after `mach configure` on a mostly idle machine yields the following:

w/ terminal output:   20:36
w/o terminal output:  20:22

While there is a difference there, it is only ~14s (or ~1.1%). That falls within the expected deviation for a build. But I wouldn't throw out writing to terminal as responsible for some of those seconds.

This is a far cry from the 5 minutes (or 12.5%) reported in the initial comment. I suspect another factor.

My money is on thermal throttling of the CPU since the machine is a laptop (build times on laptops can be all over the place due to thermal throttling whereas desktops tend to have better cooling and more consistent CPU performance). I also wouldn't be surprised if a non-standard terminal like ConEmu or MinTTY contributed horrible performance.

Need more info.

FWIW, bug 1262569 was supposed to disable printing of individual compile targets being built. But the commit that landed didn't appear to impact Windows. At the very least we should have this bug striving for parity with other platforms, where we only print the library being linked.

Comment 4

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #3)
> FWIW, bug 1262569 was supposed to disable printing of individual compile
> targets being built. But the commit that landed didn't appear to impact
> Windows. At the very least we should have this bug striving for parity with
> other platforms, where we only print the library being linked.

cl.exe is the one printing out the source file names, and AFAIK there's no way to make it not do that, short of redirecting its output, which would then hide warnings and errors.

Comment 5

[Gregory Szorc [:gps]]

The python wrapper for cl.exe is already processing every line of cl.exe output. Filtering out the line with the source file name should be doable.

Comment 6

[Mike Hommey [:glandium]]

... and wouldn't happen with sccache.

Comment 7

[Eric Rahm [:erahm]]

With this laptop I can certainly see thermal throttling being an issue. I re-ran a clean build (no obj-dir) and got a 38 minute build (this was with defender disabled, so that could be a factor) in a more ventilated location (read: on top of a printer). Then running piping to /dev/null I got a (again no obj-dir) I got a 35 minute build. So 3 minutes this time, not as drastic but still significant.

Comment 8

[(not currently active) Ted Mielczarek]

We should, at very least, make Windows builds not output more than builds on other platforms. The build should be pretty quiet in the success case.

Comment 9

[(no longer active)]

Doubling down on removing warnings helps too.  In many cases all we'd need to do is to file bugs, mentor and review patches.