Windows builds are excessively slow on dev machines

RESOLVED INVALID

Status

()

Core
Build Config
RESOLVED INVALID
4 months ago
4 months ago

People

(Reporter: erahm, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 months ago
On a snazzy new Windows 10 laptop [1] I'm seeing 40 minute clobber times. While I'd like to use it for daily development, that's just too long compared to ~18 minutes on my Linux desktop or similar on my 3 year old MacBook Pro (both have ccache enabled), 22 minute clobber build on the macbook with ccache disabled.

Please note, while super cool in theory, artifact builds are not an option for me as I'm working mostly on C++.

I fished around for ways to optimize, I came up with the following as suggested to improve build times:
- Add an exception for my `C:\dev` folder in defender (AV)
- Set power settings to "Performance"

It's possible I'm missing other settings that might improve build time. If we could enumerate those and possibly package them in with `mach bootstrap` that would be excellent. I believe the power setting is being looked into already.

On a good note I found that piping the output to /dev/null saved ~5 minutes in build time, this is a nice win but not great from a usability perspective.

It seems like a few likely action items would be:
  1) Look into spewing less console output (warnings / errors / status would still be nice). This seems like the easiest win.
  2) Investigate a ccache-like wrapper that works with cl. clcache [2] looks promising.
  3) Build directly from msvc (create a solution that builds directly rather than shelling out to mozmake), not sure how likely it is we could actually do this, but my understanding is msvc can do smarter things (like a cl-server rather than spawning new cl instances)
  4) Profile what's taking so long, it could be msys and other builds system interaction

[1] Device specs, Lenovo ThinkPad P50:
- Intel Xeon E3-1505 M @ 2.80GHz, 8 logical cores
- 32 GB RAM
- 512 GB SSD

[2] https://github.com/frerich/clcache

Comment 1

4 months ago
If you are working "mostly on C++" on Windows you should be building on a desktop (not a laptop). Full stop. If you are building on Linux or OS X and have access to an IceCream distributed compilation farm, you can get by with a laptop.

While this Xeon E3-1505 has 4 cores, those are clocked at 2.8GHz base. Compared to a desktop i7-6700K (also a SkyLake), you have 11.2 GHz total compared to 16.0 GHz, or 70%. That's a few minutes right there.

You want a dual socket Xeon desktop, probably the Lenovo ThinkStation P710 with 2x E5-2643v4 or 2x E5-2637v4. bholley just got a new machine and he can probably give you a service request to reference when you put in the hardware order.

All your points about improving build speed are valid and tracked in other bugs. I don't think this bug has much value being open, so I'm closing it.
Status: NEW → RESOLVED
Last Resolved: 4 months ago
Resolution: --- → INVALID
(Reporter)

Comment 2

4 months ago
(In reply to Gregory Szorc [:gps] from comment #1)
> All your points about improving build speed are valid and tracked in other
> bugs. I don't think this bug has much value being open, so I'm closing it.

That's great! Can you point me to the bugs? I think it would be useful to at least have a meta bug for tracking them.
(In reply to Eric Rahm [:erahm] from comment #0)
>   2) Investigate a ccache-like wrapper that works with cl. clcache [2] looks
> promising.

We have this! It's called sccache and we use it for our CI builds: https://github.com/mozilla/sccache

I'm finishing up some work to make it usable for local development in bug 1318370.
You need to log in before you can comment on or make changes to this bug.