914431 - Warn when building with less than 4 physical cores

Status: NEW

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

The build system is mostly CPU bound on C++ compilation. Building with 4 cores vs 2 can mean the difference between 18 minute builds and 35+ minute builds. I'd like the build system to very visibly warn people when they are building with less than 4 *physical* cores. I think hyperthreaded cores should be ignored because they only add ~1.25x the value of a physical core.

I'm even tempted to go so far as to suggest we fail builds on "insufficient" hardware unless a OVERRIDE_INSUFFICIENT_HARDWARE_CHECK=1 or something is defined in their mozconfig. But, I'll hold off requesting that until someone else comes out on the ledge with me.

Comment 1

[Gregory Szorc [:gps]]

OS X: http://stackoverflow.com/questions/5143839/getting-number-of-cpus-on-mac-os-x-from-c
Linux: http://xmodulo.com/2012/06/how-to-find-number-of-cpu-cores-on.html (essentially parsing /proc/cpuinfo)
Windows: GetLogicalProcessorInformation http://msdn.microsoft.com/en-us/library/ms683194

Comment 2

[Mike Hommey [:glandium]]

IMHO, we should add an API for that to psutil and submit it upstream.

Comment 3

[Benjamin Smedberg]

I don't mind a warning, but I strongly object to making the build fail. There are plenty of good reasons why people, even core developers, may still be doing some builds with slow machines, such as owning a mac mini or an older mac laptop to do the occasional mac fix but not normally needing a full mac dev environment.

Comment 4

[Gregory Szorc [:gps]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #3)
> I don't mind a warning, but I strongly object to making the build fail.
> There are plenty of good reasons why people, even core developers, may still
> be doing some builds with slow machines, such as owning a mac mini or an
> older mac laptop to do the occasional mac fix but not normally needing a
> full mac dev environment.

And I would say that it's pretty simple to make a one-time change on your system to ignore the failure. But as I said, I'm not concerned about the fatal error bit yet. Let's get a warning landed first and then we can debate about making it fatal.

Comment 5

[(not currently active) Ted Mielczarek]

Do you really think that warning people that they're building on slow hardware is going to be effective? I don't think a warning is going to cause anyone to get faster hardware. People being paid to hack on Mozilla can get fast hardware. Volunteer contributors are not going to go spend money just because we tell them their hardware is crappy. What's the point?

Comment 6

[Gregory Szorc [:gps]]

You would be surprised how often I run into someone (especially in offices) who has 40+ minute build times due to less than 4 physical cores and is complaining the build is too slow and they are blaming the build system. I would like to empower people to make intelligent decisions about hardware purchases and to take appreciate actions to correct the problem.

I agree the story for volunteer contributors is a little mucky since they can't just ask for new hardware. But if someone realizes how far a new CPU will go, perhaps they'll consider it sooner.

Comment 7

[Ed Morley [:emorley]]

Sounds like we should make the pref I_PROMISE_NOT_TO_BLAME_GPS_FOR_SLOW_BUILDS_WHEN_I_PASS_HIM_IN_THE_CORRIDOR=1 ;-)

Comment 8

[Joshua Cranmer [:jcranmer]]

I strongly object to making having too few cores an error and mildly object to even making it a warning. Buying a new computer with that many cores is going to set people back a few hundred dollars, so you're in effect telling volunteers "Spend a few hundred dollars if you want to build Mozilla!", which is not a good thing to start telling new contributors.

Comment 9

[Gregory Szorc [:gps]]

I already backed down from it being a fatal error.

What's wrong with telling somebody they'll have a bad experience? Best case it causes someone to take action. If that's a Mozilla employee or a paid contributor, we've potentially saved thousands of dollars in regained productivity per instance. Worst case is it is annoying. We can mitigate the worst case through intelligent UX design.

I just find it unacceptable that *paid* developers are unknowingly developing on crappy hardware. That's Mozilla effectively pissing away money and I don't think the realization of this will jive well with a lot of people, especially since it is easily correctable.

Comment 10

[Justin Dolske [:Dolske]]

Seems reasonable to comment on system performance if mach thinks something is awry -- we're already doing similar things in Firefox itself (eg detecting slow startups and notifying with a remedy), and want to do more.

It's also quite fair to be concerned about language and reaction; plenty of people can't afford the latest high-end hardware. Surely we can find a way to be both informative and respectful.

Random idea: Would it be useful to add a new command ("mach benchmark"?) to time and measure stuff, and for normal building just have a final check/message that says something to the effect of "Hey, your build took X% longer than is possible in the best case, run mach benchmark to learn more."? [Where "best case" is, I dunno, maybe just a simple hardcoded table of per-platform World Build Records.]

Comment 11

[Gregory Szorc [:gps]]

I would like the build system to one day automatically submit anonymous logs of build stats to Mozilla so we could get stats from people in the wild. Other companies do this so they can measure the impact of build system changes. Then, people could compare their results against others and have a feel for where they perform.

We already produce resource usage JSON files. We just need to record basic hardware info in there and we'll be set from a data production perspective. Metrics already operates a server we can use for harvesting the data (Bagheera). The big hurdle is privacy concerns. I concede this isn't a very "Mozilla" thing to do and thus we'd need to tread lightly. To be clear, I'm not proposing that here. Just an idea I've had. Data-driven decisions FTW.

Regarding the benchmark idea, we have another bug somewhere where we'd prompt the user if config.status took too long. It's a rough approximation of machine capability. Another benchmark is "did a clobber build take more than 25 minutes on non-Windows."

Comment 12

[Gregory Szorc [:gps]]

(In reply to Mike Hommey [:glandium] from comment #2)
> IMHO, we should add an API for that to psutil and submit it upstream.

Filed https://code.google.com/p/psutil/issues/detail?id=427

Considering psutil is optional, I think there is room in mozbase to write similar functionality (at least until it becomes available in psutil).

Comment 13

[Gregory Szorc [:gps]]

The psutil people warmed up to implementing this. However, I'm not actively working on it. Would be amazing if someone else could find the time.

Comment 14

[Gregory Szorc [:gps]]

Upstream psutil has implemented an API to return physical CPU count. This feature is now unblocked.

Comment 15

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bz://914431/gps

Comment 16

[Gregory Szorc [:gps]]

/r/1779 - Bug 914431 - Add configure flag to disable insufficient hardware warnings
/r/1781 - Bug 914431 - Warn when building with less than 4 physical CPU cores
/r/1783 - Bug 914431 - Print a warning when building with insufficient memory

Pull down these commits:

hg pull review -r 690a034b2d1513d341c48e0e1744386b731d27d9

Comment 17

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=690a034b2d15

Comment 18

[Mike Hommey [:glandium]]

https://reviewboard.mozilla.org/r/1777/#review1289

There is a major problem with this sledgehammer kind of approach: we have many legitimate build options that actually don't require 4 physical cores and 8 GiB of memory, and we're going to have even more in the near future. I don't think requiring all people using those options to add the relevant --disable is very useful. As it's not really easy to find out all those options, we should find heuristics that make sense. Like, if the build took more than 5 minutes and, for memory, if the average cpu usage is less than x%.

::: configure.in
(Diff revision 1)
> +AC_SUBST(MOZ_DISABLE_INSUFFICIENT_HARDWARE_WARNINGS)

There is no compelling reason for this part to be a separate changeset. In fact, there is no compelling reason for this entire series to be more than one changeset.

::: python/mozbuild/mozbuild/mach_commands.py
(Diff revision 1)
> +about acquiring a faster machine.

I'd leave out the "paid to work on Firefox" part. For one, I think we have legal ways to provide hardware to non-paid contributors. Then, there are others non-Mozilla lines of work where this is relevant advice. Finally, even within Mozilla not all the people compiling this code are working on Firefox.

::: python/mozbuild/mozbuild/mach_commands.py
(Diff revision 1)
> +at least {wanted} MB.

Why no equivalent "talk with your manager" message here?

::: python/mozbuild/mozbuild/controller/building.py
(Diff revision 1)
> +        return True, cores, WANTED_CPU_CORES

return cores < WANTED_CPU_CORES, cores, WANTED_CPU_CORES

::: python/mozbuild/mozbuild/controller/building.py
(Diff revision 1)
> +        # We want 900 MB per physical core to correspond with the -j<# cores>

900 MiB

::: python/mozbuild/mozbuild/controller/building.py
(Diff revision 1)
> +        # We want this to be 8 GB. We don't use base 2 here because machines

8 GiB

Comment 19

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/1777/#review1309

I agree.

However, perfect is the enemy of done.

Regarding future build options, I think we can carve out an exception for them later. My goal with this patch is to raise the alarm for people who are building with obviously sub-optimal hardware configurations *today*.

I'd be fine with only showing the warning if the build took more than 5 minutes. I think it would be annoying to see these warnings on light builds that take 30s.

The last time we tried to tackle heuristics, Ehsan and others complained pretty heavily and it was backed out. Granted, this was for swapping, which has more nuances than CPU and memory usage. I'd rather wait for us to collect data from people before we get too specific with heuristics. That way, we can draw lines at appropriate percentages and at levels that are achievable for the majority.

> I'd leave out the "paid to work on Firefox" part. For one, I think we have legal ways to provide hardware to non-paid contributors. Then, there are others non-Mozilla lines of work where this is relevant advice. Finally, even within Mozilla not all the people compiling this code are working on Firefox.

Yeah, the wording here could use some bikeshedding. It's difficult to address all audiences while simultaneously being more harshly worded to Mozilla employees, who we have a vested interest in ensuring they act on this message.

Comment 21

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 914431 - Add configure flag to disable insufficient hardware warnings

Comment 22

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 914431 - Warn when building with less than 4 physical CPU cores

Comment 23

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 914431 - Print a warning when building with insufficient memory