949704 - Useless warning message about swapping

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(no longer active)]

===================
PERFORMANCE WARNING

Your machine experienced a lot of swap activity during the build. This is
possibly a sign that your machine doesn't have enough physical memory or
not enough available memory to perform the build. It's also possible some
other system activity during the build is to blame.

If you feel this message is not appropriate for your machine configuration,
please file a Core :: Build Config bug at
https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Build%20Config
and tell us about your machine and build configuration so we can adjust the
warning heuristic.
===================

I was running a build and had a VM running, and have about 1GB of free ram (out of 16GB).  I don't think this message is very accurate.

Comment 1

[Mike Hommey [:glandium]]

It's not because you have free RAM that OSX doesn't swap. In fact, OSX likes swapping very much.

Comment 2

[Gregory Szorc [:gps]]

How much swapping was reported during the build?

Comment 3

[(no longer active)]

(In reply to Gregory Szorc [:gps] (in southeast Asia until Dec 14) from comment #2)
> How much swapping was reported during the build?

Hmm, it doesn't say that in the warning.  Not sure how I can find out.

(In reply to Mike Hommey [:glandium] from comment #1)
> It's not because you have free RAM that OSX doesn't swap. In fact, OSX likes
> swapping very much.

Sure, but still the warning is useless, since I cannot get more RAM on this machine no matter how much I want it.  :-)

Comment 4

[(no longer active)]

And also to the best of my knowledge, as long as you run any OS other than Linux, swapping is a fact of life, and all of the RAM in the world won't make it stop!

Comment 5

[Mike Hommey [:glandium]]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #3)
> Sure, but still the warning is useless, since I cannot get more RAM on this
> machine no matter how much I want it.  :-)

"It's also possible some other system activity during the build is to blame."

Comment 6

[Gregory Szorc [:gps]]

We know the current heuristic will need tuning. We're just trying to collect data to help us make informed decisions.

The final lines of your build should say something like:

7:05.24 Overall system resources - Wall time: 424s; CPU: 77%; Read bytes: 1757860864; Write bytes: 6498325504; Read time: 38907; Write time: 101839
7:05.24 Swap in/out (MB): 912/0

Please report the swap in/out values.

Comment 7

[(no longer active)]

(In reply to comment #6)
> We know the current heuristic will need tuning. We're just trying to collect
> data to help us make informed decisions.
> 
> The final lines of your build should say something like:
> 
> 7:05.24 Overall system resources - Wall time: 424s; CPU: 77%; Read bytes:
> 1757860864; Write bytes: 6498325504; Read time: 38907; Write time: 101839
> 7:05.24 Swap in/out (MB): 912/0
> 
> Please report the swap in/out values.

Oh, I've long lost that one log, sorry.  It would be nice if these numbers were inside the warning section, because I actually don't think I've ever notices these swap numbers.

But still the point remains that showing this warning on a machine with 16GB of RAM is probably pointless.  We *know* that is enough memory for building Firefox, and I don't think we need to tell people that building with 10 concurrently running VMs may not give you the fastest build experience.  ;-)

Comment 8

[Gregory Szorc [:gps]]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #7)
> But still the point remains that showing this warning on a machine with 16GB
> of RAM is probably pointless.  We *know* that is enough memory for building
> Firefox, and I don't think we need to tell people that building with 10
> concurrently running VMs may not give you the fastest build experience.  ;-)

We are in agreement. We didn't know what kind of results we'd get in the wild when we pushed this. We know we need to adjust the heuristic. Can you suggest a better one?

Comment 9

[(no longer active)]

(In reply to Gregory Szorc [:gps] (in southeast Asia until Dec 14) from comment #6)
> We know the current heuristic will need tuning. We're just trying to collect
> data to help us make informed decisions.
> 
> The final lines of your build should say something like:
> 
> 7:05.24 Overall system resources - Wall time: 424s; CPU: 77%; Read bytes:
> 1757860864; Write bytes: 6498325504; Read time: 38907; Write time: 101839
> 7:05.24 Swap in/out (MB): 912/0
> 
> Please report the swap in/out values.

I got this again:

 7:36.76 Overall system resources - Wall time: 456s; CPU: 78%; Read bytes: 1573799936; Write bytes: 4030499328; Read time: 29220; Write time: 45283
 7:36.77 Swap in/out (MB): 671/160

Comment 10

[(no longer active)]

(In reply to Gregory Szorc [:gps] (in southeast Asia until Dec 14) from comment #8)
> (In reply to :Ehsan Akhgari (needinfo? me!) from comment #7)
> > But still the point remains that showing this warning on a machine with 16GB
> > of RAM is probably pointless.  We *know* that is enough memory for building
> > Firefox, and I don't think we need to tell people that building with 10
> > concurrently running VMs may not give you the fastest build experience.  ;-)
> 
> We are in agreement. We didn't know what kind of results we'd get in the
> wild when we pushed this. We know we need to adjust the heuristic. Can you
> suggest a better one?

Yes: "never show this message for machines with 8GB of RAM or more".

Comment 11

[Gregory Szorc [:gps]]

I think 8GB is on the low side for "never show." I'd go with 16. Even then, if you are running some memory hogging process, no ceiling value is safe. The whole point of this warning is to let people know when swapping is likely slowing down the build [so they can take appropriate action].

Comment 12

[(no longer active)]

(In reply to comment #11)
> I think 8GB is on the low side for "never show." I'd go with 16. Even then, if
> you are running some memory hogging process, no ceiling value is safe. The
> whole point of this warning is to let people know when swapping is likely
> slowing down the build [so they can take appropriate action].

Well, see the end of comment 3 where I talk about it being (physically) impossible to put more RAM inside this machine, and also comment 4.  My contention is, if we're telling this to people on hardware that they cannot upgrade or on OSes which do not let you use your physical memory, we're just giving them some information that they cannot act on.  But if you disagree, I won't keep arguing this any more.

Comment 13

[Benjamin Smedberg]

We should only show this message if there is something the user could actually do in their build config to fix it: for example if we know that disabling unified builds would reduce memory consumption and improve the build speed for people who are swapping, we should either recommend it or do it automatically.

But otherwise, I think Ehsan's right: we should assume that the hardware people are using is basically immutable and only issues warnings for software/OS misconfiguration, not for "you don't have enough RAM".

Comment 14

[(no longer active)]

I got this again, without doing anything special on my machine while building:

 5:33.90 Overall system resources - Wall time: 333s; CPU: 69%; Read bytes: 1468243968; Write bytes: 2822218240; Read time: 28437; Write time: 33355
 5:33.90 Swap in/out (MB): 737/0
 5:33.90 Exception when writing resource usage file: u'compile_external'
 5:33.90 /usr/local/bin/terminal-notifier -title Mozilla Build System -group mozbuild -message Build complete
Your build was successful!

===================
PERFORMANCE WARNING

Your machine experienced a lot of swap activity during the build. This is
possibly a sign that your machine doesn't have enough physical memory or
not enough available memory to perform the build. It's also possible some
other system activity during the build is to blame.

If you feel this message is not appropriate for your machine configuration,
please file a Core :: Build Config bug at
https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Build%20Config
and tell us about your machine and build configuration so we can adjust the
warning heuristic.
===================

Comment 15

[(no longer active)]

And another one, again under normal conditions:

 5:12.14 Overall system resources - Wall time: 311s; CPU: 76%; Read bytes: 1000094208; Write bytes: 2336378880; Read time: 17448; Write time: 27640
 5:12.14 Swap in/out (MB): 516/0
 5:12.15 Exception when writing resource usage file: u'compile_external'
 5:12.15 /usr/local/bin/terminal-notifier -title Mozilla Build System -group mozbuild -message Build complete
Your build was successful!

===================
PERFORMANCE WARNING

Your machine experienced a lot of swap activity during the build. This is
possibly a sign that your machine doesn't have enough physical memory or
not enough available memory to perform the build. It's also possible some
other system activity during the build is to blame.

If you feel this message is not appropriate for your machine configuration,
please file a Core :: Build Config bug at
https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Build%20Config
and tell us about your machine and build configuration so we can adjust the
warning heuristic.
===================

Comment 16

[Gregory Szorc [:gps]]

Submit a patch to comment out the printing of the warning message and I'll r+ it.

I think we can come up with a more meaningful and actionable error message or error messages that don't nag as much. For now, I think we can get by reporting just the swap activity without a relatively inactionable performance warning.

Comment 17

[(no longer active)]

Attachment: Patch (v1)

Comment 18

[(no longer active)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/efca82d20128

Comment 19

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/efca82d20128