833018 - ASan: Enable memory-saving options for tests when running on test slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Christian Holler (:decoder)]

Attachment: Patch

Running our tests (especially mochitests) on ASan builds consumes a lot of memory. In the standard configuration, this memory requirement even exceeds 2 GB of RAM, which seems to be the maximum available on the test slaves.

Previously, we unconditionally enabled several memory-safety hacks when pushing to try with tests enabled. Since we're working on enabling tests for ASan builds in bug 831491, we need to get these into the tree now. However, it would be unwise to enable these options unconditionally, even if they only affect ASan builds: With the options enabled, chances are that under rare conditions, bugs don't get detected where the default options would detect them. Therefore we should always run under default options if the machine has enough memory.

In the attached patch, I solved the problem by enabling the options only when 2 GB or less of total memory are available.

Furthermore, the patch disables most ASan capabilities unconditionally for the xpcshell server that serves the mochitests. The reason is that this process consumes a lot of memory with ASan but it's not the unit being tested here so there is little benefit in running that xpcshell instance with ASan. Since it's not possible by the build system to use a second non-asanified xpcshell here, the patch tries to cut down memory consumption as far as possible.

Patch is on try now for testing, review/feedback welcome :)

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 704588 [details] [diff] [review]
Patch

Review of attachment 704588 [details] [diff] [review]:
-----------------------------------------------------------------

r=me with some error checking around the totalMemory line.

::: build/automation.py.in
@@ +814,5 @@
>      env['NS_TRACE_MALLOC_DISABLE_STACKS'] = '1'
> +
> +    # ASan specific environment stuff
> +    if IS_ASAN and (IS_LINUX or IS_MAC):
> +      totalMemory = int(os.popen("free").readlines()[1].split()[1])

have you tested this on all linux and osx platforms?  I would like to see this in a try/except block to be safer.

Comment 2

[Christian Holler (:decoder)]

I wrapped the code in a double try/except to catch OSError and possibly other errors, output a meaningful message and default to the non-memory-saving options. This is on inbound now:

https://hg.mozilla.org/integration/mozilla-inbound/rev/08c14f94fbb7

Comment 3

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/08c14f94fbb7