981023 - Expose ASAN and LSAN errors in TBPL

Status: RESOLVED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Andrew McCreight [:mccr8]]

Right now, ASAN failures just show up in TBPL as crashes or timeouts, which is not good.  ASAN seems to have a nice little summary for its failures, so it should be easy to do, once I figure out the right place to modify.  Bug 966864 does a similar thing for Valgrind.

ASAN produces output like:
SUMMARY: AddressSanitizer: use-after-poison filename:linenumber functionname

which seems pretty complete.  There's also heap-buffer-overflow and use-after-free.  Decoder, if you know of any others, let me know.

Another issue is that right now the filename seems to include a bunch of builder path gunk, which would be nice to clean up.

Once this is done for ASAN, it should be easy to add additional things for LSAN, which are probably in the same general format.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Honestly, this can probably be handled on the TBPL side if you're happy with the output in the logs and the failure codes it makes.

https://mxr.mozilla.org/webtools-central/source/tbpl/php/inc/GeneralErrorFilter.php

Comment 2

[Andrew McCreight [:mccr8]]

I guess I don't even need to know a list of the specific errors, we can just match anything starting with "SUMMARY: AddressSanitizer".

Comment 3

[Christian Holler (:decoder)]

Here's a list of things that may show up instead of heap-buffer-overflow (directly copied from the source):

heap-use-after-free (maybe just use-after-free in older versions?)
stack-buffer-underflow
stack-buffer-overflow
initialization-order-fiasco
use-after-poison
container-overflow
stack-use-after-return (not enabled right now on TBPL)
stack-use-after-scope (not enabled right now on TBPL)
global-buffer-overflow


And then there is things like:

ERROR: AddressSanitizer: attempting double-free on ...
ERROR: AddressSanitizer: attempting free on address ...
ERROR: AddressSanitizer: alloc-dealloc-mismatch (not enabled right now on TBPL)
ERROR: AddressSanitizer: attempting to call malloc_usable_size() for pointer which is not owned
ERROR: AddressSanitizer: attempting to call __asan_get_allocated_size() for pointer which is not owned
ERROR: AddressSanitizer: %s: memory ranges [%p,%p) and [%p, %p) overlap

There are more obscure ones, but I guess these are the most common.

Comment 4

[Andrew McCreight [:mccr8]]

Attachment: match ASAN and LSAN errors

Well, I think this is what I need to do, but I'm not able to get the Vagrant local testing environment to work, so I'm not sure.

Comment 5

[Andrew McCreight [:mccr8]]

Is this simple enough to get reviewed and land as-is, or should I work some more on local testing?

Things seem to be going awry with vagrant here:
notice: /Stage[main]/Tbpl-base/Exec[/bin/cp -v /var/www/tbpl/php/config.php.example /var/www/tbpl/php/config.php]/returns: /bin/cp: cannot create regular file `/var/www/tbpl/php/config.php': Permission denied

Comment 6

[Ed Morley [:emorley]]

Comment on attachment 8387786 [details] [diff] [review]
match ASAN and LSAN errors

Review of attachment 8387786 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Andrew McCreight [:mccr8] from comment #5)
> Is this simple enough to get reviewed and land as-is, or should I work some
> more on local testing?

Yeah this is fine as is - thank you for the patch :-)

> Things seem to be going awry with vagrant here:
> notice: /Stage[main]/Tbpl-base/Exec[/bin/cp -v
> /var/www/tbpl/php/config.php.example /var/www/tbpl/php/config.php]/returns:
> /bin/cp: cannot create regular file `/var/www/tbpl/php/config.php':
> Permission denied

As part of the Vagrant initial-setup, it attempts to copy the example config to the live one (which should get mirrored back to the sourcedir outside the VM) - not sure why this isn't working, unless the user/group under which the VM is running doesn't have permissions to the srcdir on your machine? A manual copy of config.php.example to config.php will get you past this step (and it would then be skipped for future runs too). I'll have a look and see if there's anything obvious amiss with the Vagrant config anyway :-)

Comment 7

[Andrew McCreight [:mccr8]]

Thanks!  Can you check it in, or should I?  I don't know what the landing procedure might be besides pushing to the HG repo.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Go ahead and push it :)

Comment 9

[Andrew McCreight [:mccr8]]

https://hg.mozilla.org/webtools/tbpl/rev/d4ae8aa22660

Comment 10

[Andrew McCreight [:mccr8]]

If updates to TBPL retroactively apply to old pushes (I assume they do?) then this try push has plenty of LSAN failures to see if it worked:
  https://tbpl.mozilla.org/?tree=Try&rev=a9a80c84b8b5

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Having the SUMMARY: in front of the line is a bit ugly (and could complicate starring bugs since it will likely affect what we need to put in the summary for TBPL to match). Can we just drop it so they're outputted as "LeakSanitizer: 32856 byte(s) leaked in 1 allocation(s)." ? Even better would be if we could get the test name on the same line - like "LeakSanitizer: TestAppShellSteadyState | 32856 byte(s) leaked in 1 allocation(s)." or something to that affect. I'm very much open to suggestions.

Comment 12

[Phil Ringnalda (:philor)]

LeakSanitizer | TestAppShellSteadyState | 3... rather than LeakSanitizer: - a test name is the thing between the first and second pipe when there is at least one character after the second pipe.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

(In reply to Andrew McCreight [:mccr8] from comment #9)
> https://hg.mozilla.org/webtools/tbpl/rev/d4ae8aa22660

In production :)

Let's track further improvements to this in follow-up bugs.