925873 - Make ASan failures something tbpl will recognize as an error, and an error in a particular test

Status: RESOLVED FIXED

Description

[Phil Ringnalda (:philor)]

Things that tbpl recognizes so they can be starred are logged in the form "{possible noise from mozharness} TEST-UNEXPECTED-FAIL | this/is/the/test_filename | what failed was this".

ASan failures are logged in the form "{noise from mozharness} ==2325==ERROR: AddressSanitizer: heap-use-after-free on address 0x60600027e0f8 at pc 0x7f104e3c04fd bp 0x7fff9baf4d50 sp 0x7fff9baf4d48".

That means that for a failure like bug 921622, we have to know to open a full log like https://tbpl.mozilla.org/php/getParsedLog.php?id=28903669&tree=Fx-Team, scroll down to the bottom and then back up to the end of the runtests step, find the failure, find the test it was in, search for the bug, copy-paste the link to the log, but for a failure like https://tbpl.mozilla.org/php/getParsedLog.php?id=28987951&tree=Mozilla-Inbound where it was in the first of the three suites in mochitest-other, we have to remember (assuming this is true of all of them) that "AddressSanitizer:" is in ASan failure messages to even find the error.

Comment 1

[Ed Morley [:emorley]]

Thank you for filing this Phil, guess this answers my/nigelb's question from earlier :-)

Comment 2

[Phil Ringnalda (:philor)]

Also decidedly hard to find: https://tbpl.mozilla.org/php/getParsedLog.php?id=29393777&tree=Profiling contains two ASan failures, one OOP (findable with ==2357==ERROR:) which didn't even stop the run or affect the success/fail count or mark that part of the suite as having failed, and a startup failure in a11y, (==2676==ERROR:) which did at least wind up calling that part of the suite a T-FAIL.

Comment 3

[Ed Morley [:emorley]]

decoder, may you take a look at this? :-)

Comment 4

[Christian Holler (:decoder)]

First of all: Is this failing on all ASan traces you see now? If it is, then a change in automation must have regressed it.

I am looking at https://tbpl.mozilla.org/php/getParsedLog.php?id=28903669&tree=Fx-Team now and that has the same way of failing.

Comment 5

[Christian Holler (:decoder)]

Fwiw, I have a patch that should at least fix the fx-team issue and one of the traces in the earlier profiling failure. It is still unclear to me, when this started happening because I remember that we once had TBPL starrable failures on mochitests.

After talking with ted, we came to the conclusion that the mochitest framework doesn't emit a TEST-UNEXPECTED-FAIL for a non-zero exit code (which ASan will use). But any other reason that would terminate the browser without producing a minidump would cause the same failure.

I'm testing the patch for fixing that behavior right now on try.

Comment 6

[Ed Morley [:emorley]]

Great! Thank you :-)

Comment 7

[Christian Holler (:decoder)]

Created attachment 819748 [details] [diff] [review]
mochitest-exit.patch

This patches solves the problem, confirmed on try with a failing mochitest.

Comment 8

[Christian Holler (:decoder)]

Thanks for the quick review.

https://hg.mozilla.org/integration/mozilla-inbound/rev/654ffd9d6b09

Comment 9

[Ed Morley [:emorley]]

Sheriffs: Note this will change (well, lengthen) the signature of annotated summaries on TBPL for all mochitest failures where the return code != 0 (and not just ASan). So think this includes crashes etc. Not necessarily a bad thing, but just so you are aware :-)

Comment 10

[Christian Holler (:decoder)]

(In reply to Ed Morley (Mostly away until 21st October) [:edmorley UTC+1] from comment #9)
> Sheriffs: Note this will change (well, lengthen) the signature of annotated
> summaries on TBPL for all mochitest failures where the return code != 0 (and
> not just ASan). So think this includes crashes etc. Not necessarily a bad
> thing, but just so you are aware :-)

If this causes you any trouble, please let me know :)

Comment 11

[Wes Kocher (:KWierso)]

https://hg.mozilla.org/mozilla-central/rev/654ffd9d6b09