Make ASan failures something tbpl will recognize as an error, and an error in a particular test

RESOLVED FIXED in mozilla27

Status

()

RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: philor, Assigned: decoder)

Tracking

(Blocks: 1 bug)

Trunk
mozilla27
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

5 years ago
Things that tbpl recognizes so they can be starred are logged in the form "{possible noise from mozharness} TEST-UNEXPECTED-FAIL | this/is/the/test_filename | what failed was this".

ASan failures are logged in the form "{noise from mozharness} ==2325==ERROR: AddressSanitizer: heap-use-after-free on address 0x60600027e0f8 at pc 0x7f104e3c04fd bp 0x7fff9baf4d50 sp 0x7fff9baf4d48".

That means that for a failure like bug 921622, we have to know to open a full log like https://tbpl.mozilla.org/php/getParsedLog.php?id=28903669&tree=Fx-Team, scroll down to the bottom and then back up to the end of the runtests step, find the failure, find the test it was in, search for the bug, copy-paste the link to the log, but for a failure like https://tbpl.mozilla.org/php/getParsedLog.php?id=28987951&tree=Mozilla-Inbound where it was in the first of the three suites in mochitest-other, we have to remember (assuming this is true of all of them) that "AddressSanitizer:" is in ASan failure messages to even find the error.

Comment 1

5 years ago
Thank you for filing this Phil, guess this answers my/nigelb's question from earlier :-)
(Reporter)

Comment 2

5 years ago
Also decidedly hard to find: https://tbpl.mozilla.org/php/getParsedLog.php?id=29393777&tree=Profiling contains two ASan failures, one OOP (findable with ==2357==ERROR:) which didn't even stop the run or affect the success/fail count or mark that part of the suite as having failed, and a startup failure in a11y, (==2676==ERROR:) which did at least wind up calling that part of the suite a T-FAIL.

Comment 3

5 years ago
decoder, may you take a look at this? :-)
Blocks: 831491
Flags: needinfo?(choller)
(Assignee)

Comment 4

5 years ago
First of all: Is this failing on all ASan traces you see now? If it is, then a change in automation must have regressed it.

I am looking at https://tbpl.mozilla.org/php/getParsedLog.php?id=28903669&tree=Fx-Team now and that has the same way of failing.
Flags: needinfo?(choller)
(Assignee)

Comment 5

5 years ago
Fwiw, I have a patch that should at least fix the fx-team issue and one of the traces in the earlier profiling failure. It is still unclear to me, when this started happening because I remember that we once had TBPL starrable failures on mochitests.

After talking with ted, we came to the conclusion that the mochitest framework doesn't emit a TEST-UNEXPECTED-FAIL for a non-zero exit code (which ASan will use). But any other reason that would terminate the browser without producing a minidump would cause the same failure.

I'm testing the patch for fixing that behavior right now on try.
Assignee: nobody → choller
Blocks: 863846

Comment 6

5 years ago
Great! Thank you :-)
(Assignee)

Comment 7

5 years ago
Created attachment 819748 [details] [diff] [review]
mochitest-exit.patch

This patches solves the problem, confirmed on try with a failing mochitest.
Attachment #819748 - Flags: review?(ted)
Attachment #819748 - Flags: review?(ted) → review+
(Assignee)

Comment 8

5 years ago
Thanks for the quick review.

https://hg.mozilla.org/integration/mozilla-inbound/rev/654ffd9d6b09
Status: NEW → ASSIGNED

Comment 9

5 years ago
Sheriffs: Note this will change (well, lengthen) the signature of annotated summaries on TBPL for all mochitest failures where the return code != 0 (and not just ASan). So think this includes crashes etc. Not necessarily a bad thing, but just so you are aware :-)
(Assignee)

Comment 10

5 years ago
(In reply to Ed Morley (Mostly away until 21st October) [:edmorley UTC+1] from comment #9)
> Sheriffs: Note this will change (well, lengthen) the signature of annotated
> summaries on TBPL for all mochitest failures where the return code != 0 (and
> not just ASan). So think this includes crashes etc. Not necessarily a bad
> thing, but just so you are aware :-)

If this causes you any trouble, please let me know :)
https://hg.mozilla.org/mozilla-central/rev/654ffd9d6b09
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla27
You need to log in before you can comment on or make changes to this bug.