1385311 - Make sure ASAN and LSAN failures turn the job orange

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Andrew Halberstadt [:ahal]]

I think LSAN failures will still turn the job orange, but there are cases of ASAN failures remaining green. Here's an example:
https://treeherder.mozilla.org/logviewer.html#?job_id=118977611&repo=try&lineNumber=2858

This was from a try push where I added a check for "ERROR: AddressSanitizer" to make the job orange:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f06adacf7fa6275a44022c632d1eff4b22b627b5

I'm still not sure if that failure is intermittent or not, but it'll need to be fixed/disabled before a patch here can land.

We should also run the mochitest selftests with an ASAN build and add a test to ensure the job turns orange. Bill, do you know how to reliably reproduce any kind of ASAN failure from a mochitest plain test (or direct me to someone who might know)?

Comment 1

[Andrew McCreight [:mccr8]]

You could probably do what BrowserTestUtils.crashBrowser() does and use ctypes to do a null deref. You'd have to use special powers to access it. There may even be some way to use ctypes to induce a use-after-free, but getting any testing would obviously be an improvement.

Comment 2

[Andrew Halberstadt [:ahal]]

Thanks, I'll take a stab at a test next week.

I was intending the test to be more for whether the task properly turns orange, rather than a test for the AddressSanitizer itself. So from the mozharness/mochitest perspective, it wouldn't be valuable to test more than 1 type of ASAN error. Of course we could expand the test's scope to include ASAN if we wanted to.

Comment 3

[Andrew McCreight [:mccr8]]

Yeah, but you could imagine if different errors caused different output then you'd need multiple tests. I guess they all just do the same thing right now.

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1385311 - Make sure AddressSanitizer errors turn the task orange,

Review commit: https://reviewboard.mozilla.org/r/164462/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/164462/

Comment 5

[Andrew Halberstadt [:ahal]]

(sigh, I need some more coffee.. sorry about all the bug spam)

Luckily it looks like there weren't any real problems that slipped in because of this, just needed to skip those two tests that depend on crashreporter.

Here's a good looking try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=02baffe6bc221ba883117d60e92aebee3bb5e85a

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8893364 [details]
Bug 1385311 - Make sure AddressSanitizer errors turn the task orange,

https://reviewboard.mozilla.org/r/164462/#review169836

nice fix

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b804014faa13
Make sure AddressSanitizer errors turn the task orange, r=jmaher

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/b804014faa13