771105 - Allow expected leaks in crashtests

Status: RESOLVED WONTFIX

(Testing :: Reftest, enhancement)

Description

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

(I'm filing this under reftests because AFAIK, crashtests are basically a special type of reftest.)

In bug 766025, there's an assertion failure, but the only test-case available to regression-test the failure also leaks.  So a) I can't regression-test the crash (because the only test available will fail regardless of the crash), and b) I can't test the leak as an expected fail so that we'll know if it's fixed and have the test checked in.

Could we have some way to mark a test as expected to leak, like we have expected assertions?

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Not really, because leaks can't easily be attributed to a single test like assertions can.

Can we just fix the leak?  What's the bug # for that?

Comment 2

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

Ah, good point.  Probably this has to be INVALID, then.  There's no separate bug for the leak yet -- I could look into fixing it, but I have enough other stuff on my plate now that I don't expect to be able to get around to it soon.

Comment 3

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Yeah, I don't think there's a reasonable way to do this.

Given the current amount of test coverage, I'd rather not have a test for a single bug than give up the leak coverage for all our crashtests.