1045316 - Add end-to-end testing of our leak checking infrastructure

Status: NEW

(Testing :: General, defect)

Description

[Andrew McCreight [:mccr8]]

The leak checking infrastructure requires many parts to work, from Gecko itself, to the test harness, to buildbot, to TBPL itself.  It would be nice if we had some more testing that it is actually working.

This is not merely an academic concern, as currently both our regular leak checking (bug 1045255) and LSan (bug 1044206) appear to have silently broken.

The simplest possible approach would be to introduce an intentional intermittent leak, that fails about once a day.  Then you get some people to follow the bug, and if it goes silent for too long, you know something is wrong.  Bug 1033464 sort of served this purpose, but it was a little too intermittent so nobody noticed when it went silent.  So this is not a great approach, but it might be better than nothing for the moment.

A more comprehensive approach would maybe just test the various bits of the leak checking infrastructure, with some kind of intentional leak that the harness ignores.  I'm not sure how that would work.

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

This is a general problem with all of our testing infrastructure; we've had similar problems with regressions in the reftest harness.

I think recent events have probably shown that we should be thinking about testing our tests.  We probably don't (yet) need the next level down.  Somehow we need to avoid having elephants all the way down, though.

Comment 2

[Gregory Szorc [:gps]]

I've long wanted to introduce something like Mercurial's "t tests" for testing shell output (like mach commands). http://mercurial.selenic.com/wiki/WritingTests It even allows you to test exit codes. You are running the actual commands that users run. https://hg.mozilla.org/hgcustom/version-control-tools/file/7efa5dc02222/hgext/reviewboard/tests/test-commits-deleted-obsolescence.t

Comment 3

[(not currently active) Ted Mielczarek]

The closest thing we have to this is the xpcshell selftests:
http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/selftest.py

I've found these really useful as they allow us to test that the harness is actually doing what we intend it to do, and we can also test failures. We could probably write an equivalent set of tests for any test harness.

Comment 4

[Andrew McCreight [:mccr8]]

This is a little less fragile than it was before, as we now produce an error if a process creates a leak log, but does not actually output leak information into that log.

Bug 1121670 is another example of why having end-to-end testing is important: I accidentally broke the Treeherder starring suggestions for four months for leakcheck, so we have no idea how much various intermittent leaks happened in that period.  This isn't the end of the world, because something that leaked frequently would still get backed out by the sheriffs, and I think that if it was happening a lot, the sheriffs would have complained earlier.  This problem was only noticed because when I turned on leak checking for content processes, there was a very frequently leak, and RyanVM complained to me about it.  But it still isn't great.

Comment 6

[Eric Rahm [:erahm]]

Geoff, do you know if anyone can look into this? It's definitely bit us in the past. mccr8 suggested that just having a task that checks if any leaks have been starred in the past week might be good enough.

Comment 7

[Geoff Brown [:gbrown]]

:ahal added mochitest/reftest harness "self-tests", including some tests for leak logging. Have those tests "missed" recent issues? :ahal -- do you have thoughts (plans!?) on next steps?

https://dxr.mozilla.org/mozilla-central/rev/c291143e24019097d087f9307e59b49facaf90cb/testing/mochitest/tests/python/test_basic_mochitest_plain.py#104
https://dxr.mozilla.org/mozilla-central/rev/c291143e24019097d087f9307e59b49facaf90cb/layout/tools/reftest/selftest/test_reftest_output.py#123

Comment 8

[Andrew Halberstadt [:ahal]]

I couldn't figure out how to reliably produce a leak for testing purposes, so those tests mock out a leaklog instead of actually getting Gecko to produce one. They basically just test that the log processing stuff in mochitest/reftest is working and nothing else.

If someone can write a mochitest/reftest (or devise some other mechanism) for me that causes a leaklog to get generated, we could start using that instead of mocking it out. But beyond that I don't have any thoughts or plans.