1229578 - run basic automated tests of our test infrastructure

Status: NEW

(Testing :: General, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

As a followup to bug 1229549, I was thinking that we should have some basic automated tests of our testing infrastructure.

What I imagine is something like this.  We have a system that, once a day/week/whatever, pushes a few try runs that are designed to have failures, and then checks that the results are as expected.  These runs would be things like:

 * push a changeset with a startup crash, and make sure all harnesses report it as a crash and turn orange, with PROCESS-CRASH and the correct signature in the topline diagnostic (i.e., testing bug 1229549)

 * push a changeset with a memory leak, and make sure all harnesses that are expected to report failures for leaks turn orange, and report a correct diagnostic for the leak (i.e., bug 1045316)

 * push a changeset with a simple test failure of each type in each harness, and make sure all of the affected tests on all platforms turn orange, and have a correct diagnostic

 * probably similar for a hang during a test, a startup hang, and a crash during a test

Given the current state of things, this system would obviously need to have a mechanism for recording known failures.

Comment 1

[Andrew Halberstadt [:ahal]]

+1, this is way too long overdue. See also bug 1048884.. though your description is better so I'll dupe that one to this. Sadly, there's always something that seems to take precedence over this.

From the other bug, ted points out that the only model we have to copy is 'selftests.py' in xpcshell. Though that didn't catch the missing stack in dbaron's startup crash test either.

Comment 3

[Andreas Tolfsen ❲:ato❳]

For testing crashes and memory leaks I’m not sure we have to actually have to push a code change to harnesses that support evaluating code in chrome context.

I imagine it would be sufficient to have a small JS library that we could use to instrument Gecko to trigger a tab/e10s browser crash, a parent process crash, a memory leak, and so on.  Possibly using the manifest expected result system we could indicate to the job that a particular test is meant to crash?