Open Bug 1229578 Opened 4 years ago Updated 3 years ago

run basic automated tests of our test infrastructure

Categories

(Testing :: General, defect)

defect
Not set

Tracking

(firefox45 affected)

Tracking Status
firefox45 --- affected

People

(Reporter: dbaron, Unassigned)

References

(Depends on 1 open bug)

Details

As a followup to bug 1229549, I was thinking that we should have some basic automated tests of our testing infrastructure.

What I imagine is something like this.  We have a system that, once a day/week/whatever, pushes a few try runs that are designed to have failures, and then checks that the results are as expected.  These runs would be things like:

 * push a changeset with a startup crash, and make sure all harnesses report it as a crash and turn orange, with PROCESS-CRASH and the correct signature in the topline diagnostic (i.e., testing bug 1229549)

 * push a changeset with a memory leak, and make sure all harnesses that are expected to report failures for leaks turn orange, and report a correct diagnostic for the leak (i.e., bug 1045316)

 * push a changeset with a simple test failure of each type in each harness, and make sure all of the affected tests on all platforms turn orange, and have a correct diagnostic

 * probably similar for a hang during a test, a startup hang, and a crash during a test

Given the current state of things, this system would obviously need to have a mechanism for recording known failures.
+1, this is way too long overdue. See also bug 1048884.. though your description is better so I'll dupe that one to this. Sadly, there's always something that seems to take precedence over this.

From the other bug, ted points out that the only model we have to copy is 'selftests.py' in xpcshell. Though that didn't catch the missing stack in dbaron's startup crash test either.
Duplicate of this bug: 1048884
Depends on: 1048446
For testing crashes and memory leaks I’m not sure we have to actually have to push a code change to harnesses that support evaluating code in chrome context.

I imagine it would be sufficient to have a small JS library that we could use to instrument Gecko to trigger a tab/e10s browser crash, a parent process crash, a memory leak, and so on.  Possibly using the manifest expected result system we could indicate to the job that a particular test is meant to crash?
You need to log in before you can comment on or make changes to this bug.