399830 - Simple test runner for pages that cause crashes/assertions/leaks

Status: RESOLVED DUPLICATE of bug 397725

(Testing :: Reftest, defect)

Description

[Matthew Gregan [:kinetik]]

We need a test runner that quickly runs through a potentially large collection of simple tests ensuring that they do not cause crashes, assertions, or leaks.  Attached is a first pass at a test runner to do this.  It will run through around 10,000 tests in 2 minutes on my laptop.  It's mostly just a minimized clone of the drive from the existing MochiTest code, so I've left the license attribution as it was in the original code.  Perhaps I should've shared more of the code with MochiTest rather than copying it, but I'm not sure how much farther this test runner will evolve away from MochiTest.

The URL of each test file is printed to stdout before it is loaded.  There is no special test environment provided to the tests when they are run--the intention is that they are standalone tests that would work as if they were loaded into a normal browser instance.

For adding a test, it works very much like MochiTest--add a new test file to your component, add the file to the list of _CRASHTEST_FILES in Makefile.in (or clone the rule if adding a test to a component without any existing tests).  The tests are linked into @OBJDIR@/_tests/crashtests/tests/@COMPONENT_PATH@ during the build.  Building this directory can be disabled at configure time with --disable-crashtests.

To run the testsuite, cd into @OBJDIR@/_tests/crashtests and run |perl runtests.pl|, specify --help to get a list of help.  The interesting options are --autorun (starts tests immediately after initialization), --close-when-done (close browser after tests have run), and --wait-for-render (alters the next-page driver so that we wait long enough for each page to draw rather than moving on as soon as DOMContentLoaded has fired).

Comment 1

[Matthew Gregan [:kinetik]]

Attachment: patch

Comment 2

[Robert O'Callahan (:roc) (email my personal email if necessary)]

sayrer, how should we proceed here? You want to review it?

Comment 3

[:Gavin Sharp [email: [e-mail]]]

This looks like a duplicate of bug 397725...

Comment 4

[Jeff Walden [:Waldo]]

Getting something that works is, of course, a good thing, but there's a lot of similarity between what's here and what's in testing/mochitest, of course.  Would it be possible to move the files back into testing/mochitest to avoid exact duplicates where possible, and when the files do differ, naming the crashing one *-crash.*?  I'd just like to make it easier to consolidate the two in the future.  (I don't think there'll be that much drift between mochitest and crashtest, to be honest.)

Also, I'd prefer not adding an option for --disable-crashtests.  The added code that's downloaded is negligible, and I don't think we should make it possible for people to think of tests as less important than the code they exercise.  Fewer options for configuring a build also mean less cognitive overhead in setting up a build, too.

As for dup, yeah, probably, but we have a patch here, so I'm fine with just using this one.

Comment 5

[Rob Campbell [:rc] (:robcee)]

(In reply to comment #3)
> This looks like a duplicate of bug 397725...

It certainly *feels* like a dupe, but it has a patch in it...

Looking briefly, it looks like a copy of Sayrer's and Gavin's tools to run mochitests in browser chrome. CC'ing Jonas to ask if this would be sufficient for his requirements in bug 397725.

Comment 6

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

See also bug 397725 comment 7.

Comment 7

[Matthew Gregan [:kinetik]]

Yup, this is a dupe of bug 397725.  I wasn't aware of that bug when I was working on this.  I'll look at finishing up the manifest work in the other bug unless someone beats me to it.

Comment 8

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Thanks Matthew, sorry about the duplicated work.