466104 - make mochitests not generate and then grovel through extra HTML when running directories

Status: RESOLVED FIXED

(Testing :: Mochitest, defect, P4)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: patch

When running directories worth of mochitests, rather than single files, the single-file mochitests run inside an iframe, print out their usual HTML output, and then the parent outside the iframe grovels through the generated HTML (rather than the data used to generate it) in a somewhat slow way in order to add to its result totals.

We can speed this up significantly by:
 * skipping the generation of the HTML in the inner frame
 * making the parent runner use the underlying data instead

This makes a significant difference (I think especially for tests where a single test file has a large number of tests in it).

Here's the patch as I have it now (which I wrote back at the beginning of October, but didn't perf-test until today).  I might want to tweak a little bit before requesting review.

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: performance test results

After one run to warm up the disk, I ran six runs of the mochitests in layout/style (see command in this file), alternating between runs with and without attachment 349360 [details] [diff] [review].  (I excluded a single test due to bug 466102, but included a test or two that only exist in my tree as well.)  This was on a Linux AMD64 debug+optimized Firefox.

This shows that the real clock time improved from about 65.5 seconds to about 44 seconds.

(I suspect layout/style will show more improvement than directories where the tests have fewer assertions per file, but I haven't checked.)

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I need to fix that this breaks the chrome tests before relanding.  (browser tests work, though.)

Comment 3

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

To be clear, I landed this on mozilla-central (and thus also mozilla-1.9.1):
http://hg.mozilla.org/mozilla-central/rev/eac2f8f3cd29
and then backed it out because it broke mochichrome and chrome tests:
http://hg.mozilla.org/mozilla-central/rev/33d2527d796b
http://hg.mozilla.org/mozilla-central/rev/2529fb8e20d1

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: patch

This version just passes SimpleTest._tests through, since we've already gone to the trouble of unwrapping parentRunner.

It does work with the chrome tests this time.  I also tested regular, a11y, and browser.

Comment 5

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Landed on mozilla-central:
http://hg.mozilla.org/mozilla-central/rev/957a4fed14af

I'll plan to land on 1.9.1 next time I do landings there, as well.

Comment 6

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Had to back out again due to leaks (which we should really print to the tinderbox to show why there was a failure).

(I'm used to ignoring the leaks output because I have a lot more classes instrumented in my tree...)

Comment 7

[Jeff Walden [:Waldo]]

Could you file a bug listing those classes so we can individually enable them as we fix whatever leaks you often hit?

Comment 8

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

The patch in question is from bug 456864, so that's not trivial.

Comment 9

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: performance results from last landing

I can get useful performance results from the Mac tinderboxes, since they're not VMs.  There wasn't much change, although if this was a progressive leak there could have been slowdown due to the leak.

Comment 10

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: leak-gauge output for full mochitest run

Comment 11

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I should try this again now that bug 501577 landed.

Comment 12

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Nope, still leaks on mochitest-plain.  It seems that the test that leaks is content/xbl/test/test_bug398492.xul .

Comment 13

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

The leaks seem to have gone away, so I landed this.

http://hg.mozilla.org/mozilla-central/rev/d242d4ecdbff