Closed Bug 466104 Opened 14 years ago Closed 13 years ago

make mochitests not generate and then grovel through extra HTML when running directories


(Testing :: Mochitest, defect, P4)



(Not tracked)



(Reporter: dbaron, Assigned: dbaron)



(Keywords: perf)


(4 files, 1 obsolete file)

Attached patch patch (obsolete) — Splinter Review
When running directories worth of mochitests, rather than single files, the single-file mochitests run inside an iframe, print out their usual HTML output, and then the parent outside the iframe grovels through the generated HTML (rather than the data used to generate it) in a somewhat slow way in order to add to its result totals.

We can speed this up significantly by:
 * skipping the generation of the HTML in the inner frame
 * making the parent runner use the underlying data instead

This makes a significant difference (I think especially for tests where a single test file has a large number of tests in it).

Here's the patch as I have it now (which I wrote back at the beginning of October, but didn't perf-test until today).  I might want to tweak a little bit before requesting review.
After one run to warm up the disk, I ran six runs of the mochitests in layout/style (see command in this file), alternating between runs with and without attachment 349360 [details] [diff] [review].  (I excluded a single test due to bug 466102, but included a test or two that only exist in my tree as well.)  This was on a Linux AMD64 debug+optimized Firefox.

This shows that the real clock time improved from about 65.5 seconds to about 44 seconds.

(I suspect layout/style will show more improvement than directories where the tests have fewer assertions per file, but I haven't checked.)
Attachment #349360 - Flags: review?(sayrer)
Component: Testing → Mochitest
Product: Core → Testing
QA Contact: testing → mochitest
Version: Trunk → unspecified
Attachment #349360 - Flags: review?(sayrer) → review+
I need to fix that this breaks the chrome tests before relanding.  (browser tests work, though.)
To be clear, I landed this on mozilla-central (and thus also mozilla-1.9.1):
and then backed it out because it broke mochichrome and chrome tests:
Attached patch patchSplinter Review
This version just passes SimpleTest._tests through, since we've already gone to the trouble of unwrapping parentRunner.

It does work with the chrome tests this time.  I also tested regular, a11y, and browser.
Attachment #349360 - Attachment is obsolete: true
Attachment #350238 - Flags: review?(sayrer)
Attachment #350238 - Flags: review?(sayrer) → review+
Landed on mozilla-central:

I'll plan to land on 1.9.1 next time I do landings there, as well.
Closed: 14 years ago
Resolution: --- → FIXED
Had to back out again due to leaks (which we should really print to the tinderbox to show why there was a failure).

(I'm used to ignoring the leaks output because I have a lot more classes instrumented in my tree...)
Resolution: FIXED → ---
Could you file a bug listing those classes so we can individually enable them as we fix whatever leaks you often hit?
The patch in question is from bug 456864, so that's not trivial.
I can get useful performance results from the Mac tinderboxes, since they're not VMs.  There wasn't much change, although if this was a progressive leak there could have been slowdown due to the leak.
Depends on: 452834
I should try this again now that bug 501577 landed.
Nope, still leaks on mochitest-plain.  It seems that the test that leaks is content/xbl/test/test_bug398492.xul .
The leaks seem to have gone away, so I landed this.
Closed: 14 years ago13 years ago
Priority: -- → P4
Resolution: --- → FIXED
Target Milestone: --- → mozilla1.9.3a4
Flags: in-testsuite-
Version: unspecified → Trunk
You need to log in before you can comment on or make changes to this bug.