Closed Bug 169770 Opened 23 years ago Closed 21 years ago

Need DHTML performance tests running in Tinderbox

Categories

(Core Graveyard :: Tracking, defect)

Product:
Core Graveyard
Component:
Tracking
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: roc, Assigned: chofmann)

References

Details

(Keywords: perf)

Attachments

(1 file, 3 obsolete files)

Patch with that fixed
5.40 KB, patch
dbaron
: review+
Details | Diff | Splinter Review
We need to track DHTML performance in Tinderbox or we'll never stop the performance regressions. I have asked Markus Hübner to prepare a set of tests suitable for Tinderbox. When he's ready he'll attach them to this bug. The tests should cover the most commonly used DHTML idioms but not take too long to run (a few minutes?). There should be a single page that runs the tests and then outputs a single number (via Javascript dump()?). The tests should not be bound by clock granularity. I.e., the tests should not be trying to display more than 30-50 frames per second. We should also make the tests accessible either from this bug or somewhere at mozilla.org so that anyone can run them.
I actually have a set of pages, I'll talk with john and try to get them posted publically somewhere. They run using john's pageload harnass, that has been slightly modified.
kerz: can you outline the 'hook' that is required in each test page (i.e., the function that must be supplied to start and stop timing).
I think what we need to measure DHTML performance is to compute the harmonic mean of a set of frame rates.
I think we can do this just on the client side using a zip-file full of pages with a main page which loads each test into a frame, runs it, and collects the results.
Right now, the test will call a function each time it loads a page, called "mzDhtmlStart()" and inside that I grab the current time. When the DHTML finishes, it needs to do the time diff by grabbing the current time and comparing it to the start (end time - start time). It then calls a function called "mzDhtmlStop(time)", which passes the final time back to the pageload test. More of this could be moved into the test itself, but at the time, it was easiest to do it this way.
Will coordinate efforts with Jason and review/extend the test-suite.
I think that Marcus should create 3 types of tests. 1) Movment speed (like in http://alladyn.art.pl/gandalf/MozillaDHTML ) 2) Dynamic window (like in dhtmlcentral.com ) with scrolling text and moving box with text 3) Some 3D to 2D throw with movement And optionally - 4 - some type of dynamic cascade menu (animated)
Jason, can you post the URL of the current version accessible to all users - so that they can get an impression.
Severity: normal → major
Sure, the most current external version of the test (I'm not sure if it's the same as the latest version I have inside the Netscape firewall, as I'm out of the office for christmas) is linked from http://pageload.craptastic.org/. I believe it still needs some of the automation bits readded to it, and some other stuff.
Thx for posting. Would be fine if you could update that URL after christmas when you are back at the office.
Just had a talk with Jason, he said he is no longer working on that and it would be fine if Chris or John can go on with integrating the DHTML tests we have so far into tinderbox.
Keywords: perf
John: can you please give me a feedback about the current status of this.
URL works for me. If we want to post a number here, tbox currently searches stdout (js dump) for strings, e.g. the jrgm test dumps something like _x_x_pageload:1050 can this test dump something like this? Find me online or send me email. mcafee@netscape.com, aim:sleestack8. It should be straightforward to add this test.
The test I ran here printed this out: Test id: 3E7B95E953 Combind Results: Avg. Median: 7613 msec Minimum: 500 msec Average: 7688 msec Maximum: 80360 msec Individual Page's Results: ID PATH AVG MED MAX MIN TIMES..... 0 colorfade 2534 2515 2625 2484 2625 2484 2485 2562 2515 1 cowtools 1743 1625 1984 1579 1984 1625 1609 1579 1922 2 diagball 4709 4641 4844 4610 4813 4610 4641 4640 4844 3 fadespacing 3837 3781 4094 3750 3797 3750 3765 3781 4094 [...] What should we print out here?
over to kerz for help here, give back to me for client work.
Assignee: mcafee → kerz
Keywords: helpwanted
Blocks: 203448
Is it fine for you to to have a result of the way: __dhtml_avg:1234 __dhtml_median:2345 on the summary page at the end of the test-series?
it would be best to seperate out regular html pageload and dhtml pageload scores
sure! who is capable of doing this?
mcafee, kerz is no longer working on that at all he told me - can you take this bug. should be really close to get this finally in.
marcus, those tokens should be fine, taking this from kerz.
Assignee: kerz → mcafee
I have the test in the tinderbox script, but I don't see the "__dhtml_avg:" messages, only "Document <url> loaded successfully". Has the part that prints out the messages been done yet?
The test seems to work, but I report failure since I can't scrape any tokens out of the log. Fix that, and I can get this in and fire up a build.
kerz/mcafee: could you attach the latest version of the tests here, so that someone can make the required changes? (or make that changes yourself :) )
Where is the source for this test? I don't currently have a copy, only the url in this bug.
kerz is having the latest version. @kerz: could you please upload them here.
actually we need the test working somewhere, getting a fix onto craptastic.org would be the best thing here.
Keywords: helpwanted → nsbeta1
adt: nsbeta1-
Keywords: nsbeta1 → nsbeta1-
Jason uploaded them to: http://pageload.craptastic.org/dhtml.tar.gz mcafee: can you please arrange to finish the necessary adaptions.
Keywords: nsbeta1- → nsbeta1
I need someone to own the test, this someone should also help me edit the content. I don't have time to set this up, we're gonna need a test owner anyways; someone to monitor this and decide what to do when the numbers go up, etc. I think we're ready to go, I just need some tokens spit out with a dump() call.
It sounds like the owner needs to be inside the Netscape firewall or have access to a machine running a Tinderbox ... maybe dbaron can help?
Making the necessary modifications to the tinderbox scripts doesn't require being behind the firewall. They're in mozilla/tools/tinderbox/. To set up a tinderbox, you need to do roughly the following: make a directory for the whole thing. I'll call it /builds/tinderbox/ cd /builds/tinderbox cvs co mozilla/tools/tinderbox [ with appropriate CVSROOT or -d ] mkdir trunk [or branch, or trunk-clobber, or whatever you want] cd trunk ../mozilla/tools/tinderbox/install-links [ or, for Windows, make the equivalent copies ] ./build-seamonkey.pl --example-config > tinder-config.pl Edit tinder-config.pl to make the appropriate changes (note that you must 'make' rather than 'gmake' on Windows) Create /builds/tinderbox/trunk/mozconfig Then, to start the tinderbox: [Unix] Make sure that the X server is correct (and has the necessary 'xhost +local:' or 'xhost +localhost', and run "./tinderbox depend start" or "./tinderbox clobber start") [Mac] Make sure you're at the console and not telnetted/sshed in, and run "./tinderbox depend start" or "./tinderbox clobber start" [Windows] Make a batch file that does: call vcvars32.bat [or anything else you need to do to set up env vars ] cd /builds/tinderbox/trunk build-seamonkey.pl --depend --mozconfig mozconfig [ or --clobber instead of --depend ] and then use that batch file to start the tinderbox. Once you've done that, you have a tinderbox that builds and reports to the MozillaTest tree. Then you can hack the tinderbox scripts (mainly build-seamonkey-utils.pl, but with a few changes to tinder-defaults.pl) to add code for the new tests.
The owner needs to publish and maintain a URL for the test to point at. It could be on-site or off-site.
So all we need is a pretty reliable url for the tests? I could do that off my MIT web-space, methinks...
Indeed. Again, I could maintain such a testsuite if that's all that's needed.
Boris, please let's try to go with anything that can be done here - it's really time to monitor DHTML performance(regressions) as some major changes are to be taken.
In case you missed it, no one has gotten back to me with tests, information on adding tests to tinderbox, or anything of that sort after I volunteered to maintain the tests... dbaron is right that these tests could just live on mozilla.org. If you point me to some tests, I'll start by checking them in there; then the ball is in the court of someone with access to actual tinderboxen to point them to that test... (and tell me whether the test needs to produce output in any particular format).
No, once somebody puts the tests there, the ball is in the court of whoever wants to add support for the tests to the tinderbox scripts (which are in mozilla/tools/tinderbox/).
Ah, ok. I didn't realize that the scripts could just be modified by anyone with CVS access... That's the sort of information I needed. ;) I can probably change the scripts once we have tests.
You have the test-suite sent by jason?
Attached file tests (obsolete) —
Meaby i missed something but I wasn't able to run those tests from ./pages directory from within attachment 142836 [details]. It missess settime() function in some of them, one had faked adress (not exists in real)... I also wasn't able to find very important for me tests about Opacity changes and it's influence on movement speed nor anthing related to tests for with(),DOM nodes accessing, overwriting values in properties and other important JS methods for DHTML.
-> Boris
Assignee: mcafee → bzbarsky
I just attached what kerz sent me. If you have other tests you want, feel free to put them in the right format and check them in. I won't be able to work on this in the foreseeable future (and frankly, no longer have any interest in doing so even if I could).
Assignee: bzbarsky → chofmann
It really would be good / necessary if we could at least get the tests I compiled with kerz into tinderbox. Finding regressions is a tough task anyway - and having that into tinderbox would make things so much easier. @Boris: would it be possible for you to get these tests into tinderbox resp. if not who would be able to help?
I've checked in the tests themselves at http://www.mozilla.org/performance/test-cases/dhtml/ The tests are not final yet: I'm still fiddling with the algorithm a bit, I'm planning to remove a few of the tests because they're pointless and I'm open to suggestions of more tests.
Attachment #124403 - Attachment is obsolete: true
Attachment #142836 - Attachment is obsolete: true
Comment on attachment 154035 [details] [diff] [review] Patch to tinderbox scripts to run DHTML tests bryner, would you review?
Attachment #154035 - Flags: review?(bryner)
Comment on attachment 154035 [details] [diff] [review] Patch to tinderbox scripts to run DHTML tests >+ print_log "TinderboxPrint:" . >+ "<a title=\"DHTML time\" href=\"http://$Settings::results_server/graph/query.cgi?testname=dhtml&tbox=" . >+ ::hostname() . "&autoscale=1&days=7&avg=1&showpoint=$time,$dhtml_time\">Tdhtml:$dhtml_time" . "ms</a>\n"; >+ >+ if($Settings::TestsPhoneHome) { >+ send_results_to_server($dhtml_time, $dhtml_time, "dhtml", ::hostname()); >+ } You shouldn't print links to the graph server if TestsPhoneHome is false. (BloatTest and BloatTest2 do this right; the other tests don't.)
Attachment #154035 - Flags: review?(bryner) → review-
Attachment #154035 - Attachment is obsolete: true
Attachment #155143 - Flags: superreview?(dbaron)
Attachment #155143 - Flags: review?(dbaron)
Comment on attachment 155143 [details] [diff] [review] Patch with that fixed >+ send_results_to_server($dhtml_time, $dhtml_time, "dhtml", ::hostname()); The second argument should just be "--" instead of $dhtml_time. Other than that, r=dbaron
Attachment #155143 - Flags: superreview?(dbaron)
Attachment #155143 - Flags: review?(dbaron)
Attachment #155143 - Flags: review+
Luna's running this test now. Marking fixed, though I may tweak the tests some going forward...
Status: NEW → RESOLVED
Closed: 21 years ago
Resolution: --- → FIXED
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: