Closed
Bug 169770
Opened 21 years ago
Closed 19 years ago
Need DHTML performance tests running in Tinderbox
Categories
(Core Graveyard :: Tracking, defect)
Core Graveyard
Tracking
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: roc, Assigned: chofmann)
References
Details
(Keywords: perf)
Attachments
(1 file, 3 obsolete files)
5.40 KB,
patch
|
dbaron
:
review+
|
Details | Diff | Splinter Review |
We need to track DHTML performance in Tinderbox or we'll never stop the performance regressions. I have asked Markus Hübner to prepare a set of tests suitable for Tinderbox. When he's ready he'll attach them to this bug. The tests should cover the most commonly used DHTML idioms but not take too long to run (a few minutes?). There should be a single page that runs the tests and then outputs a single number (via Javascript dump()?). The tests should not be bound by clock granularity. I.e., the tests should not be trying to display more than 30-50 frames per second. We should also make the tests accessible either from this bug or somewhere at mozilla.org so that anyone can run them.
Comment 1•21 years ago
|
||
I actually have a set of pages, I'll talk with john and try to get them posted publically somewhere. They run using john's pageload harnass, that has been slightly modified.
Comment 2•21 years ago
|
||
kerz: can you outline the 'hook' that is required in each test page (i.e., the function that must be supplied to start and stop timing).
Reporter | ||
Comment 3•21 years ago
|
||
I think what we need to measure DHTML performance is to compute the harmonic mean of a set of frame rates.
Reporter | ||
Comment 4•21 years ago
|
||
I think we can do this just on the client side using a zip-file full of pages with a main page which loads each test into a frame, runs it, and collects the results.
Comment 5•21 years ago
|
||
Right now, the test will call a function each time it loads a page, called "mzDhtmlStart()" and inside that I grab the current time. When the DHTML finishes, it needs to do the time diff by grabbing the current time and comparing it to the start (end time - start time). It then calls a function called "mzDhtmlStop(time)", which passes the final time back to the pageload test. More of this could be moved into the test itself, but at the time, it was easiest to do it this way.
Comment 6•21 years ago
|
||
Will coordinate efforts with Jason and review/extend the test-suite.
Comment 7•21 years ago
|
||
I think that Marcus should create 3 types of tests. 1) Movment speed (like in http://alladyn.art.pl/gandalf/MozillaDHTML ) 2) Dynamic window (like in dhtmlcentral.com ) with scrolling text and moving box with text 3) Some 3D to 2D throw with movement And optionally - 4 - some type of dynamic cascade menu (animated)
Comment 8•21 years ago
|
||
Jason, can you post the URL of the current version accessible to all users - so that they can get an impression.
Severity: normal → major
Comment 9•21 years ago
|
||
Sure, the most current external version of the test (I'm not sure if it's the same as the latest version I have inside the Netscape firewall, as I'm out of the office for christmas) is linked from http://pageload.craptastic.org/. I believe it still needs some of the automation bits readded to it, and some other stuff.
Comment 10•21 years ago
|
||
Thx for posting. Would be fine if you could update that URL after christmas when you are back at the office.
Comment 11•21 years ago
|
||
Just had a talk with Jason, he said he is no longer working on that and it would be fine if Chris or John can go on with integrating the DHTML tests we have so far into tinderbox.
Keywords: perf
Comment 12•21 years ago
|
||
John: can you please give me a feedback about the current status of this.
Comment 13•21 years ago
|
||
URL works for me. If we want to post a number here, tbox currently searches stdout (js dump) for strings, e.g. the jrgm test dumps something like _x_x_pageload:1050 can this test dump something like this? Find me online or send me email. mcafee@netscape.com, aim:sleestack8. It should be straightforward to add this test.
Comment 14•21 years ago
|
||
The test I ran here printed this out: Test id: 3E7B95E953 Combind Results: Avg. Median: 7613 msec Minimum: 500 msec Average: 7688 msec Maximum: 80360 msec Individual Page's Results: ID PATH AVG MED MAX MIN TIMES..... 0 colorfade 2534 2515 2625 2484 2625 2484 2485 2562 2515 1 cowtools 1743 1625 1984 1579 1984 1625 1609 1579 1922 2 diagball 4709 4641 4844 4610 4813 4610 4641 4640 4844 3 fadespacing 3837 3781 4094 3750 3797 3750 3765 3781 4094 [...] What should we print out here?
Comment 15•21 years ago
|
||
over to kerz for help here, give back to me for client work.
Assignee: mcafee → kerz
Updated•21 years ago
|
Keywords: helpwanted
Comment 16•21 years ago
|
||
Is it fine for you to to have a result of the way: __dhtml_avg:1234 __dhtml_median:2345 on the summary page at the end of the test-series?
Comment 17•21 years ago
|
||
it would be best to seperate out regular html pageload and dhtml pageload scores
Comment 18•21 years ago
|
||
sure! who is capable of doing this?
Comment 19•21 years ago
|
||
mcafee, kerz is no longer working on that at all he told me - can you take this bug. should be really close to get this finally in.
Comment 20•21 years ago
|
||
marcus, those tokens should be fine, taking this from kerz.
Assignee: kerz → mcafee
Comment 21•21 years ago
|
||
I have the test in the tinderbox script, but I don't see the "__dhtml_avg:" messages, only "Document <url> loaded successfully". Has the part that prints out the messages been done yet?
Comment 22•21 years ago
|
||
Comment 23•21 years ago
|
||
The test seems to work, but I report failure since I can't scrape any tokens out of the log. Fix that, and I can get this in and fire up a build.
Comment 24•21 years ago
|
||
kerz/mcafee: could you attach the latest version of the tests here, so that someone can make the required changes? (or make that changes yourself :) )
Comment 25•21 years ago
|
||
Where is the source for this test? I don't currently have a copy, only the url in this bug.
Comment 26•21 years ago
|
||
kerz is having the latest version. @kerz: could you please upload them here.
Comment 27•21 years ago
|
||
actually we need the test working somewhere, getting a fix onto craptastic.org would be the best thing here.
Updated•21 years ago
|
Keywords: helpwanted → nsbeta1
Comment 29•21 years ago
|
||
Jason uploaded them to: http://pageload.craptastic.org/dhtml.tar.gz mcafee: can you please arrange to finish the necessary adaptions.
Updated•21 years ago
|
Comment 30•21 years ago
|
||
I need someone to own the test, this someone should also help me edit the content. I don't have time to set this up, we're gonna need a test owner anyways; someone to monitor this and decide what to do when the numbers go up, etc. I think we're ready to go, I just need some tokens spit out with a dump() call.
Reporter | ||
Comment 31•21 years ago
|
||
It sounds like the owner needs to be inside the Netscape firewall or have access to a machine running a Tinderbox ... maybe dbaron can help?
Comment 32•20 years ago
|
||
Making the necessary modifications to the tinderbox scripts doesn't require being behind the firewall. They're in mozilla/tools/tinderbox/. To set up a tinderbox, you need to do roughly the following: make a directory for the whole thing. I'll call it /builds/tinderbox/ cd /builds/tinderbox cvs co mozilla/tools/tinderbox [ with appropriate CVSROOT or -d ] mkdir trunk [or branch, or trunk-clobber, or whatever you want] cd trunk ../mozilla/tools/tinderbox/install-links [ or, for Windows, make the equivalent copies ] ./build-seamonkey.pl --example-config > tinder-config.pl Edit tinder-config.pl to make the appropriate changes (note that you must 'make' rather than 'gmake' on Windows) Create /builds/tinderbox/trunk/mozconfig Then, to start the tinderbox: [Unix] Make sure that the X server is correct (and has the necessary 'xhost +local:' or 'xhost +localhost', and run "./tinderbox depend start" or "./tinderbox clobber start") [Mac] Make sure you're at the console and not telnetted/sshed in, and run "./tinderbox depend start" or "./tinderbox clobber start" [Windows] Make a batch file that does: call vcvars32.bat [or anything else you need to do to set up env vars ] cd /builds/tinderbox/trunk build-seamonkey.pl --depend --mozconfig mozconfig [ or --clobber instead of --depend ] and then use that batch file to start the tinderbox. Once you've done that, you have a tinderbox that builds and reports to the MozillaTest tree. Then you can hack the tinderbox scripts (mainly build-seamonkey-utils.pl, but with a few changes to tinder-defaults.pl) to add code for the new tests.
Comment 33•20 years ago
|
||
The owner needs to publish and maintain a URL for the test to point at. It could be on-site or off-site.
![]() |
||
Comment 34•20 years ago
|
||
So all we need is a pretty reliable url for the tests? I could do that off my MIT web-space, methinks...
Comment 35•20 years ago
|
||
What's wrong with something under http://mozilla.org/performance/ ?
![]() |
||
Comment 36•20 years ago
|
||
Indeed. Again, I could maintain such a testsuite if that's all that's needed.
Comment 37•20 years ago
|
||
Boris, please let's try to go with anything that can be done here - it's really time to monitor DHTML performance(regressions) as some major changes are to be taken.
![]() |
||
Comment 38•20 years ago
|
||
In case you missed it, no one has gotten back to me with tests, information on adding tests to tinderbox, or anything of that sort after I volunteered to maintain the tests... dbaron is right that these tests could just live on mozilla.org. If you point me to some tests, I'll start by checking them in there; then the ball is in the court of someone with access to actual tinderboxen to point them to that test... (and tell me whether the test needs to produce output in any particular format).
Comment 39•20 years ago
|
||
No, once somebody puts the tests there, the ball is in the court of whoever wants to add support for the tests to the tinderbox scripts (which are in mozilla/tools/tinderbox/).
![]() |
||
Comment 40•20 years ago
|
||
Ah, ok. I didn't realize that the scripts could just be modified by anyone with CVS access... That's the sort of information I needed. ;) I can probably change the scripts once we have tests.
Comment 41•20 years ago
|
||
You have the test-suite sent by jason?
![]() |
||
Comment 42•20 years ago
|
||
Comment 43•20 years ago
|
||
Meaby i missed something but I wasn't able to run those tests from ./pages
directory from within attachment 142836 [details].
It missess settime() function in some of them, one had faked adress (not exists
in real)...
I also wasn't able to find very important for me tests about Opacity changes and
it's influence on movement speed nor anthing related to tests for with(),DOM
nodes accessing, overwriting values in properties and other important JS methods
for DHTML.
![]() |
||
Comment 45•20 years ago
|
||
I just attached what kerz sent me. If you have other tests you want, feel free to put them in the right format and check them in. I won't be able to work on this in the foreseeable future (and frankly, no longer have any interest in doing so even if I could).
Assignee: bzbarsky → chofmann
Comment 46•20 years ago
|
||
It really would be good / necessary if we could at least get the tests I compiled with kerz into tinderbox. Finding regressions is a tough task anyway - and having that into tinderbox would make things so much easier. @Boris: would it be possible for you to get these tests into tinderbox resp. if not who would be able to help?
![]() |
||
Comment 47•19 years ago
|
||
I've checked in the tests themselves at http://www.mozilla.org/performance/test-cases/dhtml/ The tests are not final yet: I'm still fiddling with the algorithm a bit, I'm planning to remove a few of the tests because they're pointless and I'm open to suggestions of more tests.
Attachment #124403 -
Attachment is obsolete: true
Attachment #142836 -
Attachment is obsolete: true
![]() |
||
Comment 48•19 years ago
|
||
Comment on attachment 154035 [details] [diff] [review] Patch to tinderbox scripts to run DHTML tests bryner, would you review?
Attachment #154035 -
Flags: review?(bryner)
Comment 49•19 years ago
|
||
Comment on attachment 154035 [details] [diff] [review] Patch to tinderbox scripts to run DHTML tests >+ print_log "TinderboxPrint:" . >+ "<a title=\"DHTML time\" href=\"http://$Settings::results_server/graph/query.cgi?testname=dhtml&tbox=" . >+ ::hostname() . "&autoscale=1&days=7&avg=1&showpoint=$time,$dhtml_time\">Tdhtml:$dhtml_time" . "ms</a>\n"; >+ >+ if($Settings::TestsPhoneHome) { >+ send_results_to_server($dhtml_time, $dhtml_time, "dhtml", ::hostname()); >+ } You shouldn't print links to the graph server if TestsPhoneHome is false. (BloatTest and BloatTest2 do this right; the other tests don't.)
![]() |
||
Updated•19 years ago
|
Attachment #154035 -
Flags: review?(bryner) → review-
![]() |
||
Comment 50•19 years ago
|
||
Attachment #154035 -
Attachment is obsolete: true
![]() |
||
Updated•19 years ago
|
Attachment #155143 -
Flags: superreview?(dbaron)
Attachment #155143 -
Flags: review?(dbaron)
Comment 51•19 years ago
|
||
Comment on attachment 155143 [details] [diff] [review] Patch with that fixed >+ send_results_to_server($dhtml_time, $dhtml_time, "dhtml", ::hostname()); The second argument should just be "--" instead of $dhtml_time. Other than that, r=dbaron
Attachment #155143 -
Flags: superreview?(dbaron)
Attachment #155143 -
Flags: review?(dbaron)
Attachment #155143 -
Flags: review+
![]() |
||
Comment 52•19 years ago
|
||
Luna's running this test now. Marking fixed, though I may tweak the tests some going forward...
Status: NEW → RESOLVED
Closed: 19 years ago
Resolution: --- → FIXED
Updated•7 years ago
|
Product: Core → Core Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•