Closed Bug 169770 Opened 21 years ago Closed 19 years ago

Need DHTML performance tests running in Tinderbox

Categories

(Core Graveyard :: Tracking, defect)

Product:
Core Graveyard
Component:
Tracking
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: roc, Assigned: chofmann)

References

Details

(Keywords: perf)

Attachments

(1 file, 3 obsolete files)

Patch with that fixed
5.40 KB, patch
dbaron
: review+
Details | Diff | Splinter Review 
We need to track DHTML performance in Tinderbox or we'll never stop the
performance regressions.

I have asked Markus Hübner to prepare a set of tests suitable for Tinderbox.
When he's ready he'll attach them to this bug.

The tests should cover the most commonly used DHTML idioms but not take too long
to run (a few minutes?). There should be a single page that runs the tests and
then outputs a single number (via Javascript dump()?).

The tests should not be bound by clock granularity. I.e., the tests should not
be trying to display more than 30-50 frames per second.

We should also make the tests accessible either from this bug or somewhere at
mozilla.org so that anyone can run them.
I actually have a set of pages, I'll talk with john and try to get them posted
publically somewhere.  They run using john's pageload harnass, that has been
slightly modified.
kerz: can you outline the 'hook' that is required in each test page (i.e., 
the function that must be supplied to start and stop timing).
I think what we need to measure DHTML performance is to compute the harmonic
mean of a set of frame rates.
I think we can do this just on the client side using a zip-file full of pages
with a main page which loads each test into a frame, runs it, and collects the
results.
Right now, the test will call a function each time it loads a page, called
"mzDhtmlStart()" and inside that I grab the current time.  When the DHTML
finishes, it needs to do the time diff by grabbing the current time and
comparing it to the start (end time - start time).  It then calls a function
called "mzDhtmlStop(time)", which passes the final time back to the pageload
test.  More of this could be moved into the test itself, but at the time, it was
easiest to do it this way.
Will coordinate efforts with Jason and review/extend the test-suite.
I think that Marcus should create 3 types of tests.
1) Movment speed (like in http://alladyn.art.pl/gandalf/MozillaDHTML )
2) Dynamic window (like in dhtmlcentral.com ) with scrolling text and moving box
with text
3) Some 3D to 2D throw with movement

And optionally - 4 - some type of dynamic cascade menu (animated)
Jason, can you post the URL of the current version accessible to all users - 
so that they can get an impression.
Severity: normal → major
Sure, the most current external version of the test (I'm not sure if it's the
same as the latest version I have inside the Netscape firewall, as I'm out of
the office for christmas) is linked from http://pageload.craptastic.org/.  I
believe it still needs some of the automation bits readded to it, and some other
stuff.
Thx for posting. Would be fine if you could update that URL after christmas 
when you are back at the office.

Just had a talk with Jason, he said he is no longer working on that and it 
would be fine if Chris or John can go on with integrating the DHTML tests we 
have so far into tinderbox.
Keywords: perf
John: can you please give me a feedback about the current status of this.

URL works for me.  If we want to post a number here, tbox currently searches
stdout (js dump) for strings, e.g. the jrgm test dumps something like

  _x_x_pageload:1050

can this test dump something like this?  Find me online or send me email.
mcafee@netscape.com, aim:sleestack8.  It should be straightforward to add
this test.

The test I ran here printed this out:

Test id: 3E7B95E953
Combind Results: Avg. Median: 7613 msec Minimum: 500 msec
Average: 7688 msec Maximum: 80360 msec
Individual Page's Results:
ID PATH AVG MED MAX MIN TIMES.....
0 colorfade 2534 2515 2625 2484 2625 2484 2485 2562 2515
1 cowtools 1743 1625 1984 1579 1984 1625 1609 1579 1922
2 diagball 4709 4641 4844 4610 4813 4610 4641 4640 4844
3 fadespacing 3837 3781 4094 3750 3797 3750 3765 3781 4094
[...]

What should we print out here?
over to kerz for help here, give back to me for client work.
Assignee: mcafee → kerz
Keywords: helpwanted
Blocks: 203448
Is it fine for you to to have a result of the way:
__dhtml_avg:1234
__dhtml_median:2345
on the summary page at the end of the test-series?

it would be best to seperate out regular html pageload and dhtml pageload scores
sure! who is capable of doing this?

mcafee, kerz is no longer working on that at all he told me - can you take 
this bug. should be really close to get this finally in.
marcus, those tokens should be fine, taking this from kerz.
Assignee: kerz → mcafee
I have the test in the tinderbox script, but I don't see the "__dhtml_avg:"
messages, only "Document <url> loaded successfully".  Has the part that
prints out the messages been done yet?

The test seems to work, but I report failure since I can't scrape any tokens out
of the log.  Fix that, and I can get this in and fire up a build.

kerz/mcafee: could you attach the latest version of the tests here, so that
someone can make the required changes? (or make that changes yourself :) )
Where is the source for this test?  I don't currently have a copy, only the url
in this bug.
kerz is having the latest version.
@kerz: could you please upload them here.

actually we need the test working somewhere, getting a fix onto
craptastic.org would be the best thing here.
Keywords: helpwanted → nsbeta1
adt: nsbeta1-
Keywords: nsbeta1 → nsbeta1-
Jason uploaded them to: http://pageload.craptastic.org/dhtml.tar.gz
mcafee: can you please arrange to finish the necessary adaptions.
Keywords: nsbeta1- → nsbeta1
I need someone to own the test, this someone should also help me edit
the content.  I don't have time to set this up, we're gonna need
a test owner anyways; someone to monitor this and decide what to do
when the numbers go up, etc.  I think we're ready to go, I just need
some tokens spit out with a dump() call.
It sounds like the owner needs to be inside the Netscape firewall or have access
to a machine running a Tinderbox ... maybe dbaron can help?
Making the necessary modifications to the tinderbox scripts doesn't require
being behind the firewall.  They're in mozilla/tools/tinderbox/.


To set up a tinderbox, you need to do roughly the following:

make a directory for the whole thing.  I'll call it /builds/tinderbox/

cd /builds/tinderbox

cvs co mozilla/tools/tinderbox    [ with appropriate CVSROOT or -d ]

mkdir trunk     [or branch, or trunk-clobber, or whatever you want]

cd trunk

../mozilla/tools/tinderbox/install-links
  [ or, for Windows, make the equivalent copies ]

./build-seamonkey.pl  --example-config > tinder-config.pl

Edit tinder-config.pl to make the appropriate changes (note that you must 'make'
rather  than 'gmake' on Windows)

Create /builds/tinderbox/trunk/mozconfig

Then, to start the tinderbox:
[Unix] Make sure that the X server is correct (and has the necessary 'xhost
+local:' or 'xhost +localhost', and run "./tinderbox depend start" or
"./tinderbox clobber start")

[Mac] Make sure you're at the console and not telnetted/sshed in, and run
"./tinderbox depend start" or "./tinderbox clobber start"

[Windows] Make a batch file that does:
  call vcvars32.bat     [or anything else you need to do to set up env vars ]
  cd /builds/tinderbox/trunk
  build-seamonkey.pl --depend --mozconfig mozconfig
     [ or --clobber instead of --depend ]
and then use that batch file to start the tinderbox.



Once you've done that, you have a tinderbox that builds and reports to the
MozillaTest tree.  Then you can hack the tinderbox scripts (mainly
build-seamonkey-utils.pl, but with a few changes to tinder-defaults.pl) to add
code for the new tests.
The owner needs to publish and maintain a URL for the test
to point at.  It could be on-site or off-site.
So all we need is a pretty reliable url for the tests?  I could do that off my 
MIT web-space, methinks...
What's wrong with something under http://mozilla.org/performance/ ?
Indeed.  Again, I could maintain such a testsuite if that's all that's needed.
Boris, please let's try to go with anything that can be done here - it's 
really time to monitor DHTML performance(regressions) as some major changes 
are to be taken.
In case you missed it, no one has gotten back to me with tests, information on
adding tests to tinderbox, or anything of that sort after I volunteered to
maintain the tests...  dbaron is right that these tests could just live on
mozilla.org.  If you point me to some tests, I'll start by checking them in
there; then the ball is in the court of someone with access to actual
tinderboxen to point them to that test... (and tell me whether the test needs to
produce output in any particular format).
No, once somebody puts the tests there, the ball is in the court of whoever
wants to add support for the tests to the tinderbox scripts (which are in
mozilla/tools/tinderbox/).
Ah, ok.  I didn't realize that the scripts could just be modified by anyone with
CVS access...  That's the sort of information I needed.  ;)

I can probably change the scripts once we have tests.
You have the test-suite sent by jason?
Attached file tests (obsolete) — 

    
    

    
  
Meaby i missed something but I wasn't able to run those tests from ./pages
directory from within attachment 142836 [details].
It missess settime() function in some of them, one had faked adress (not exists
in real)...
I also wasn't able to find very important for me tests about Opacity changes and
it's influence on movement speed nor anthing related to tests for with(),DOM
nodes accessing, overwriting values in properties and other important JS methods
for DHTML.

    
    

    
  
-> Boris
Assignee: mcafee → bzbarsky

    
    

    
  
I just attached what kerz sent me.  If you have other tests you want, feel free
to put them in the right format and check them in.

I won't be able to work on this in the foreseeable future (and frankly, no
longer have any interest in doing so even if I could).
Assignee: bzbarsky → chofmann

    
    

    
  
It really would be good / necessary if we could at least get the tests I 
compiled with kerz into tinderbox.
Finding regressions is a tough task anyway - and having that into tinderbox 
would make things so much easier.
@Boris: would it be possible for you to get these tests into tinderbox resp. 
if not who would be able to help?

    
    

    
  


  
I've checked in the tests themselves at
http://www.mozilla.org/performance/test-cases/dhtml/

The tests are not final yet: I'm still fiddling with the algorithm a bit, I'm
planning to remove a few of the tests because they're pointless and I'm open to
suggestions of more tests.

        Attachment #124403 -
        Attachment is obsolete: true

        Attachment #142836 -
        Attachment is obsolete: true

    
    

    
  
Comment on attachment 154035 [details] [diff] [review]
Patch to tinderbox scripts to run DHTML tests

bryner, would you review?

        Attachment #154035 -
        Flags: review?(bryner)

    
    

    
  
Comment on attachment 154035 [details] [diff] [review]
Patch to tinderbox scripts to run DHTML tests

>+      print_log "TinderboxPrint:" .
>+        "<a title=\"DHTML time\" href=\"http://$Settings::results_server/graph/query.cgi?testname=dhtml&tbox=" .
>+          ::hostname() . "&autoscale=1&days=7&avg=1&showpoint=$time,$dhtml_time\">Tdhtml:$dhtml_time" . "ms</a>\n";
>+      
>+      if($Settings::TestsPhoneHome) {
>+        send_results_to_server($dhtml_time, $dhtml_time, "dhtml", ::hostname());
>+      }

You shouldn't print links to the graph server if TestsPhoneHome is false. 
(BloatTest and BloatTest2 do this right; the other tests don't.)

        Attachment #154035 -
        Flags: review?(bryner) → review-

    
    

    
  


  

        Attachment #154035 -
        Attachment is obsolete: true

        Attachment #155143 -
        Flags: superreview?(dbaron)

        Attachment #155143 -
        Flags: review?(dbaron)

    
    

    
  
Comment on attachment 155143 [details] [diff] [review]
Patch with that fixed

>+        send_results_to_server($dhtml_time, $dhtml_time, "dhtml", ::hostname());

The second argument should just be "--" instead of $dhtml_time.

Other than that, r=dbaron

        Attachment #155143 -
        Flags: superreview?(dbaron)

        Attachment #155143 -
        Flags: review?(dbaron)

        Attachment #155143 -
        Flags: review+

    
    

    
  
Luna's running this test now.  Marking fixed, though I may tweak the tests some
going forward...
Status: NEW → RESOLVED
Closed: 19 years ago
Resolution: --- → FIXED
Product: Core → Core Graveyard

          You need to log in
          before you can comment on or make changes to this bug.